Image processing method and device, computer readable storage medium and electronic device

Abstract

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, and relates to the field of image processing technology. The image processing method includes: obtaining a current frame image, the current frame image includes an image area, and dividing the image area into multiple image blocks; determining an image block filtering threshold of the image area based on a histogram, the histogram is obtained according to the characteristic value statistics of the image block, the characteristic value of the image block is the characteristic value of each image block in the image area, or is the characteristic value of each image block in the reference image area corresponding to the image area in the previous frame image of the current frame image; using the image block filtering threshold to filter multiple image blocks in the image area to determine a target image block from the multiple image blocks in the image area; generating a registration image corresponding to the current frame image according to the target image block and the previous frame image of the current frame image. The present disclosure can reduce the hardware overhead of image registration and improve processing efficiency.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Image registration is an important method in image processing technology. In the process of shooting multiple frames, the captured images will have slight differences in content due to the movement of the shooting device. In multi-frame image processing, it is often necessary to first register the images, and then perform other operations such as multi-frame noise reduction, HDR (High Dynamic Range Imaging) exposure fusion, etc.

[0003] Currently, image registration solutions generally have the problems of high hardware overhead and low processing efficiency. Summary of the invention

[0004] The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby overcoming the problems of large hardware overhead and low processing efficiency of image registration solutions at least to a certain extent.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring a current frame image, the current frame image comprising an image region, and dividing the image region into a plurality of image blocks; determining an image block filtering threshold of the image region based on a histogram, the histogram being obtained according to characteristic value statistics of the image blocks, the characteristic values ​​of the image blocks being characteristic values ​​of each image block in the image region, or being characteristic values ​​of each image block in a reference image region corresponding to the image region in a previous frame image of the current frame image; filtering the plurality of image blocks in the image region using the image block filtering threshold to determine a target image block from the plurality of image blocks in the image region; and generating a registration image corresponding to the current frame image according to the target image block and a previous frame image of the current frame image.

[0006] According to a second aspect of the present disclosure, an image processing device is provided, comprising: an image division module, used to obtain a current frame image, the current frame image includes an image area, and the image area is divided into a plurality of image blocks; a threshold determination module, used to determine an image block filtering threshold of the image area based on a histogram, the histogram is obtained according to characteristic value statistics of the image block, the characteristic value of the image block is the characteristic value of each image block in the image area, or is the characteristic value of each image block in a reference image area corresponding to the image area in a previous frame image of the current frame image; an image block filtering module, used to filter the plurality of image blocks in the image area by using the image block filtering threshold, so as to determine a target image block from the plurality of image blocks in the image area; and an image generation module, used to generate a registration image corresponding to the current frame image according to the target image block and a previous frame image of the current frame image.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image processing method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing one or more programs, wherein when the one or more programs are executed by the processor, the processor implements the above-mentioned image processing method.

[0009] In the technical solutions provided by some embodiments of the present disclosure, a histogram is calculated based on the characteristic values ​​of the image blocks to determine the image block filtering threshold of the image area in the current frame image, and the image block is filtered using the threshold to determine the target image block, and registration is performed based on the target image block. The disclosed solution filters the image blocks, and not all image blocks participate in the registration. The specific filtering method relies on the histogram determined based on the characteristic values. In view of the fact that the image feature point detection and histogram acquisition process have less system hardware overhead than the registration process of directly matching the image blocks, the disclosed solution can reduce hardware overhead and improve the processing efficiency of image registration.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing the implementation stage of the image processing solution of the embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram showing a data flow of an embodiment of the present disclosure;

[0014] Figure 3 A flowchart schematically shows an image processing method according to an exemplary embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram showing a method of dividing a current frame image into multiple image regions according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram showing an image, an image region, and an image block in an image region according to an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram showing an operator used in calculating a gradient in the present disclosure is shown;

[0018] Figure 7 A schematic diagram showing a first histogram and a second histogram of an embodiment of the present disclosure is shown;

[0019] Figure 8 A schematic diagram of filtering an image block using a threshold value according to an embodiment of the present disclosure is shown;

[0020] Fig. 9 A schematic diagram showing a result of feature point detection according to an embodiment of the present disclosure;

[0021] Fig.10 A schematic diagram of determining a motion vector according to an embodiment of the present disclosure is shown;

[0022] Fig.11 A schematic diagram showing a motion model of an embodiment of the present disclosure is shown;

[0023] Fig.12 A flowchart showing the entire processing process of the image processing solution of the embodiment of the present disclosure;

[0024] Fig.13 A block diagram schematically shows an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0025] Fig.14 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0027] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation. In addition, all the terms "first", "second", and "third" below are only for the purpose of distinction and should not be used as limitations of the present disclosure.

[0029] It should be noted that the image processing method of the embodiment of the present disclosure can be executed by a terminal device. In this case, the image processing device can be configured in the terminal device. The terminal device may include but is not limited to a smart phone, a tablet computer, a smart wearable device, a personal computer, etc., and the present disclosure does not limit the type of the terminal device.

[0030] In addition, the image processing method of the embodiment of the present disclosure can also be executed by a server. Specifically, the terminal device can send the captured multiple frames of images to the server through the network, and the server executes the image processing method of the embodiment of the present disclosure according to the received images. In this case, the image processing device can be configured in the server. Among them, the server can be a single server, or a server cluster composed of multiple servers, etc.

[0031] The following description will be made by taking a terminal device executing the image processing solution of the embodiment of the present disclosure as an example.

[0032] Figure 1 A schematic diagram showing the implementation stages of the image processing solution of an embodiment of the present disclosure is shown.

[0033] The terminal device can use the current frame image 10 and the previous frame image 11 of the current frame image 10 to perform the image processing process of the exemplary embodiment of the present disclosure to obtain the registered image 12. Then, according to different implementation scenarios, different image post-processing algorithms can be performed on the registered image 12. For example, noise reduction, super-resolution processing, etc.

[0034] Regarding the process of generating the registration image 12, first, the terminal device can obtain the current frame image 10, divide the current frame image 10 into multiple image regions, and divide the image region into multiple image blocks. The terminal device can also determine the image block filtering threshold of the image region based on a histogram calculated based on the feature values ​​of the image blocks.

[0035] Then, the terminal device may filter the image blocks in the image area using the image block filtering threshold value, so as to select the target image block from the image blocks included in the image area.

[0036] Next, the terminal device may generate a registration image corresponding to the current frame according to the target image block and the previous frame image 11 .

[0037] On the one hand, in order to reduce processing delay, the above process of determining the image block filtering threshold may also be determined using a histogram calculated based on feature values ​​of the image blocks contained in the image region of the previous frame of image 11 .

[0038] On the other hand, in order to further reduce hardware overhead and data storage requirements, the image block filtering threshold may be determined based on the histogram corresponding to the high bits of the eigenvalues ​​and the histogram corresponding to the low bits of the eigenvalues.

[0039] In addition, the current frame image can be used as a reference frame, the previous frame image can be used as an input frame, the input frame can be registered, and the registered frame can be output. Figure 2 A schematic diagram showing the data flow of an embodiment of the present disclosure is shown. Figure 2 , when the current frame image is the Nth frame, the terminal device can use the cached N-1th frame (ie, the previous frame) to complete the registration.

[0040] The disclosed embodiment filters the image blocks, and not all image blocks participate in the registration. The specific filtering method relies on the histogram determined based on the eigenvalues. In view of the fact that the image feature point detection and histogram acquisition process have less system hardware overhead than the registration process of directly matching the image blocks, the disclosed solution can reduce the hardware overhead and improve the processing efficiency of image registration.

[0041] It should be noted that the implementation strategies based on delay, storage pressure, etc. in the embodiments of the present disclosure all involve hardware considerations, which can further reduce the processing pressure of the hardware. The solutions are highly universal and have a wide range of application scenarios.

[0042] Figure 3 The flowchart of the image processing method according to the exemplary embodiment of the present disclosure is schematically shown. Figure 3 , the image processing method may include the following steps:

[0043] S32. Obtain a current frame image, where the current frame image includes an image area, and divide the image area into a plurality of image blocks.

[0044] The current frame image may be a video frame image captured in real time by the camera module equipped by the terminal device, or an image that needs to be analyzed and processed in a video previously captured by the camera module equipped by the terminal device. Alternatively, the current frame image may be one of multiple frames of images acquired by the terminal device from other devices or servers.

[0045] In an exemplary embodiment of the present disclosure, the current frame image is a grayscale image.

[0046] The terminal device can obtain the current original image. When the current original image is a RAW image or an RGB image, the terminal device can perform grayscale conversion on the current original image to obtain the current frame image. Taking an RGB image as an example, grayscale conversion can be performed using Formula 1:

[0047] Y=0.299*R+0.587*G+0.114*B(Formula 1)

[0048] When the current original image is a YUV image, the terminal device can extract the brightness component (ie, the Y part) of the current original image to obtain the current frame image.

[0049] After acquiring the current frame image, the terminal device may divide the current frame image into multiple image regions. The sizes of the divided image regions may be the same or different. Figure 4 As shown, the current frame image can be divided into 4×4 image regions, a total of 16 regions.

[0050] After the image regions are obtained, each image region may be divided into a plurality of image blocks, wherein the side length of the image block may be 8 to 32 pixels.

[0051] Figure 5 Schematic diagram showing an image, an image region, and an image block in an image region according to an embodiment of the present disclosure. Figure 5, the current frame image 51 can be divided into 16 image regions, including the image region 52. The image region 52 can be divided into a plurality of image blocks, including the image block 53.

[0052] In addition to dividing each image area into multiple image blocks, in some other embodiments of the present disclosure, the terminal device may also only divide one or more image areas into multiple image blocks, and use these image blocks to perform the following process, which can also achieve the effect of reducing hardware overhead to a certain extent.

[0053] S34. Determine the image block filtering threshold of the image area based on the histogram, the histogram is obtained according to the characteristic value statistics of the image block, the characteristic value of the image block is the characteristic value of each image block in the image area, or is the characteristic value of each image block in the reference image area corresponding to the image area in the previous frame image of the current frame image.

[0054] The image block filtering threshold of the embodiment of the present disclosure is used to filter the image blocks in the image area to filter out the image blocks that meet the threshold requirement. Specifically, the image blocks that are less than the image block filtering threshold can be filtered out, and the image blocks that are greater than or equal to the image block filtering threshold can be filtered out and stored separately.

[0055] It should be noted that the image block filtering thresholds corresponding to different image regions are usually different. The following will describe how to determine the image block filtering threshold for an image region. It is easy to understand that for each divided image region, the following process of determining the image block filtering threshold can be performed.

[0056] According to some embodiments of the present disclosure, the characteristic values ​​of the image blocks corresponding to the image area in the current frame image may be calculated, and then a histogram may be statistically generated to determine the image block filtering threshold of the image area.

[0057] However, during the hardware execution, the histogram statistics can only be started after the feature point calculations in the image area are all completed, which will cause processing delays.

[0058] Considering that when shooting multiple frames, the interval between frames is short and the image content of each image area does not change much, according to other embodiments of the present disclosure, the image block filtering threshold can be determined using the data of the previous frame, thereby eliminating the delay.

[0059] The following describes the process of determining the image block filtering threshold of the image area by taking the previous frame of image data as an example.

[0060] First, the terminal device can determine a reference image area in the previous frame image that corresponds to an image area in the current frame image. Given that the shooting time interval between the current frame image and the previous frame image is very small, it can be understood that the position of the reference image area in the previous frame image is equivalent to the position of the image area in the current frame image.

[0061] Next, the terminal device may calculate the characteristic values ​​of each image block in the reference image region, and generate a histogram based on the characteristic values.

[0062] Regarding the process of calculating the eigenvalue, the terminal device may calculate the gradient of the pixel points of the image block in the reference image area in the horizontal direction (ie, the x direction) and the gradient in the vertical direction (ie, the y direction).

[0063] For example, the Sobel operator can be used to calculate the gradient. Figure 6 The forms of the horizontal operator and the vertical operator are schematically shown. It should be understood that Figure 6 The operator is only an example, and the present disclosure may also use other operators to implement gradient calculation.

[0064] After determining the horizontal and vertical gradients of each pixel in the image block, the characteristic value R of the image block can be calculated using Formula 2 to Formula 5:

[0065]

[0066] R = det(M) - k(trace(M)) 2 (Formula 3)

[0067] det(M)=λ1λ2 (Formula 4)

[0068] trace(M)=λ1+λ2 (Formula 5)

[0069] Among them, I x is the gradient of the pixel in the horizontal direction, I y is the gradient of the pixel in the vertical direction, k can be 0.0625, and w(x,y) = 1. In addition, λ1 and λ2 represent the two eigenvalues ​​of the determinant M, or, by determining det(M) as the value of the determinant M and trace(M) as the sum of the diagonal data of the determinant M, the eigenvalue R of the image block can also be calculated.

[0070] The disclosed embodiment adopts the Harris detection algorithm to calculate the eigenvalues. Compared with other methods, it has advantages such as direction invariance, which helps to improve the accuracy of subsequent matching operations.

[0071] After the characteristic value of the image block is calculated, a histogram may be generated based on the characteristic value of each image block in the image region.

[0072] Then, an image block filtering threshold of the image region may be determined based on the histogram.

[0073] According to some embodiments of the present disclosure, the terminal device may determine the first image block filtering threshold according to the frequency of the characteristic value in the histogram, wherein the frequency of the characteristic value in the histogram refers to the number of characteristic values ​​in one characteristic value or one range in the image block.

[0074] Specifically, the characteristic value corresponding to the first predetermined critical ratio can be determined in descending order of the frequency of the histogram as the first image block filtering threshold. For example, the 10% with the largest frequency in the histogram is taken, and the characteristic value corresponding to the 10% quantile in the histogram is determined as the first image block filtering threshold.

[0075] After the first image block filtering threshold is determined, in one embodiment, the first image block filtering threshold may be determined as the image block filtering threshold of the image region.

[0076] In another embodiment, the terminal device may determine a preset second image block filtering threshold, compare the first image block filtering threshold with the second image block filtering threshold, and determine the larger threshold between the first image block filtering threshold and the second image block filtering threshold as the image block filtering threshold of the image area. That is, when the first image block filtering threshold is greater than the second image block filtering threshold, the first image block filtering threshold may be determined as the image block filtering threshold of the image area; when the first image block filtering threshold is less than the second image block filtering threshold, the second image block filtering threshold may be determined as the image block filtering threshold of the image area.

[0077] By combining the first image block filtering threshold and the second image block filtering threshold, it is possible to avoid the situation where screening cannot be performed due to the determined first image block filtering threshold being low.

[0078] In addition, there are multiple multiplication operations in the calculation of the above eigenvalues. For example, if the input image is 8 bits wide, the calculated eigenvalue R will exceed 32 bits, which will increase the number of bins storing eigenvalues ​​when calculating the histogram, and require high storage resources during hardware execution.

[0079] In order to reduce the hardware overhead in this regard, in some other embodiments of the present disclosure, a solution is provided for determining an image block filtering threshold of an image region based on a quantile histogram.

[0080] First, the distribution range threshold of the eigenvalue can be determined, and the eigenvalue is constrained by the distribution range threshold to obtain the constrained eigenvalue, and a histogram of the constrained eigenvalue is statistically calculated as the histogram used to determine the image block filtering threshold of the image area.

[0081] In practice, the main distribution range of the eigenvalue R is [-32768, 65535], which is the storage capacity of 17-bit signed data. Considering that only image blocks with positive eigenvalues ​​are retained. In these embodiments, the distribution range threshold of the eigenvalue can be determined as [0, 65535]. That is, if the eigenvalue is less than 0, it is constrained to 0; if the eigenvalue is greater than 65535, it is constrained to 65535. Thus, the eigenvalue can be limited to unsigned 16-bit data.

[0082] The determined histogram includes a first histogram and a second histogram respectively calculated according to the number of eigenvalue bits, wherein the number of eigenvalue bits corresponding to the first histogram is higher than the number of eigenvalue bits corresponding to the second histogram. Still taking the above 16 bits as an example, the first histogram corresponds to the upper 8 bits, and the second histogram corresponds to the lower 8 bits.

[0083] Specifically, the mask can be used to obtain the high and low bits of the eigenvalue fp_v, for example:

[0084] Low_fp_v = fp_v & 0xff00

[0085] High_fp_v=(fp_v&0x00ff)>>8

[0086] In addition, you can add conditional judgments to the low bits, and change the characteristic values ​​of the low bits if the conditions are met:

[0087] if(high_fp_v>0):low_fp_v=255

[0088] Thus, the first histogram of the high position and the second histogram of the low position can be statistically calculated respectively. Figure 7 The manner of determining the first histogram and the second histogram is depicted as an example.

[0089] After the first histogram and the second histogram are counted, on the one hand, the first intermediate threshold value can be determined according to the frequency of the eigenvalues ​​in the first histogram. Specifically, the eigenvalue corresponding to the second predetermined ratio critical point can be determined in the order of the frequency of the first histogram from high to low, as the first intermediate threshold value. On the other hand, the second intermediate threshold value can be determined according to the frequency of the eigenvalues ​​in the second histogram. Specifically, the eigenvalue corresponding to the third predetermined ratio critical point can be determined in the order of the frequency of the second histogram from high to low, as the second intermediate threshold value.

[0090] The second predetermined ratio and the third predetermined ratio may be the same as the first predetermined ratio, for example, both are 10%, that is, the corresponding thresholds are the characteristic values ​​corresponding to the maximum 10% quantiles in the histogram. In addition, the second predetermined ratio and the third predetermined ratio may also be different from the first predetermined ratio, and the present disclosure does not limit the specific values ​​of the three.

[0091] Then, the terminal device may determine the first image block filtering threshold based on the first intermediate threshold and the second intermediate threshold. For example, when the first intermediate threshold is greater than 0, the first intermediate threshold may be determined as the first image block filtering threshold. In addition, the second intermediate threshold may be determined as the first image block filtering threshold.

[0092] The first intermediate threshold is recorded as high_fp_th, and the second intermediate threshold is recorded as low_fp_th. Then, the method of determining the first image block filtering threshold can be expressed as follows:

[0093] If(high_fp_th>0)

[0094] th_s=(high_fp_th<<8)

[0095] Else

[0096] th_s=low_fp_th

[0097] After the first image block filtering threshold is obtained, in one embodiment, the first image block filtering threshold may be used to determine the image block filtering threshold of the image region.

[0098] In another embodiment, the terminal device may determine a preset second image block filtering threshold, compare the first image block filtering threshold with the second image block filtering threshold, and determine the larger threshold between the first image block filtering threshold and the second image block filtering threshold as the image block filtering threshold of the image area. That is, when the first image block filtering threshold is greater than the second image block filtering threshold, the first image block filtering threshold may be determined as the image block filtering threshold of the image area; when the first image block filtering threshold is less than the second image block filtering threshold, the second image block filtering threshold may be determined as the image block filtering threshold of the image area.

[0099] By combining the first image block filtering threshold and the second image block filtering threshold, it is possible to avoid the situation where screening cannot be performed due to the determined first image block filtering threshold being low.

[0100] S36. Filter the multiple image blocks in the image area using the image block filtering threshold to determine a target image block from the multiple image blocks in the image area.

[0101] Figure 8 The process of determining the target image block from the image area is schematically shown. It should be noted that in the solution based on hardware implementation, the terminal device can configure storage space for the target image block separately, that is, the determined target image block can be stored separately, and the present disclosure does not adjust the original image of the current frame image, and the terminal device can still store the original image of the current frame image.

[0102] The above-mentioned image block filtering process based on the image block filtering threshold helps the matching of the model in the subsequent processing. It can be imagined that if the feature points are concentrated in one area of ​​the picture, the matching model finally calculated will tend to fit this area. The threshold-based filtering method of this solution can avoid this situation and help global matching.

[0103] It should be noted that the above process of determining the target image blocks can be performed for each divided image area to obtain a set of target image blocks for the current frame image.

[0104] In addition, the center point of the determined target image block may be used as a feature point, and its coordinates may be recorded. Fig. 9 The detection result of feature points is schematically shown.

[0105] S38. Generate a registration image corresponding to the current frame image according to the target image block and the previous frame image of the current frame image.

[0106] First, the terminal device may use the target image block and the previous frame image to calculate the motion vector of the current frame image relative to the previous frame image.

[0107] Specifically, the terminal device can calculate the similarity between the target image block and the image block included in the previous frame of the image. The image block similarity calculation method used in the embodiment of the present disclosure can be a SAD (Sum of Absolute Differences) algorithm. In view of the small size of the image block and the small consumption of SAD resources, a better processing effect can be achieved in engineering. However, MAD (Mean Absolute Differences) algorithm, SSD (Sum of Squared Differences) algorithm, SSIM (Structural Similarity) algorithm, etc. can also be used, and the present disclosure does not limit this.

[0108] Based on the calculation result of the similarity, the motion vector of the current frame image relative to the previous frame image can be determined.

[0109] Specifically, when the similarity between the target image block and a reference image block in the previous frame image is greater than a similarity threshold, the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image are obtained, and based on the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image, the motion vector of the current frame image relative to the previous frame image is determined.

[0110] For example, a motion vector can be expressed as [mv x ,mv y ], if the center point coordinates of the target image block are [x, y], and the center point coordinates of the reference image block are [x', y'], then mv x =x'-x,mv y =y'-y.

[0111] In addition, the operation of determining the image block match to determine the reference image block may be implemented by using a search window. Fig.10 A method of determining a motion vector based on a search window is shown. The image block with the highest similarity can be found through the search window to achieve the best image block matching.

[0112] Next, a motion model may be calculated based on the motion vector, and the motion model may be used to generate a registration image corresponding to the current frame image.

[0113] The present disclosure does not limit the type of motion model, for example, it may be an affine transformation model, a homography transformation model, etc. Fig.11 The affine transformation model is used as an example to illustrate the Fig.11 The motion model of each image area is determined, and the pixel alignment operation is realized to generate a registered image.

[0114] Specifically, first, the terminal device can use the motion model to determine the coordinates of the matching pixel point corresponding to the target pixel point on the current frame image, where the target pixel point is any pixel point on the current frame image. Next, the terminal device can determine the pixel value corresponding to the coordinates of the matching pixel point in the previous frame image. Then, the terminal device can assign the pixel value to the target pixel point.

[0115] After performing the above pixel value assignment operation on all pixels in the current frame image pixel by pixel, a new image can be obtained, that is, a registered image corresponding to the current frame image.

[0116] The following will refer to Fig.12 The whole process of determining the registration image according to the embodiment of the present disclosure is described.

[0117] In step S1202, the terminal device may divide the current frame image into multiple image areas.

[0118] In step S1204, the terminal device may divide each image area into multiple image blocks.

[0119] In step S1206, the terminal device may calculate the feature value of the image block in the image area of ​​the previous frame of image.

[0120] In step S1208, the terminal device may generate a histogram based on the calculated characteristic values.

[0121] In step S1210, the terminal device may determine a first image block filtering threshold according to the histogram.

[0122] In step S1212, the terminal device may determine a preset second image block filtering threshold.

[0123] In step S1214, the terminal device may determine a final filtering threshold of the image block corresponding to the image area according to the first image block filtering threshold and the second image block filtering threshold.

[0124] In step S1216, the terminal device may filter the image blocks in the image area using the filtering threshold determined in step S1214 to determine the target image block.

[0125] In step S1218, the terminal device may determine a registered image using the target image block and the previous frame image.

[0126] After the registered image is determined, processing such as noise reduction and super-resolution can be performed.

[0127] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0128] Furthermore, this exemplary embodiment also provides an image processing device.

[0129] Fig.13 FIG. 1 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Fig.13 According to an exemplary embodiment of the present disclosure, the image processing device 13 may include an image dividing module 131 , a threshold determining module 133 , an image block filtering module 135 , and an image generating module 137 .

[0130] Specifically, the image division module 131 can be used to obtain a current frame image, which includes an image area, and divides the image area into multiple image blocks; the threshold determination module 133 can be used to determine the image block filtering threshold of the image area based on a histogram, and the histogram is obtained according to the characteristic value statistics of the image block, and the characteristic value of the image block is the characteristic value of each image block in the image area, or is the characteristic value of each image block in the reference image area corresponding to the image area in the previous frame image of the current frame image; the image block filtering module 135 can be used to filter multiple image blocks in the image area using the image block filtering threshold to determine a target image block from the multiple image blocks in the image area; the image generation module 137 can be used to generate a registration image corresponding to the current frame image based on the target image block and the previous frame image of the current frame image.

[0131] According to an exemplary embodiment of the present disclosure, the process in which the threshold determination module 133 determines the image block filtering threshold of the image area based on the histogram can be configured to perform: determining the first image block filtering threshold according to the frequency of the characteristic value in the histogram; and determining the image block filtering threshold of the image area based on the first image block filtering threshold.

[0132] According to an exemplary embodiment of the present disclosure, the process of determining the first image block filtering threshold by the threshold determination module 133 can be configured to execute: determining the eigenvalue corresponding to the first predetermined proportion critical point in the order of the frequency of the histogram from high to low, as the first image block filtering threshold.

[0133] According to an exemplary embodiment of the present disclosure, the process of the threshold determination module 133 generating a histogram based on the eigenvalue can also be configured to perform: determining a distribution range threshold of the eigenvalue; constraining the eigenvalue using the distribution range threshold to obtain the constrained eigenvalue; and generating a histogram of the constrained eigenvalue as a histogram used to determine the image block filtering threshold of the image area.

[0134] According to an exemplary embodiment of the present disclosure, the histogram includes a first histogram and a second histogram respectively counted according to the number of eigenvalue bits, and the number of eigenvalue bits corresponding to the first histogram is higher than the number of eigenvalue bits corresponding to the second histogram. In this case, the process of the threshold determination module 133 determining the image block filtering threshold of the image area based on the histogram can also be configured to perform: determining a first intermediate threshold according to the frequency of the eigenvalues ​​in the first histogram; determining a second intermediate threshold according to the frequency of the eigenvalues ​​in the second histogram; determining a first image block filtering threshold based on the first intermediate threshold and the second intermediate threshold; determining the image block filtering threshold of the image area based on the first image block filtering threshold.

[0135] According to an exemplary embodiment of the present disclosure, the threshold determination module 133 can be configured to perform: determining the eigenvalue corresponding to the second predetermined proportion critical point in the order of the frequency of the first histogram from high to low, as the first intermediate threshold; and determining the eigenvalue corresponding to the third predetermined proportion critical point in the order of the frequency of the second histogram from high to low, as the second intermediate threshold.

[0136] According to an exemplary embodiment of the present disclosure, the process in which the threshold determination module 133 determines the image block filtering threshold of the image area based on the first image block filtering threshold can be configured to perform: determining a preset second image block filtering threshold; comparing the first image block filtering threshold with the second image block filtering threshold; and determining the larger threshold between the first image block filtering threshold and the second image block filtering threshold as the image block filtering threshold of the image area.

[0137] According to an exemplary embodiment of the present disclosure, the process of calculating the characteristic value of each image block in the reference image area by the threshold determination module 133 can be configured to perform: calculating the gradient of the pixel points of the image block in the reference image area in the horizontal direction and the gradient in the vertical direction; calculating the characteristic value of the image block based on the gradient of the pixel points of the image block in the horizontal direction and the gradient in the vertical direction.

[0138] According to an exemplary embodiment of the present disclosure, the image generation module 137 can be configured to perform: using the target image block and the previous frame image, calculating the motion vector of the current frame image relative to the previous frame image; calculating the motion model based on the motion vector; using the motion model, generating a registration image corresponding to the current frame image.

[0139] According to an exemplary embodiment of the present disclosure, the process of calculating the motion vector by the image generation module 137 can be configured to perform: calculating the similarity between the target image block and the image block included in the previous frame image; based on the calculation result of the similarity, determining the motion vector of the current frame image relative to the previous frame image.

[0140] According to an exemplary embodiment of the present disclosure, the process of calculating the motion vector by the image generation module 137 can be configured to perform: when the similarity between the target image block and a reference image block in the previous frame image is greater than a similarity threshold, obtaining the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image; and determining the motion vector of the current frame image relative to the previous frame image based on the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image.

[0141] According to an exemplary embodiment of the present disclosure, the process of the image generation module 137 generating a registration image corresponding to the current frame image using a motion model can be configured to perform: using the motion model to determine the matching pixel point coordinates corresponding to the target pixel point on the current frame image, where the target pixel point is any pixel point on the current frame image; determining the pixel value corresponding to the matching pixel point coordinates in the previous frame image; assigning the pixel value corresponding to the matching pixel point coordinates in the previous frame image to the target pixel point to generate a registration image corresponding to the current frame image.

[0142] According to an exemplary embodiment of the present disclosure, the image division module 131 may also be configured to execute: acquiring a current original image, which is a RAW image or an RGB image; and performing grayscale conversion on the current original image to obtain a current frame image.

[0143] According to an exemplary embodiment of the present disclosure, the image division module 131 may also be configured to execute: acquiring a current original image, which is a YUV image; and extracting a brightness component of the current original image to obtain a current frame image.

[0144] Since the functional modules of the image processing device in the embodiment of the present disclosure are the same as those in the above-mentioned method embodiment, they will not be described in detail here.

[0145] Fig.14 Schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. The terminal device of the exemplary embodiment of the present disclosure may be configured as follows Fig.14 It should be noted that Fig.14 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0146] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the image processing method of the exemplary embodiment of the present disclosure.

[0147] Specifically, Fig.14As shown, the electronic device 140 may include: a processor 1410, an internal memory 1421, an external memory interface 1422, a Universal Serial Bus (USB) interface 1430, a charging management module 1440, a power management module 1441, a battery 1442, an antenna 1, an antenna 2, a mobile communication module 1450, a wireless communication module 1460, an audio module 1470, a sensor module 1480, a display screen 1490, a camera module 1491, an indicator 1492, a motor 1493, a button 1494, and a Subscriber Identification Module (SIM) card interface 1495, etc. The sensor module 1480 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0148] It is to be understood that the structure illustrated in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 140. In other embodiments of the present disclosure, the electronic device 140 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0149] The processor 1410 may include one or more processing units, for example, the processor 1410 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 1410 for storing instructions and data.

[0150] The electronic device 140 can realize the shooting function through the ISP, the camera module 1491, the video codec, the GPU, the display screen 1490 and the application processor. In some embodiments, the electronic device 140 may include 1 or N camera modules 1491, N is a positive integer greater than 1, and if the electronic device 140 includes N cameras, one of the N cameras is a main camera. The frame images described in the embodiments of the present disclosure can all be obtained by shooting the camera module 1491.

[0151] The internal memory 1421 may be used to store computer executable program codes, which may include instructions. The internal memory 1421 may include a program storage area and a data storage area. The external memory interface 1422 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 140.

[0152] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0153] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0155] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the embodiments of the present disclosure.

[0156] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0158] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0159] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0160] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a current frame image, wherein the current frame image includes an image area, and divide the image area into a plurality of image blocks; Determine an image block filtering threshold of the image region based on a histogram, wherein the histogram is obtained according to characteristic values ​​of the image blocks, and the characteristic values ​​of the image blocks are characteristic values ​​of each of the image blocks in the image region, or characteristic values ​​of each of the image blocks in a reference image region corresponding to the image region in a previous frame image of the current frame image; filtering the multiple image blocks in the image area by using the image block filtering threshold value, so as to determine a target image block from the multiple image blocks in the image area; A registration image corresponding to the current frame image is generated according to the target image block and a previous frame image of the current frame image.

2. The image processing method according to claim 1, characterized in that: Determining the image block filtering threshold of the image area based on the histogram includes: Determining a first image block filtering threshold according to the frequency of the characteristic values ​​in the histogram; An image block filtering threshold for the image area is determined based on the first image block filtering threshold.

3. The image processing method according to claim 2, characterized in that: Determining a first image block filtering threshold according to the frequency of the characteristic value in the histogram includes: According to the order of the frequencies of the eigenvalues ​​in the histogram from high to low, the eigenvalue corresponding to the first predetermined critical ratio is determined as the first image block filtering threshold.

4. The image processing method according to claim 1, characterized in that: The image processing method further comprises: Determine the distribution range threshold of the feature value; Using the distribution range threshold to constrain the eigenvalue to obtain a constrained eigenvalue; A histogram of the constrained eigenvalues ​​is calculated as a histogram used to determine an image block filtering threshold of the image region.

5. The image processing method according to claim 4, characterized in that: The histogram includes a first histogram and a second histogram respectively obtained by counting the number of eigenvalue bits, and the number of eigenvalue bits corresponding to the first histogram is higher than the number of eigenvalue bits corresponding to the second histogram; wherein determining the image block filtering threshold of the image area based on the histogram includes: Determining a first intermediate threshold value according to the frequency of the characteristic value in the first histogram; Determining a second intermediate threshold value according to the frequency of the characteristic value in the second histogram; Determining a first image block filtering threshold based on the first intermediate threshold and the second intermediate threshold; An image block filtering threshold for the image area is determined based on the first image block filtering threshold.

6. The image processing method according to claim 5, characterized in that: Determining a first intermediate threshold according to the frequency of the characteristic value in the first histogram includes: Determine, in descending order of the frequencies of the eigenvalues ​​in the first histogram, the eigenvalue corresponding to the second predetermined critical ratio as the first intermediate threshold; and Determining a second intermediate threshold according to the frequency of the characteristic value in the second histogram includes: According to the order of the frequencies of the eigenvalues ​​in the second histogram from high to low, the eigenvalue corresponding to the third predetermined critical ratio is determined as the second intermediate threshold.

7. The image processing method according to claim 2 or 5, characterized in that: Determining the image block filtering threshold of the image area based on the first image block filtering threshold comprises: Determining a preset second image block filtering threshold; comparing the first image block filtering threshold with the second image block filtering threshold; A larger threshold value between the first image block filtering threshold value and the second image block filtering threshold value is determined as the image block filtering threshold value of the image area.

8. The image processing method according to claim 1, characterized in that: The image processing method further comprises: Calculate the horizontal and vertical gradients of the pixels of the image block; The characteristic value of the image block is calculated according to the gradient of the pixel points of the image block in the horizontal direction and the gradient in the vertical direction.

9. The image processing method according to claim 1, characterized in that: Generating a registration image corresponding to the current frame image according to the target image block and a previous frame image of the current frame image, comprising: Calculating a motion vector of the current frame image relative to the previous frame image by using the target image block and the previous frame image; Calculating a motion model based on the motion vector; Using the motion model, a registration image corresponding to the current frame image is generated.

10. The image processing method according to claim 9, characterized in that: Calculating a motion vector of the current frame image relative to the previous frame image by using the target image block and the previous frame image includes: Calculating the similarity between the target image block and the image block included in the previous frame image; Based on the calculation result of the similarity, a motion vector of the current frame image relative to the previous frame image is determined.

11. The image processing method according to claim 10, characterized in that: Determining a motion vector of the current frame image relative to the previous frame image based on a similarity calculation result includes: When the similarity between the target image block and a reference image block in the previous frame image is greater than a similarity threshold, obtaining the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image; The motion vector of the current frame image relative to the previous frame image is determined according to the coordinates of the center point of the target image block in the current frame image and the coordinates of the center point of the reference image block in the previous frame image.

12. The image processing method according to claim 9, characterized in that: Using the motion model, generating a registration image corresponding to the current frame image, including: Determine the coordinates of a matching pixel point corresponding to a target pixel point on the current frame image by using the motion model, wherein the target pixel point is any pixel point on the current frame image; Determine the pixel value corresponding to the coordinates of the matching pixel point in the previous frame image; The pixel value corresponding to the coordinates of the matching pixel point in the previous frame image is assigned to the target pixel point to generate a registered image corresponding to the current frame image.

13. The image processing method according to claim 1, characterized in that: Getting the current frame image includes: Obtain a current original image, where the current original image is a RAW image or an RGB image; Grayscale conversion is performed on the current original image to obtain the current frame image.

14. The image processing method according to claim 1, characterized in that: Getting the current frame image includes: Obtain a current original image, where the current original image is a YUV image; The brightness component of the current original image is extracted to obtain the current frame image.

15. An image processing device, characterized in that: include: An image division module is used to obtain a current frame image, wherein the current frame image includes an image area, and divide the image area into a plurality of image blocks; a threshold determination module, configured to determine an image block filtering threshold of the image area based on a histogram, wherein the histogram is obtained according to characteristic values ​​of the image blocks, and the characteristic values ​​of the image blocks are characteristic values ​​of each of the image blocks in the image area, or characteristic values ​​of each of the image blocks in a reference image area corresponding to the image area in a previous frame of the current frame of the image; an image block filtering module, configured to filter the multiple image blocks in the image area by using the image block filtering threshold, so as to determine a target image block from the multiple image blocks in the image area; The image generation module is used to generate a registration image corresponding to the current frame image according to the target image block and the previous frame image of the current frame image.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 14 is implemented.

17. An electronic device, characterized in that: include: processor; A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the image processing method according to any one of claims 1 to 14.

Images