Image processing method and device, electronic device, and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device, and a storage medium. The method includes: obtaining the depth of field data of an input image, and obtaining the weight matrix and weighted image of the input image based on the depth of field data; performing lateral integration on the weight matrix and the weighted image respectively to obtain a first lateral integral map corresponding to the weight matrix and a second lateral integral map corresponding to the weighted image; based on the above two lateral integral maps, respectively obtaining the fuzzy values ​​corresponding to each pixel in the weight matrix to obtain a weighted fuzzy image corresponding to the weight matrix and a weighted fuzzy image corresponding to the weighted image; obtaining a target image based on the weighted fuzzy image and the weighted fuzzy image. The embodiment of the present disclosure obtains a fuzzy kernel suitable for any shape by obtaining a lateral integral map to obtain a fuzzy value, which can improve the scope of application.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] Currently, when using electronic devices for shooting, users use cameras or smart terminals to adjust parameters such as focal length, aperture or shutter to adjust the depth of field range, thereby obtaining a blurred image with a clear target and a blurred background, with the degree of blur gradually changing with the depth of the image, thereby achieving an artistic effect of highlighting the target and weakening the background.

[0003] In the related art, commonly used image depth of field rendering methods may include a gathering blur method, which is to add the image pixels corresponding to the current blur kernel to obtain the blur value of the current pixel. Since the blur value of the current pixel is only related to a single blur kernel sliding window, it can be calculated in parallel and meet the requirements of high performance. When parallelizing the calculation, an integral image can be used in the related art. For example, the mean blur calculation of a rectangle is simplified to the calculation of the four corner points of the integral image, that is, the pixel value of each pixel in the integral image corresponds to the sum of the pixel values ​​of all pixels in its upper left corner. In this way, the mean blur value of the square can be obtained by subtracting the pixel values ​​of the four corner points of the square area.

[0004] However, it is precisely because the integral image corresponds to the sum of the pixel values ​​of all pixels in the upper left corner that it is not applicable when the blur kernel is in a shape other than a square, and its scalability is relatively poor. Summary of the invention

[0005] The present disclosure provides an image processing method and device, an electronic device, and a storage medium to solve the deficiencies of the related art.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0007] Acquire depth data of an input image, and acquire a weight matrix of the input image based on the depth data;

[0008] Sequentially obtain the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; the product is the pixel value of each pixel in the weighted image;

[0009] Performing horizontal integration on the weight matrix and the weighted image respectively to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel;

[0010] Based on the first lateral integral image, obtaining a fuzzy value corresponding to each pixel in the weight matrix to obtain a weighted fuzzy image corresponding to the weight matrix; and based on the second lateral integral image, obtaining a fuzzy value corresponding to each pixel in the weighted image to obtain a weighted fuzzy image corresponding to the weighted image;

[0011] The ratios of the fuzzy values ​​corresponding to each pixel in the weighted fuzzy image and the fuzzy values ​​corresponding to each pixel in the weighted fuzzy image are obtained in sequence, and the ratios are used as the pixel values ​​of each pixel in the target image; the target image is a blurred image after blurring the input image.

[0012] In one embodiment, obtaining a weight matrix of the input image based on the depth of field data includes:

[0013] According to the depth data, the weights of pixels within the depth range are reset to zero and the weights of pixels outside the depth range are reset to preset values, so as to obtain a weight matrix of the input image.

[0014] Optionally, resetting the weights of pixels outside the depth of field range to preset values ​​includes:

[0015] Comparing pixel values ​​of pixels outside the depth of field range with a preset pixel value threshold;

[0016] When the pixel value is less than the pixel value threshold, the weight of the corresponding pixel is reset to a preset first value; when the pixel value exceeds the pixel value threshold, the weight of the corresponding pixel is reset to a preset second value; the first value is less than the second value.

[0017] In one embodiment, performing a transverse integration on the weight matrix to obtain a first transverse integral map corresponding to the weight matrix includes:

[0018] Filling the weight matrix and the weighted image respectively to obtain the filled weight matrix and weighted image;

[0019] For each pixel in the filled weight matrix and weighted image, the sum of the pixel values ​​of the pixel before the current pixel in the row and the current pixel are obtained in turn, and the sum of the pixel values ​​is used as the horizontal integral value of the current pixel to obtain the first horizontal integral map corresponding to the weight matrix and the second horizontal integral map corresponding to the weighted image.

[0020] In one embodiment, obtaining the blur value corresponding to each pixel in the weight matrix based on the first lateral integral image includes:

[0021] Acquire an initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image;

[0022] Performing a modulo operation on the initial blur kernel radius and a preset number of levels, and taking the quotient of the operation result as the actual blur kernel radius corresponding to each pixel;

[0023] Based on the first lateral integral image, the blur value corresponding to each pixel is determined according to the actual blur kernel radius and the preset blur kernel shape, and a weighted blurred image corresponding to the weight matrix is ​​obtained.

[0024] In one embodiment, obtaining a blur value corresponding to each pixel in the weighted image based on the second lateral integral image includes:

[0025] Acquire an initial blur kernel radius corresponding to each pixel in the weighted image according to the depth data of the input image;

[0026] Performing a modulo operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient of the operation result as the actual blur kernel radius corresponding to each pixel;

[0027] Based on the second lateral integral image, the blur value corresponding to each pixel is determined according to the actual blur kernel radius and the preset blur kernel shape, so as to obtain a weighted blurred image corresponding to the weighted image.

[0028] Optionally, determining the blur value corresponding to each pixel according to the actual blur kernel radius and a preset blur kernel shape includes:

[0029] Based on the preset correspondence between the blur kernel offset and the blur kernel radius in the blur kernel shape, obtaining a first offset and a second offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the correspondence;

[0030] For each row of pixels in the blur kernel shape, respectively obtaining a first lateral integral value corresponding to the first offset in the lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when obtaining a weighted blurred image, the lateral integral map is a first lateral integral map; and when obtaining a weighted blurred image, the lateral integral map is a second lateral integral map;

[0031] Obtaining the sum of pixel values ​​of corresponding rows of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value;

[0032] The sum of the pixel values ​​corresponding to each row of pixels in the blur kernel shape is accumulated to obtain an initial blur value of each pixel.

[0033] Optionally, after obtaining the initial blur value of each pixel, the method further includes:

[0034] If the blur kernel shape is a convex shape, then determining the initial blur value of each pixel is the target blur value;

[0035] If the blur kernel shape is a non-convex shape, a corrected blur value corresponding to each pixel is obtained and the initial blur value is corrected by using the corrected blur value to obtain a target blur value.

[0036] Optionally, obtaining a corrected blur value corresponding to each pixel includes:

[0037] Based on the preset correction relationship between the blur kernel offset and the blur kernel radius within the blur kernel shape, obtaining a first correction offset and a second correction offset corresponding to each row of pixels within the blur kernel shape according to the actual blur kernel radius of each pixel and the correction relationship;

[0038] For each row of pixels in the blur kernel shape, respectively obtaining a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map;

[0039] Acquire the sum of the corrected pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value;

[0040] The sum of the corrected pixel values ​​corresponding to each row of pixels in the blur kernel shape is accumulated to obtain the corrected blur value of each pixel.

[0041] Optionally, after determining the blur value corresponding to each pixel according to the actual blur kernel radius and the preset blur kernel shape, the method further includes:

[0042] When the remainder in the operation result is a non-zero value, the actual blur kernel radius is subtracted by 1 to obtain a first blur kernel radius, and a blur value corresponding to each pixel is obtained based on the first blur kernel radius;

[0043] A weighted blur value of each pixel is obtained based on the actual blur kernel radius, the remainder, the blur value of each pixel corresponding to the actual blur kernel radius, and the blur value of each pixel corresponding to the first blur kernel radius, and the weighted blur value is used as the target blur value corresponding to each pixel.

[0044] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0045] A depth of field data acquisition module, used to acquire the depth of field data of an input image;

[0046] A weight matrix acquisition module, used to acquire a weight matrix of the input image based on the depth of field data;

[0047] A weighted image acquisition module, used to sequentially acquire the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; the product is the pixel value of each pixel in the weighted image;

[0048] An integral image acquisition module is used to perform horizontal integration on the weight matrix and the weighted image respectively to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel;

[0049] A fuzzy image acquisition module, configured to acquire, based on the first lateral integral image, a fuzzy value corresponding to each pixel in the weight matrix, and obtain a weighted fuzzy image corresponding to the weight matrix; and to acquire, based on the second lateral integral image, a fuzzy value corresponding to each pixel in the weighted image, and obtain a weighted fuzzy image corresponding to the weighted image;

[0050] The target image acquisition module is used to acquire the target image corresponding to the input image based on the weighted blurred image and the weight blurred image; the target image is a blurred image after the input image is blurred.

[0051] In one embodiment, the weight matrix acquisition module includes:

[0052] The weight matrix acquisition submodule is used to reset the weights of pixels within the depth of field range to zero and reset the weights of pixels outside the depth of field range to preset values ​​according to the depth of field data, so as to obtain the weight matrix of the input image.

[0053] Optionally, the weight matrix acquisition submodule includes:

[0054] A pixel value comparison unit, used for comparing the pixel value of the pixel outside the depth of field range with a preset pixel value threshold;

[0055] A weight setting unit is used to reset the weight of the corresponding pixel to a preset first value when the pixel value is less than the pixel value threshold; when the pixel value exceeds the pixel value threshold, reset the weight of the corresponding pixel to a preset second value; the first value is less than the second value.

[0056] In one embodiment, the integral image acquisition module includes:

[0057] A filling submodule, used to fill the weight matrix and the weighted image respectively to obtain the filled weight matrix and weighted image;

[0058] The integral submodule is used to obtain the sum of the pixel values ​​of the pixels before the current pixel in the row and the current pixel in turn for each pixel in the filled weight matrix and weighted image, and use the sum of the pixel values ​​as the horizontal integral value of the current pixel to obtain the first horizontal integral image corresponding to the weight matrix and the second horizontal integral image corresponding to the weighted image.

[0059] In one embodiment, the blurred image acquisition module includes:

[0060] An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image;

[0061] An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of levels, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel;

[0062] The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the first lateral integral image, according to the actual blur kernel radius and the preset blur kernel shape, and obtain the weighted blurred image corresponding to the weight matrix.

[0063] In one embodiment, the blurred image acquisition module includes:

[0064] An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the weighted image according to the depth data of the input image;

[0065] An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel;

[0066] The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the second lateral integral image according to the actual blur kernel radius and the preset blur kernel shape, so as to obtain a weighted blurred image corresponding to the weighted image.

[0067] Optionally, the fuzzy value acquisition submodule includes:

[0068] an offset acquisition unit, configured to acquire, based on a preset correspondence between a blur kernel offset and a blur kernel radius in the blur kernel shape, a first offset and a second offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the correspondence;

[0069] an integral value acquisition unit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a first lateral integral value corresponding to the first offset in a lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when acquiring a weighted blurred image, the lateral integral map is a first lateral integral map; and when acquiring a weighted blurred image, the lateral integral map is a second lateral integral map;

[0070] A pixel sum acquisition unit, configured to acquire the sum of pixel values ​​of the corresponding row of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value;

[0071] The initial fuzzy value acquisition unit is used to accumulate the sum of the pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the initial fuzzy value of each pixel.

[0072] Optionally, the fuzzy value acquisition submodule further includes:

[0073] a first fuzzy value determining unit, configured to determine, when the fuzzy kernel is convex, that the initial fuzzy value of each pixel is the target fuzzy value;

[0074] The second fuzzy value determining unit is used to obtain a corrected fuzzy value corresponding to each pixel when the fuzzy kernel shape is a non-convex shape and use the corrected fuzzy value to correct the initial fuzzy value to obtain a target fuzzy value.

[0075] Optionally, the second fuzzy value determining unit includes:

[0076] an offset acquisition subunit, configured to acquire, based on a preset correction relationship between the blur kernel offset and the blur kernel radius within the blur kernel shape, a first correction offset and a second correction offset corresponding to each row of pixels within the blur kernel shape according to the actual blur kernel radius of each pixel and the correction relationship;

[0077] an integral value acquisition subunit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map;

[0078] a correction value acquisition subunit, configured to acquire the sum of correction pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value;

[0079] The fuzzy value acquisition subunit is used to accumulate the sum of the corrected pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the corrected fuzzy value of each pixel.

[0080] Optionally, the blurred image acquisition module further includes:

[0081] A first radius acquisition submodule, configured to, when a remainder in a calculation result is a non-zero value, subtract 1 from the actual blur kernel radius to obtain a first blur kernel radius, and to obtain a blur value corresponding to each pixel based on the first blur kernel radius;

[0082] The target fuzzy value acquisition submodule is used to obtain the weighted fuzzy value of each pixel based on the actual fuzzy kernel radius, the remainder, the fuzzy value corresponding to each pixel at the actual fuzzy kernel radius and the fuzzy value corresponding to each pixel at the first fuzzy kernel radius, and use the weighted fuzzy value as the target fuzzy value corresponding to each pixel.

[0083] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0084] processor;

[0085] a memory for storing a computer program executable by the processor;

[0086] The processor is configured to execute the computer program in the memory to implement the above method.

[0087] In one embodiment, the processor includes a digital signal processing DSP (Digital Signal Processor, DSP) chip; the DSP chip is used to parallelly process pixels with the same blur kernel radius in the same row of pixels within the blur kernel shape to achieve single instruction multiple data stream processing.

[0088] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the method as described above is implemented.

[0089] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0090] It can be seen from the above embodiments that the solution provided by the embodiments of the present disclosure obtains a lateral integral map. Since lateral integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel, when obtaining the sum of the pixel values ​​of a single row of pixels, it can be converted into the integral value of 2 pixels for addition and subtraction operations, which is less than 4 operations in the related art, which can save calculation time and achieve the effect of fast rendering of images. Moreover, after obtaining the sum of the pixel values ​​of a single row of pixels, the fuzzy value of the current pixel can be obtained by accumulating the sum of the pixel values ​​of each row; in other words, the length of each row of pixels is not limited in this embodiment, and pixels of different lengths can constitute a fuzzy kernel of any shape, that is, this embodiment can be applied to fuzzy kernels of any shape, which can improve the scope of application.

[0091] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0093] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0094] Figure 2 The figure is a flow chart of obtaining a weight matrix according to an exemplary embodiment.

[0095] Figure 3 is a schematic diagram of lateral integration according to an exemplary embodiment.

[0096] Figure 4 The figure is a flow chart showing a method of obtaining a fuzzy value according to an exemplary embodiment.

[0097] Figure 5 The figure is another flow chart of obtaining a fuzzy value according to an exemplary embodiment.

[0098] Figure 6 is a schematic diagram showing that the blur kernel shape is circular according to an exemplary embodiment.

[0099] Figure 7a is a schematic diagram showing that the blur kernel is in a square shape according to an exemplary embodiment.

[0100] Figure 7b is a schematic diagram showing that the blur kernel is in a hexagonal shape according to an exemplary embodiment.

[0101] Figure 7c is a schematic diagram showing that the blur kernel shape is a heart shape according to an exemplary embodiment.

[0102] Figure 7d is a schematic diagram showing that the blur kernel is in the shape of a pentagram according to an exemplary embodiment.

[0103] Figure 8a is a schematic diagram of a minimum circumscribed circle when the blur kernel is in the shape of a five-pointed star according to an exemplary embodiment.

[0104] Figure 8b is a schematic diagram of a shape to be subtracted when the blur kernel shape is a pentagram according to an exemplary embodiment.

[0105] Fig. 9 The figure is a flow chart of obtaining a corrected blur value according to an exemplary embodiment.

[0106] Fig.10a It is a schematic diagram showing the effect of not adopting fuzzy value smoothing according to an exemplary embodiment.

[0107] Fig.10b The figure is a schematic diagram showing the effect of using fuzzy value smoothing according to an exemplary embodiment.

[0108] Fig.11a It is a schematic diagram showing the effect of obtaining a target image when the blur kernel shape is circular according to an exemplary embodiment.

[0109] Fig.11b It is a schematic diagram showing the effect of obtaining a target image when the blur kernel shape is square according to an exemplary embodiment.

[0110] Fig.11c It is a schematic diagram showing the effect of obtaining a target image when the blur kernel shape is a hexagon according to an exemplary embodiment.

[0111] Fig.11d It is a schematic diagram showing the effect of obtaining a target image when the blur kernel shape is an onion ring shape according to an exemplary embodiment.

[0112] Fig.12 The figure is a schematic diagram showing the effect of obtaining a blur value by subtracting two adjacent pixels according to an exemplary embodiment.

[0113] Fig.13 The figure is a schematic diagram showing the effect of obtaining a blur value by subtracting two adjacent pixels according to an exemplary embodiment.

[0114] Fig.14 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0115] Fig.15 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0116] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described below by way of example do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0117] In order to solve the technical problems existing in the related art, the embodiments of the present disclosure provide an image processing method, which can be applied to electronic devices such as smart phones, personal computers, tablet computers, servers, etc. Figure 1is a flow chart of an image processing method shown in an embodiment of the present disclosure. Figure 1 , an image processing method comprises steps 11 to 15:

[0118] In step 11, depth data of an input image is obtained, and a weight matrix of the input image is obtained based on the depth data.

[0119] In this embodiment, a camera module may be provided in the electronic device, and the camera module may work in a preset working mode or a previous working mode after startup; at this time, the touch screen or control button in the electronic device may detect the user's trigger operation in real time or periodically, and send an operation signal to the processor when the user's trigger operation is detected. After receiving the operation signal, the processor may parse the operation signal and generate a corresponding control signal to send to the camera module. After receiving the control signal, the camera module may take pictures or record videos in response to the control signal, and send them to the processor or store them in a specified location (such as a local memory, a cloud, an external memory connected to the electronic device, etc.).

[0120] It should be noted that, considering that a video is composed of multiple frames of images, processing each frame of image can achieve the effect of video blurring. For the convenience of description, the subsequent embodiments are described by taking images as an example.

[0121] In this embodiment, the processor in the electronic device can directly receive the image sent by the camera module as the input image, or read the image from a specified position as the input image. After acquiring the input image, the processor can obtain the depth of field data of the input image. For example, the input image can be an image containing depth data, and the processor can obtain the configuration data corresponding to the input image (such as aperture, focal length, etc.), and the depth of field data, such as the depth of field range, position, etc., can be determined based on the above configuration data and depth data.

[0122] In this embodiment, the processor in the electronic device obtains the weight matrix of the input image according to the depth of field data. The weight matrix refers to a set of weights obtained by setting weights for each pixel in the input image, that is, the weight value of each pixel in the weight matrix corresponds to each pixel in the input image one by one, and the size of the weight matrix is ​​the same as the size of the input image.

[0123] In this embodiment, the electronic device can obtain the weight matrix of the input image, see Figure 2 In step 21, the processor may reset the weights of pixels within the depth of field range to zero and reset the weights of pixels outside the depth of field range to preset values ​​to obtain a weight matrix of the input image.

[0124] It should be noted that in this example, the weights of pixels within the depth of field range are reset to zero, so that the pixels within the depth of field range do not participate in the blur value calculation in subsequent steps, thereby preventing the colors within the depth of field range (which can be understood as the target area) from diffusing to outside the depth of field range (which can be understood as the background area).

[0125] It should be noted that the weights of pixels outside the depth of field range are reset to a preset value, which is a non-zero value, and the weights of pixels outside the depth of field range are the same value, thereby ensuring that the subsequent blur value calculation step does not affect each pixel.

[0126] In practical applications, considering that there will be light spots outside the depth of field range, if they are all set to the same value, the blur value of the pixel at the light spot will be reduced in the subsequent blur value calculation step. To this end, the light spot can also be brightened in this example. For example, the processor can input the input image into a preset image recognition model (such as a convolutional neural network, etc.), and the image recognition model determines the position of the light spot in the input image, and then increases the weight of each pixel at the light spot position. For another example, the processor can obtain a preset pixel value threshold (such as 235, which can be adjusted), and compare the pixel value of the pixel outside the depth of field range with the preset pixel value threshold; when the pixel value is less than the pixel value threshold, the weight of the corresponding pixel can be reset to a preset first value (such as 10, which can be adjusted); when the pixel value exceeds the pixel value threshold, the weight of the corresponding pixel can be reset to a preset second value (such as 180, which can be adjusted). The first value is less than the second value.

[0127] In practical applications, the weight Alhpa = a*scale1. When the pixel value is less than the pixel value threshold, a takes a value of 10 and scale1 takes a value of 1. When the pixel value exceeds the pixel value threshold, a takes a value of 10 and scale1 takes a value of 18. In other words, when the value of a is fixed, the effect of brightening the light spot can be achieved by adjusting scale1.

[0128] After setting weights for each pixel in the input image, the weight matrix corresponding to the input image can be obtained. The weight matrix can also be regarded as a frame of image, in which the pixel value of each pixel is the weight value. For the convenience of description, the weight matrix will be regarded as a frame of image in the following.

[0129] In step 12, the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image is obtained in sequence to obtain a weighted image; the product is the pixel value of each pixel in the weighted image.

[0130] In this embodiment, the processor in the electronic device can sequentially obtain the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; the product is the pixel value of each pixel in the weighted image. The relationship between the weighted image, the weight matrix and the input image is expressed as:

[0131] I weight =I sharp Alpha;

[0132] Among them, I weight represents a weighted image, Alpha represents the weight matrix, I sharp represents the input image and represents the dot product operation.

[0133] In step 13, the weight matrix and the weighted image are respectively integrated horizontally to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel.

[0134] In this embodiment, the processor in the electronic device can obtain the weight matrix and the horizontal integral map of the weighted image respectively. The horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel. Figure 3 , assuming that the pixel value of each pixel in the original image is 1, then there are 7 pixels to the left of pixel I(x,y), plus itself as the current pixel, and the horizontal integral value can be calculated to be 8, Figure 3 The second row shows the case where the pixel value of each pixel is 1. Figure 3 The third row in FIG. 1 shows the case where the horizontal integrated value SAT(x, y)=8 of the eighth pixel I(x, y).

[0135] It should be noted that the horizontal integration is used instead of the vertical integration in this embodiment, which is beneficial to ensure the continuity of the memory traversal when the subsequent DSP chip processes the image, and is beneficial to saving processing time. It is understandable that without considering the DSP chip, the technician can use the vertical integration to implement the solution of the present disclosure; or, when adjusting the image traversal order, such as changing the previous row traversal to column traversal, the above-mentioned horizontal integration can be transformed into vertical integration, which can also achieve the effect of ensuring the continuity of the memory traversal. The technician can select a suitable solution according to the specific scenario, and the corresponding solution falls within the protection scope of the present disclosure.

[0136] In practical applications, considering that the subsequent calculation of the blur value of each pixel is to calculate the sum of the pixel values ​​of all pixels within the blur kernel shape, then for the pixels in the first few rows or columns of the weighted image or weight matrix, the corresponding blur kernel may include the edge area of ​​the image. There are no pixels in these edge areas and it will affect the calculation of the blur values ​​of the first few rows or columns of pixels. Therefore, in this embodiment, the processor can first fill the image. For example, the processor can fill the weight matrix and the weighted image respectively to obtain the filled weight matrix and weighted image. Among them, the number of rows or columns of filled pixels is related to the radius of the blur kernel. For example, if the maximum blur kernel radius corresponding to the first row of pixels in the weight matrix is ​​9 pixels, it is necessary to fill 9 rows of pixels before the first row of pixels. The pixel value of the filled pixel can be a preset value, or the pixel value of the first row of pixels can be copied.

[0137] Based on the above principle, the processor can obtain the lateral integral map corresponding to the filled weight matrix, which is subsequently referred to as the first lateral integral map for distinction; and obtain the lateral integral map corresponding to the filled weighted image, which is subsequently referred to as the second lateral integral map for distinction.

[0138] In step 14, based on the first horizontal integral image, the fuzzy value corresponding to each pixel in the weight matrix is ​​obtained to obtain a weighted blurred image corresponding to the weight matrix; and based on the second horizontal integral image, the fuzzy value corresponding to each pixel in the weighted image is obtained to obtain a weighted blurred image corresponding to the weighted image.

[0139] In this embodiment, the processor in the electronic device can obtain the weighted blurred image corresponding to the weight matrix, and obtain the weighted blurred image corresponding to the weighted image.

[0140] To obtain a weighted blurred image, see Figure 4 In step 41, the processor can obtain the initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image. The depth data and the initial blur kernel radius are positively correlated, that is, as the pixel depth increases, the corresponding initial blur kernel radius also increases. The technician can set the corresponding model of the depth data and the initial blur kernel radius according to the specific scene, and the corresponding model falls within the protection scope of the present disclosure.

[0141] In step 42, the processor can obtain a preset number of levels (such as 4), and then perform a modulo operation on the initial blur kernel radius with respect to the preset number of levels, and use the quotient in the operation result as the actual blur kernel radius corresponding to each pixel. The reason for performing the above operation in this step is that the input image is composed of discrete pixels, and the calculated initial blur kernel radius is an integer; in small-size images or images with a high downsampling ratio, there will be obvious blur stratification between different blur kernel radii, especially in high-frequency texture areas. The blur stratification phenomenon is more obvious. For this reason, in this step, the integer multiple blur radius is further subdivided into multiple levels, namely the preset number of levels mentioned above, and renumbered according to integers. A blur kernel radius with a fractional radius is implemented at these levels. After further subdivision, the radius value scale_radius after subdivision of the current pixel, then the actual blur kernel radius radius of the current pixel can be obtained:

[0142]

[0143] In the formula, scale represents the preset number of levels, which is 4 in one example; radius represents the actual blur kernel radius, which is scale_radius The result of the upward modulus operation of scale; res represents the remainder, which can be used for weighting in subsequent fuzzy value calculations.

[0144] In step 43, the processor can confirm the blur value corresponding to each pixel according to the first lateral integral map, the actual blur kernel radius and the preset blur kernel shape. The preset blur kernel shape can be pre-configured in the camera module, and can include but is not limited to circle, square, polygon, heart shape and pentagon, etc., and the user can configure it according to the specific scene. The model kernel shape can be used when calculating the blur value, and the shape of the light spot in the subsequent target image is the same or similar to the above blur kernel shape, which is conducive to improving the user experience.

[0145] Taking the weighted blurred image corresponding to the weighted image as an example, the processor can obtain the initial blur kernel radius corresponding to each pixel in the weighted image according to the depth of field data. Afterwards, the processor can perform a remainder operation on the initial blur kernel radius with respect to the preset number of grades. The quotient in the operation result is used as the actual blur kernel radius corresponding to each pixel; the remainder in the operation result is used as the weight of the subsequent smoothing step, and the smoothing step can be omitted when the remainder is zero, and the smoothing step is adopted when the remainder is not zero, thereby reducing the amount of calculation. Finally, the processor can confirm the blur value corresponding to each pixel according to the second lateral integral map, the actual blur kernel radius and the preset blur kernel shape. It can be understood that the method of obtaining the weighted blurred image is the same as the method of obtaining the weighted blurred image. For details, please refer to Figure 4 The content of the illustrated embodiment.

[0146] See also Figure 5 The processor determines the blur value corresponding to each pixel according to the first lateral integral image, the actual blur kernel radius and the preset blur kernel shape, including:

[0147] In step 51, the processor may obtain the first offset and the second offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the corresponding relationship based on the preset corresponding relationship between the blur kernel offset and the blur kernel radius in the blur kernel shape. The preset corresponding relationship between the blur kernel offset and the blur kernel radius is obtained as follows:

[0148] Different shapes of blur kernels correspond to different blur kernel offsets and blur kernel radius. The above correspondence includes the blur kernel offsets corresponding to different blur kernel radii. The blur kernel offset refers to the lateral offset of the leftmost and rightmost pixels in each row of pixels in the current blur kernel shape relative to the center pixel. Figure 6 , when the blur kernel shape is circular and the radius is 9 pixels, the relationship shown in Table 1 can be obtained

[0149] Table 1 Offset of circular blur kernel with radius 9

[0150] Line number 0 1 2 3 4 5 6 7 8 9 Left offset 9 9 9 8 8 7 6 5 6 0 Right offset 9 9 9 8 8 7 6 5 6 0

[0151] Since the circle is symmetrical from top to bottom, Table 1 only records the lower half of the circle, and the symmetrical part searches the same offset table. Taking row 0 as an example, the horizontal offset of the leftmost pixel ① in the current row from the center pixel 0 in the current row is 9, and the offset of the rightmost pixel is also 9. Figure 6 Pixel ① in the middle represents the pixel before (i.e., on the left side) the leftmost pixel, and pixel ② represents the rightmost pixel. In subsequent calculations, the lateral integral value corresponding to pixel ② is subtracted from the lateral integral value corresponding to pixel ① to obtain the sum of the pixel values ​​in the 0th row within the blur kernel, that is, the blur value of the current pixel 0.

[0152] It should be noted that when the blur kernel shape is other than a circle, such as the multiple shapes shown in FIG. 7 , Figure 7a , Figure 7b , Figure 7c and Figure 7d They are square, hexagon, heart and pentagon respectively. The corresponding relationship of the offset is the same as that of the circle, so it will not be repeated here. It can be understood that the numbers 0 to 9 marked on the pixels in the blur kernel shape of each figure of the present disclosure indicate that the pixel can be used as the center pixel of the row, and pixel ① and pixel ② respectively represent the pixel before the leftmost and rightmost pixels of the row. For details, please refer to Figure 6 The contents shown will not be repeated here.

[0153] It should also be noted that the blur kernel shape may include convex shapes, such as circles, squares, polygons, and heart shapes, etc.; it may also include non-convex shapes, such as pentagons, etc. For convex shapes, when calculating the blur value of each row of pixels in the blur kernel, it can be obtained by subtracting the lateral integral values ​​of the leftmost and rightmost pixels in the row of pixels, but for non-convex shapes, it cannot be obtained by subtraction. For this reason, when the blur kernel shape is a non-convex shape, the non-convex shape can be converted into a combination of a minimum circumscribed shape and a shape to be subtracted, wherein the minimum circumscribed shape satisfies the condition of "covering the blur kernel, and each row of pixels has only two endpoints". Taking the blur kernel shape as a pentagon as an example, it can be converted into the following: Figure 8a The minimum circumscribed shape shown (taking a circle as an example) and Figure 8b The combination of the shapes to be subtracted shown in FIG. 1 can obtain two convex shapes. It can be understood that the corresponding relationship and the correction relationship of the offset are established, or the corresponding relationship obtained through the above steps at this time refers to the corresponding relationship between the blur kernel offset and the blur kernel radius of the minimum circumscribed shape of the blur kernel shape, and the correction relationship between the blur kernel offset and the blur kernel radius of the shape to be subtracted is the correction relationship between the blur kernel offset and the blur kernel radius within the preset blur kernel shape. In this way, when calculating the blur value of each pixel, a blur value can be calculated according to the corresponding relationship, and then a blur value can be calculated according to the correction relationship. The target blur value of each pixel can be obtained by subtracting the two blur values.

[0154] Continue to see Figure 5 In step 52, the processor can obtain, for each row of pixels in the blurred shape, a first lateral integral value corresponding to the first offset in the lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when obtaining a weighted blurred image, the lateral integral map is the first lateral integral map; and when obtaining a weighted blurred image, the lateral integral map is the second lateral integral map.

[0155] In step 53, the processor may obtain the sum of the pixel values ​​of the corresponding row of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value. That is, in this step, the calculation of the sum of the pixel values ​​of a single row of pixels is converted into the calculation of the lateral integral values ​​of the two endpoints in the row of pixels, as shown in the following formula:

[0156] Blur bias (x,y)=SAT(x+bias,y+right_offset)-SAT(x+bias,y-left_offset-1);

[0157] In the formula, Blur bias(x, y) represents the sum of the pixel values ​​of a single row of pixels within the blur kernel shape (i.e., the blur value of a single row of pixels), SAT(x+bias, y+right_offset) represents the horizontal integral value corresponding to the rightmost pixel right_offset in the x+bias row of pixels, i.e., the second horizontal integral value; SAT(x+bias, y-left_offset-1) represents the horizontal integral value corresponding to the leftmost pixel y-left_offset-1 in the x+bias row of pixels, i.e., the first horizontal integral value.

[0158] In step 54, the processor may accumulate the sum of the pixel values ​​corresponding to each row of pixels in the blur kernel shape to obtain an initial blur value of each pixel, as shown in the following formula:

[0159]

[0160] It is understandable that when the width of the input image is M and N, the complexity of the above blur value calculation is M*N*radius*2, which is less than M*N*radius*radius in the related art, that is, the horizontal integral map can reduce the time complexity of the disclosed solution.

[0161] It should be noted that the solution in which the processor determines the blur value corresponding to each pixel according to the second lateral integral image, the actual blur kernel radius and the preset blur kernel shape is the same as Figure 5 The scheme shown is the same, please refer to Figure 5 The contents shown will not be repeated here.

[0162] Considering that the blur kernel shape includes a convex shape and a non-convex shape, in one embodiment, the processor may determine whether the blur kernel shape is a convex shape or a non-convex shape. When the blur kernel shape is determined to be a convex shape, no correction is required at this time, and the processor may determine that the initial blur value of each pixel is the target blur value;

[0163] When it is determined that the blur kernel shape is a non-convex shape, correction is required. The processor can obtain the corrected blur value corresponding to each pixel according to the above correction relationship, and use the corrected blur value to correct the above initial blur value, so as to obtain the target blur value. Fig. 9 , the correction blur value is obtained as follows:

[0164] In step 91, the processor may obtain a preset correction relationship between the blur kernel offset and the blur kernel radius, and then obtain a first correction offset and a second correction offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the above correction relationship. The specific scheme of step 91 is the same as that of step 51, and the content of step 51 may be referred to. In step 92, for each row of pixels in the blur kernel shape, the processor may respectively obtain a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map. The specific scheme of step 92 is the same as that of step 52, and the content of step 52 may be referred to. In step 93, the processor may obtain the sum of the correction pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value. The specific scheme of step 93 is the same as that of step 53, and the content of step 53 may be referred to. In step 94, the processor may accumulate the sum of the correction pixel values ​​corresponding to each row of pixels in the blur kernel shape to obtain the correction blur value of each pixel. The specific solution of step 94 is the same as that of step 54 , and the content of step 53 can be referred to.

[0165] In one embodiment, after obtaining the blur value corresponding to each pixel, the processor may also determine whether the remainder of the operation result in step 43 is zero. When the remainder is zero, the processor may use the blur value of each pixel as the final target blur value; when the remainder is a non-zero value, the processor may subtract 1 from the actual blur kernel radius to obtain the first blur kernel radius radius-1, and obtain the blur value corresponding to each pixel based on the first blur kernel radius radius-1. For a specific acquisition scheme, see Figure 5 Then, the processor can obtain the weighted blur value of each pixel based on the actual blur kernel radius, the remainder, the blur value corresponding to each pixel at the actual blur kernel radius, and the blur value corresponding to each pixel at the first blur kernel radius, and use the weighted blur value as the target blur value corresponding to each pixel, as shown in the following formula:

[0166] Blur(x,y)=(radius-res)*Blur radius (x,y)+res*Blur radius-1 (x,y);

[0167] In the formula, Blur radius (x, y) represents the blur value corresponding to the actual blur kernel radius. radius-1 (x,y) represents the blur value corresponding to the first blur kernel radius.

[0168] It is understandable that in this embodiment, a smooth transition can be achieved by weighting the blur values ​​of adjacent radii, thereby solving the problem that it is difficult to achieve a decimal blur kernel radius on a discrete image.

[0169] It should be noted that the above-mentioned smoothing operation needs to be performed on both the weight matrix and the weighted image.

[0170] In step 15, the ratio of the fuzzy value corresponding to each pixel in the weighted fuzzy image to the fuzzy value corresponding to each pixel in the weighted fuzzy image is obtained in sequence, and the ratio is used as the pixel value of each pixel in the target image; the target image is a blurred image after the input image is blurred. FIG10 shows a comparison of the effects before and after using an image processing method provided by the present disclosure, wherein Fig.10a There is a fuzzy stratification in Fig.10b No blur stratification occurs in the image. Figure 11 shows the effects of different blur kernel shapes on the target image, where: Fig.11a The corresponding blur kernel shape is circular. Fig.11b The corresponding blur kernel shape is square, Fig.11c The corresponding blur kernel shape is hexagonal, Fig.11d The corresponding blur kernel shape is an onion ring, and the corresponding shapes at the light spot are circle, square, hexagon and onion ring respectively.

[0171] In one embodiment, the above-mentioned image processing method can be implemented by using a DSP chip, that is, the DSP chip can process the blur values ​​between different pixels in parallel to improve the processing speed. Moreover, for pixels with the same blur kernel radius in the same row of pixels, it is equivalent to performing the same processing on adjacent pixels with the same offset, which conforms to the acceleration concept of single instruction multiple data (SIMD). Fig.12 As shown in the figure, the blurring of adjacent central pixels A and B can be obtained by subtracting the adjacent pixels A_left, B_left and A_right, B_left, and the pixel sliding window can be used to implement parallel processing of 32 pixels on the DSP. The final target image can be obtained by looping through the entire image. At this time, the computational complexity of image blurring is further reduced to M*N*radius*2 / 32. High-performance blur rendering is achieved.

[0172] The electronic device is a smartphone, and the image taken on the Qualcomm Snapdragon 865 platform is downsampled to an image with a resolution of 1500*2000. The scatter blur (scatter) and the gather blur (gather) after DSP acceleration proposed in the present disclosure are compared. The data are shown in Table 2 and Fig.13 shown.

[0173] Table 2 Comparison of time between scattered fuzzy and aggregated fuzzy

[0174] method Scatter Gather Difference Time (ms) 401.29 167.97 233.32

[0175] Based on the above-mentioned image processing method, the present disclosure also provides an image processing device, see Fig.14 ,include:

[0176] A depth of field data acquisition module 141 is used to acquire the depth of field data of an input image;

[0177] A weight matrix acquisition module 142, configured to acquire a weight matrix of the input image based on the depth of field data;

[0178] The weighted image acquisition module 143 is used to sequentially acquire the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; the product is the pixel value of each pixel in the weighted image;

[0179] The integral image acquisition module 144 is used to perform horizontal integration on the weight matrix and the weighted image respectively to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel;

[0180] A fuzzy image acquisition module 145 is used to acquire the fuzzy value corresponding to each pixel in the weight matrix based on the first horizontal integral image, and obtain the weighted fuzzy image corresponding to the weight matrix; and acquire the fuzzy value corresponding to each pixel in the weighted image based on the second horizontal integral image, and obtain the weighted fuzzy image corresponding to the weighted image;

[0181] The target image acquisition module 146 is used to acquire a target image corresponding to the input image based on the weighted blurred image and the weight blurred image; the target image is a blurred image after blurring the input image.

[0182] Optionally, the weight matrix acquisition module includes:

[0183] The weight matrix acquisition submodule is used to reset the weights of pixels within the depth of field range to zero and reset the weights of pixels outside the depth of field range to preset values ​​according to the depth of field data, so as to obtain the weight matrix of the input image.

[0184] Optionally, the weight matrix acquisition submodule includes:

[0185] A pixel value comparison unit, used for comparing the pixel value of the pixel outside the depth of field range with a preset pixel value threshold;

[0186] A weight setting unit is used to reset the weight of the corresponding pixel to a preset first value when the pixel value is less than the pixel value threshold; when the pixel value exceeds the pixel value threshold, reset the weight of the corresponding pixel to a preset second value; the first value is less than the second value.

[0187] Optionally, the integral image acquisition module includes:

[0188] A filling submodule, used to fill the weight matrix and the weighted image respectively to obtain the filled weight matrix and weighted image;

[0189] The integral submodule is used to obtain the sum of the pixel values ​​of the pixels before the current pixel in the row and the current pixel in turn for each pixel in the filled weight matrix and weighted image, and use the sum of the pixel values ​​as the horizontal integral value of the current pixel to obtain the first horizontal integral image corresponding to the weight matrix and the second horizontal integral image corresponding to the weighted image.

[0190] Optionally, the blurred image acquisition module includes:

[0191] An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image;

[0192] An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel;

[0193] The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the first lateral integral image, according to the actual blur kernel radius and the preset blur kernel shape, and obtain the weighted blurred image corresponding to the weight matrix.

[0194] Optionally, the blurred image acquisition module includes:

[0195] An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the weighted image according to the depth data of the input image;

[0196] An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel;

[0197] The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the second lateral integral image according to the actual blur kernel radius and the preset blur kernel shape, so as to obtain a weighted blurred image corresponding to the weighted image.

[0198] Optionally, the fuzzy value acquisition submodule includes:

[0199] an offset acquisition unit, configured to acquire, based on a preset correspondence relationship between a blur kernel offset and a blur kernel radius within the blur kernel shape, a first offset and a second offset corresponding to each row of pixels within the blur kernel shape according to an actual blur kernel radius of each pixel and the correspondence relationship;

[0200] an integral value acquisition unit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a first lateral integral value corresponding to the first offset in a lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when acquiring a weighted blurred image, the lateral integral map is a first lateral integral map; and when acquiring a weighted blurred image, the lateral integral map is a second lateral integral map;

[0201] A pixel sum acquisition unit, configured to acquire the sum of pixel values ​​of the corresponding row of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value;

[0202] The initial fuzzy value acquisition unit is used to accumulate the sum of the pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the initial fuzzy value of each pixel.

[0203] Optionally, the fuzzy value acquisition submodule further includes:

[0204] a first fuzzy value determining unit, configured to determine, when the fuzzy kernel is convex, that the initial fuzzy value of each pixel is the target fuzzy value;

[0205] The second fuzzy value determining unit is used to obtain a corrected fuzzy value corresponding to each pixel when the fuzzy kernel shape is a non-convex shape and use the corrected fuzzy value to correct the initial fuzzy value to obtain a target fuzzy value.

[0206] Optionally, the second fuzzy value determining unit includes:

[0207] an offset acquisition subunit, configured to acquire, based on a preset correction relationship between the blur kernel offset and the blur kernel radius within the blur kernel shape, a first correction offset and a second correction offset corresponding to each row of pixels within the blur kernel shape according to an actual blur kernel radius of each pixel and the correction relationship;

[0208] an integral value acquisition subunit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map;

[0209] a correction value acquisition subunit, configured to acquire the sum of correction pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value;

[0210] The fuzzy value acquisition subunit is used to accumulate the sum of the corrected pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the corrected fuzzy value of each pixel.

[0211] Optionally, the blurred image acquisition module further includes:

[0212] A first radius acquisition submodule, configured to, when a remainder in a calculation result is a non-zero value, subtract 1 from the actual blur kernel radius to obtain a first blur kernel radius, and to obtain a blur value corresponding to each pixel based on the first blur kernel radius;

[0213] The target fuzzy value acquisition submodule is used to obtain the weighted fuzzy value of each pixel based on the actual fuzzy kernel radius, the remainder, the fuzzy value corresponding to each pixel at the actual fuzzy kernel radius and the fuzzy value corresponding to each pixel at the first fuzzy kernel radius, and use the weighted fuzzy value as the target fuzzy value corresponding to each pixel.

[0214] At this point, the solution provided by the embodiment of the present disclosure can obtain the horizontal integral map, and when obtaining the sum of the pixel values ​​of a single row of pixels, the integral values ​​of two pixels can be used for addition and subtraction operations, which is less than the four operations in the related art, and can save calculation time. In addition, after obtaining the sum of the pixel values ​​of a single row of pixels, the fuzzy value of the current pixel can be obtained by accumulating the sum of the pixel values ​​of each row; in other words, the length of each row of pixels is not limited in this embodiment, and pixels of different lengths can form a fuzzy kernel of any shape, that is, this embodiment can be applied to fuzzy kernels of any shape, which can increase the scope of application.

[0215] It is understandable that the device provided by the embodiment of the present disclosure is different from the above Figure 1 The method shown corresponds to the above, and the specific content can be referred to the content of each embodiment of the method, which will not be repeated here.

[0216] Fig.15 1 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 1500 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0217] Reference Fig.15 , the electronic device 1500 may include one or more of the following components: a processing component 1502, a memory 1504, a power component 1506, a multimedia component 1508, an audio component 1510, an input / output (I / O) interface 1512, a sensor component 1514, a communication component 1516, and an image acquisition component 1518.

[0218] The processing component 1502 generally controls the overall operation of the electronic device 1500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1502 may include one or more processors 1520 to execute computer programs. In addition, the processing component 1502 may include one or more modules to facilitate interaction between the processing component 1502 and other components. For example, the processing component 1502 may include a multimedia module to facilitate interaction between the multimedia component 1508 and the processing component 1502.

[0219] The memory 1504 is configured to store various types of data to support operations on the electronic device 1500. Examples of such data include computer programs for any application or method operating on the electronic device 1500, contact data, phone book data, messages, pictures, videos, etc. The memory 1504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0220] The power supply component 1506 provides power to various components of the electronic device 1500. The power supply component 1506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1500. The power supply component 1506 may include a power chip, and the controller may communicate with the power chip to control the power chip to turn on or off a switch device, so that the battery supplies power to the mainboard circuit or not.

[0221] The multimedia component 1508 includes a screen that provides an output interface between the electronic device 1500 and the target object. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the target object. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0222] The audio component 1510 is configured to output and / or input audio signals. For example, the audio component 1510 includes a microphone (MIC), and when the electronic device 1500 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1504 or sent via the communication component 1516. In some embodiments, the audio component 1510 also includes a speaker for outputting audio signals.

[0223] The I / O interface 1512 provides an interface between the processing component 1502 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.

[0224] The sensor component 1514 includes one or more sensors for providing various aspects of status assessment for the electronic device 1500. For example, the sensor component 1514 can detect the open / closed state of the electronic device 1500, the relative positioning of components, such as the display screen and keypad of the electronic device 1500, and the sensor component 1514 can also detect the position change of the electronic device 1500 or a component, the presence or absence of contact between the target object and the electronic device 1500, the orientation or acceleration / deceleration of the electronic device 1500, and the temperature change of the electronic device 1500. In this example, the sensor component 1514 may include a magnetic sensor, a gyroscope, and a magnetic field sensor, wherein the magnetic field sensor includes at least one of the following: a Hall sensor, a thin film magnetoresistive sensor, and a magnetic liquid acceleration sensor.

[0225] The communication component 1516 is configured to facilitate wired or wireless communication between the electronic device 1500 and other devices. The electronic device 1500 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 1516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0226] In an exemplary embodiment, the electronic device 1500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components. In one embodiment, the processor includes a digital signal processor DSP; the DSP is used to parallelly process pixels of the same blur kernel radius in the same row of pixels within the blur kernel shape to implement single instruction multiple data stream processing.

[0227] In an exemplary embodiment, a non-transitory readable storage medium including an executable computer program is also provided, such as a memory 1504 including instructions, and the executable computer program can be executed by a processor. The readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0228] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0229] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire depth data of an input image, and acquire a weight matrix of the input image based on the depth data; Sequentially obtain the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; The product is the pixel value of each pixel in the weighted image; Performing horizontal integration on the weight matrix and the weighted image respectively to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel; Based on the first lateral integral image, obtaining a fuzzy value corresponding to each pixel in the weight matrix to obtain a weighted fuzzy image corresponding to the weight matrix; and obtaining, based on the second lateral integral image, a fuzzy value corresponding to each pixel in the weighted image to obtain a weighted fuzzy image corresponding to the weighted image; Sequentially obtain the ratio of the fuzzy value corresponding to each pixel in the weighted fuzzy image to the fuzzy value corresponding to each pixel in the weighted fuzzy image, and use the ratio as the pixel value of each pixel in the target image; The target image is a blurred image obtained by blurring the input image.

2. The method according to claim 1, characterized in that: Acquiring a weight matrix of the input image based on the depth of field data includes: The weights of pixels within a depth range are reset to zero and the weights of pixels outside the depth range are reset to preset values ​​according to the depth data, so as to obtain the weight matrix of the input image.

3. The method according to claim 2, characterized in that Resetting the weights of pixels outside the depth of field range to preset values ​​includes: Comparing pixel values ​​of pixels outside the depth of field range with a preset pixel value threshold; When the pixel value is less than the pixel value threshold, the weight of the corresponding pixel is reset to a preset first value; when the pixel value exceeds the pixel value threshold, the weight of the corresponding pixel is reset to a preset second value; the first value is less than the second value.

4. The method according to claim 1, characterized in that: Performing a transverse integration on the weight matrix to obtain a first transverse integral graph corresponding to the weight matrix includes: Filling the weight matrix and the weighted image respectively to obtain the filled weight matrix and the weighted image; For each pixel in the filled weight matrix and the weighted image, the sum of the pixel values ​​of the pixel before the current pixel in the row and the current pixel are obtained in turn, and the sum of the pixel values ​​is used as the horizontal integral value of the current pixel, so as to obtain the first horizontal integral image corresponding to the weight matrix and the second horizontal integral image corresponding to the weighted image respectively.

5. The method according to claim 1, characterized in that Based on the first lateral integral image, obtaining a fuzzy value corresponding to each pixel in the weight matrix includes: Acquire an initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image; Performing a modulo operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient of the operation result as the actual blur kernel radius corresponding to each pixel; Based on the first lateral integral image, the blur value corresponding to each pixel is determined according to the actual blur kernel radius and the preset blur kernel shape, and a weighted blurred image corresponding to the weight matrix is ​​obtained.

6. The method according to claim 1, characterized in that Based on the second transverse integral image, obtaining a fuzzy value corresponding to each pixel in the weighted image includes: Acquire an initial blur kernel radius corresponding to each pixel in the weighted image according to the depth data of the input image; Performing a modulo operation on the initial blur kernel radius and a preset number of levels, and taking the quotient of the operation result as the actual blur kernel radius corresponding to each pixel; Based on the second lateral integral image, a blur value corresponding to each pixel is determined according to the actual blur kernel radius and a preset blur kernel shape, so as to obtain a weighted blurred image corresponding to the weighted image.

7. The method according to claim 5 or 6, characterized in that: Determining the blur value corresponding to each pixel according to the actual blur kernel radius and the preset blur kernel shape includes: Based on the preset correspondence between the blur kernel offset and the blur kernel radius in the blur kernel shape, obtaining the first offset and the second offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the correspondence; For each row of pixels in the blur kernel shape, respectively obtaining a first lateral integral value corresponding to the first offset in the lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when obtaining a weighted blurred image, the lateral integral map is a first lateral integral map; and when obtaining a weighted blurred image, the lateral integral map is a second lateral integral map; Obtaining the sum of pixel values ​​of corresponding rows of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value; The sum of the pixel values ​​corresponding to each row of pixels in the blur kernel shape is accumulated to obtain an initial blur value of each pixel.

8. The method according to claim 7, characterized in that After obtaining the initial blur value of each pixel, the method further includes: If the blur kernel shape is a convex shape, then determining the initial blur value of each pixel is the target blur value; If the blur kernel shape is a non-convex shape, a corrected blur value corresponding to each pixel is obtained and the initial blur value is corrected by using the corrected blur value to obtain a target blur value.

9. The method according to claim 8, characterized in that Get the corrected blur value corresponding to each pixel, including: Based on the preset correction relationship between the blur kernel offset and the blur kernel radius within the blur kernel shape, obtaining a first correction offset and a second correction offset corresponding to each row of pixels within the blur kernel shape according to the actual blur kernel radius of each pixel and the correction relationship; For each row of pixels in the blur kernel shape, respectively obtaining a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map; Acquire the sum of the corrected pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value; The sum of the corrected pixel values ​​corresponding to each row of pixels in the blur kernel shape is accumulated to obtain the corrected blur value of each pixel.

10. The method according to claim 5 or 6, characterized in that: After determining the blur value corresponding to each pixel according to the actual blur kernel radius and the preset blur kernel shape, the method further includes: When the remainder in the operation result is a non-zero value, the actual blur kernel radius is subtracted by 1 to obtain a first blur kernel radius, and a blur value corresponding to each pixel is obtained based on the first blur kernel radius; A weighted blur value of each pixel is obtained based on the actual blur kernel radius, the remainder, the blur value of each pixel corresponding to the actual blur kernel radius, and the blur value of each pixel corresponding to the first blur kernel radius, and the weighted blur value is used as the target blur value corresponding to each pixel.

11. An image processing device, characterized in that: include: A depth of field data acquisition module, used to acquire the depth of field data of an input image; A weight matrix acquisition module, used to acquire a weight matrix of the input image based on the depth of field data; A weighted image acquisition module, used to sequentially acquire the product of the weight corresponding to each pixel in the weight matrix and the pixel value corresponding to each pixel in the input image to obtain a weighted image; The product is the pixel value of each pixel in the weighted image; An integral image acquisition module is used to perform horizontal integration on the weight matrix and the weighted image respectively to obtain a first horizontal integral image corresponding to the weight matrix and a second horizontal integral image corresponding to the weighted image; the horizontal integration refers to sequentially obtaining the sum of the pixel values ​​of the previous pixel in the row and the current pixel of each pixel; A fuzzy image acquisition module, configured to acquire, based on the first lateral integral image, a fuzzy value corresponding to each pixel in the weight matrix, and obtain a weighted fuzzy image corresponding to the weight matrix; and to acquire, based on the second lateral integral image, a fuzzy value corresponding to each pixel in the weighted image, and obtain a weighted fuzzy image corresponding to the weighted image; The target image acquisition module is used to acquire the target image corresponding to the input image based on the weighted blurred image and the weight blurred image; the target image is a blurred image after the input image is blurred.

12. The device according to claim 11, characterized in that The weight matrix acquisition module includes: The weight matrix acquisition submodule is used to reset the weights of pixels within the depth of field range to zero and reset the weights of pixels outside the depth of field range to preset values ​​according to the depth of field data, so as to obtain the weight matrix of the input image.

13. The device according to claim 12, characterized in that The weight matrix acquisition submodule includes: A pixel value comparison unit, used for comparing the pixel value of the pixel outside the depth of field range with a preset pixel value threshold; A weight setting unit is used to reset the weight of the corresponding pixel to a preset first value when the pixel value is less than the pixel value threshold; when the pixel value exceeds the pixel value threshold, reset the weight of the corresponding pixel to a preset second value; the first value is less than the second value.

14. The device according to claim 11, characterized in that The integral graph acquisition module comprises: A filling submodule, used to fill the weight matrix and the weighted image respectively to obtain the filled weight matrix and the weighted image; The integral submodule is used to obtain the sum of the pixel values ​​of the pixels before the current pixel in the row and the current pixel in turn for each pixel in the filled weight matrix and the weighted image, and use the sum of the pixel values ​​as the horizontal integral value of the current pixel to obtain the first horizontal integral image corresponding to the weight matrix and the second horizontal integral image corresponding to the weighted image.

15. The device according to claim 11, characterized in that The blurred image acquisition module comprises: An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the input image according to the depth data of the input image; An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel; The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the first lateral integral image, according to the actual blur kernel radius and the preset blur kernel shape, and obtain the weighted blurred image corresponding to the weight matrix.

16. The device according to claim 11, characterized in that The blurred image acquisition module comprises: An initial radius acquisition submodule, used to acquire an initial blur kernel radius corresponding to each pixel in the weighted image according to the depth data of the input image; An actual radius acquisition submodule, used for performing a remainder operation on the initial blur kernel radius and a preset number of classifications, and taking the quotient in the operation result as the actual blur kernel radius corresponding to each pixel; The fuzzy value acquisition submodule is used to determine the fuzzy value corresponding to each pixel based on the second lateral integral image according to the actual blur kernel radius and the preset blur kernel shape, so as to obtain a weighted blurred image corresponding to the weighted image.

17. The device according to claim 15 or 16, characterized in that The fuzzy value acquisition submodule includes: an offset acquisition unit, configured to acquire, based on a preset correspondence between a blur kernel offset and a blur kernel radius in the blur kernel shape, a first offset and a second offset corresponding to each row of pixels in the blur kernel shape according to the actual blur kernel radius of each pixel and the correspondence; an integral value acquisition unit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a first lateral integral value corresponding to the first offset in a lateral integral map and a second lateral integral value corresponding to the second offset in the lateral integral map; when acquiring a weighted blurred image, the lateral integral map is a first lateral integral map; and when acquiring a weighted blurred image, the lateral integral map is a second lateral integral map; A pixel sum acquisition unit, configured to acquire the sum of pixel values ​​of the corresponding row of pixels in the blur kernel shape according to the first lateral integral value and the second lateral integral value; The initial fuzzy value acquisition unit is used to accumulate the sum of the pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the initial fuzzy value of each pixel.

18. The device according to claim 17, characterized in that The fuzzy value acquisition submodule also includes: a first fuzzy value determining unit, configured to determine, when the fuzzy kernel is convex, that the initial fuzzy value of each pixel is the target fuzzy value; The second fuzzy value determining unit is used to obtain a corrected fuzzy value corresponding to each pixel when the fuzzy kernel shape is a non-convex shape and use the corrected fuzzy value to correct the initial fuzzy value to obtain a target fuzzy value.

19. The device according to claim 18, characterized in that The second fuzzy value determining unit comprises: an offset acquisition subunit, configured to acquire, based on a preset correction relationship between the blur kernel offset and the blur kernel radius within the blur kernel shape, a first correction offset and a second correction offset corresponding to each row of pixels within the blur kernel shape according to the actual blur kernel radius of each pixel and the correction relationship; an integral value acquisition subunit, configured to respectively acquire, for each row of pixels within the blur kernel shape, a third lateral integral value corresponding to the first correction offset in the first lateral integral map and a fourth lateral integral value corresponding to the second correction offset in the first lateral integral map; a correction value acquisition subunit, configured to acquire the sum of correction pixel values ​​corresponding to each row of pixels in the blur kernel shape according to the third lateral integral value and the fourth lateral integral value; The fuzzy value acquisition subunit is used to accumulate the sum of the corrected pixel values ​​corresponding to each row of pixels in the fuzzy kernel shape to obtain the corrected fuzzy value of each pixel.

20. The device according to claim 15 or 16, characterized in that The blurred image acquisition module also includes: A first radius acquisition submodule, configured to, when a remainder in a calculation result is a non-zero value, subtract 1 from the actual blur kernel radius to obtain a first blur kernel radius, and to obtain a blur value corresponding to each pixel based on the first blur kernel radius; The target fuzzy value acquisition submodule is used to obtain the weighted fuzzy value of each pixel based on the actual fuzzy kernel radius, the remainder, the fuzzy value corresponding to each pixel at the actual fuzzy kernel radius and the fuzzy value corresponding to each pixel at the first fuzzy kernel radius, and use the weighted fuzzy value as the target fuzzy value corresponding to each pixel.

21. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 10.

22. The electronic device according to claim 21, characterized in that: The processor comprises a digital signal processing DSP chip; the DSP chip is used to perform parallel processing on pixels with the same blur kernel radius in the same row of pixels within the blur kernel shape, so as to realize single instruction multiple data stream processing.

23. A computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 10 can be implemented.

Images