Image processing method and device, electronic equipment and computer readable storage medium
The image processing method reduces storage space by using non-zero weight coordinates and a compressed weight matrix to simulate the Bokeh effect, addressing the storage requirements of software-based simulation.
Patent Information
- Application Number
- CN202310662779.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
Existing electronic devices use software methods to simulate the Bokeh effect, which require large storage space for filter matrices.
An image processing method that stores only non-zero weight coordinates and weights, using a compressed weight matrix to reduce storage space by determining the weight matrix based on these coordinates and weights, and generating a second image with the same blurring effect as the uncompressed matrix.
Reduces storage space and hardware costs while maintaining the blurring quality, enabling efficient simulation of the Bokeh effect.
Smart Images

Figure CN119090786A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, an image processing device, an electronic device, and a non-volatile computer-readable storage medium. Background Art
[0002] At present, electronic devices can achieve various imaging effects to enrich the artistic creation of users. For example, the Bokeh effect during shooting is that the area within the depth of field is clear, and the area outside the depth of field is blurred, and there will be a gradually blurred image effect.
[0003] Achieving the Bokeh effect often requires expensive lenses. To reduce costs, electronic devices usually use software methods to filter images using a filter matrix to simulate the Bokeh effect. However, using software methods to simulate the Bokeh effect often requires a large amount of storage space to store the corresponding filter matrix. Summary of the invention
[0004] Embodiments of the present application provide an image processing method, an image processing device, an electronic device, and a non-volatile computer-readable storage medium.
[0005] The image processing method of the embodiment of the present application includes: obtaining an image to be processed and a blur scale; obtaining pre-stored weight coordinates and weights corresponding to the weight coordinates according to the blur scale, wherein the weights corresponding to the weight coordinates are not zero; determining a weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates; calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generating a first image according to the convolution; and calculating the cumulative sum of pixel values of pixels of the first image in sequence along the preset direction, and generating a second image according to the cumulative sum of pixel values.
[0006] The image processing device of the embodiment of the present application includes: an input module, a selection module, a matrix acquisition module, a first processing module and a second processing module. The input module is used to obtain the image to be processed and the blur scale. The selection module is used to obtain the pre-stored weight coordinates and the weights corresponding to the weight coordinates according to the blur scale, and the weights corresponding to the weight coordinates are not zero. The matrix acquisition module is used to determine the weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates. The first processing module is used to calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generate a first image according to the convolution. The second processing module is used to calculate the cumulative sum of pixel values for the pixels of the first image in sequence along the preset direction, and generate a second image according to the cumulative sum of pixel values.
[0007] The electronic device of the embodiment of the present application includes a shell and a processor, the processor is accommodated in the shell, and the processor is used to execute an image processing method, the image processing method includes: obtaining an image to be processed and a blur scale; obtaining pre-stored weight coordinates and weights corresponding to the weight coordinates according to the blur scale, and the weights corresponding to the weight coordinates are not zero; determining a weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates; calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generating a first image according to the convolution; and calculating the cumulative sum of pixel values for pixels of the first image in sequence along the preset direction, and generating a second image according to the cumulative sum of pixel values.
[0008] The non-volatile computer-readable storage medium containing a computer program of an embodiment of the present application includes one or more processors. When the computer program is executed by the one or more processors, the processors implement instructions of an image processing method, and the image processing method includes: obtaining an image to be processed and a blur scale; obtaining pre-stored weight coordinates and weights corresponding to the weight coordinates according to the blur scale, and the weights corresponding to the weight coordinates are not zero; determining a weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates; calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generating a first image according to the convolution; and calculating the cumulative sum of pixel values for pixels of the first image in sequence along the preset direction, and generating a second image according to the cumulative sum of pixel values.
[0009] The image processing method, image processing device, electronic device and non-volatile computer-readable storage medium of the embodiments of the present application only store weight coordinates whose weights are not zero. When blur filtering is required, the weight matrix used for filtering is determined based on the weight coordinates whose weights are not zero and the corresponding weights, so as to reduce the storage space occupied.
[0010] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0012] Figure 1 is a flowchart of an image processing method of certain embodiments of the present application;
[0013] Figure 2 It is a schematic diagram of an application scenario of a weight matrix of certain embodiments of the present application;
[0014] Figure 3 It is a schematic diagram of an application scenario of filtering processing in some embodiments of the present application;
[0015] Figure 4 It is a schematic diagram of an application scenario of filtering processing in some embodiments of the present application;
[0016] Figure 5 It is a schematic diagram of an application scenario of filtering processing in some embodiments of the present application;
[0017] Figure 6 is a schematic diagram of the structure of an image processing device in some embodiments of the present application;
[0018] Figure 7 is a schematic diagram of the structure of an electronic device according to some embodiments of the present application;
[0019] Figure 8 is a flowchart of an image processing method of certain embodiments of the present application;
[0020] Fig. 9 It is a schematic diagram of a scenario in which an electronic device according to some embodiments of the present application applies an image processing method;
[0021] Fig.10 is a flowchart of an image processing method of certain embodiments of the present application;
[0022] Fig.11 It is a schematic diagram of an application scenario of a diffusion algorithm in certain embodiments of the present application;
[0023] Fig.12 It is a schematic diagram of an application scenario of constructing a weight matrix in certain embodiments of the present application;
[0024] Fig.13 is a schematic diagram of an application scenario of obtaining a second image in certain embodiments of the present application;
[0025] Fig.14 is a flowchart of an image processing method of certain embodiments of the present application;
[0026] Fig.15 It is a schematic diagram of an application scenario of adjusting weight interpolation in certain embodiments of the present application;
[0027] Fig.16 is a flowchart of an image processing method of certain embodiments of the present application;
[0028] Fig.17 It is a schematic diagram of an application scenario of adjusting weight coordinate weights in certain implementation modes of the present application;
[0029] Fig.18is a flowchart of an image processing method of certain embodiments of the present application;
[0030] Fig.19 is a flowchart of an image processing method of certain embodiments of the present application;
[0031] Fig. 20 It is a schematic diagram of an application scenario of determining a weight matrix in certain embodiments of the present application;
[0032] Fig.21 is a flowchart of an image processing method of certain embodiments of the present application;
[0033] Fig. 22 It is a schematic diagram of an application scenario of determining a weight matrix in certain embodiments of the present application;
[0034] Fig.23 It is a schematic diagram of an application scenario of determining a weight matrix in certain embodiments of the present application;
[0035] Fig.24 is a flowchart of an image processing method of certain embodiments of the present application;
[0036] Fig.25 It is a schematic diagram of an application scenario of determining a weight matrix in certain embodiments of the present application;
[0037] Fig.26 It is a schematic diagram of the structure of a computer-readable storage medium of certain embodiments of the present application. DETAILED DESCRIPTION
[0038] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.
[0039] See also Figure 1 The present application provides an image processing method for saving storage space required for blurring. The image processing method comprises the following steps:
[0040] 01: Get the image to be processed and the blur scale;
[0041] 02: Obtain the pre-stored weight coordinates and the weights corresponding to the weight coordinates according to the blur scale, and the weights corresponding to the weight coordinates are not zero;
[0042] 03: Determine the weight matrix according to the blur scale, weight coordinates and the weights corresponding to the weight coordinates;
[0043] 04: sequentially calculating the convolution of the weight of each weight unit in the weight matrix and the pixel value of each pixel in the image to be processed along a preset direction, and generating a first image according to the convolution; and
[0044] 05: Calculate the cumulative sum of pixel values of pixels of the first image in sequence along a preset direction, and generate a second image according to the cumulative sum of pixel values.
[0045] The image to be processed in step 01 is an image to be blurred. It should be noted that in some application scenarios, the video needs to be blurred, and any frame image in the video can be used as the image to be processed in step 01 for blurring. For example, the image processing method of the present application can be used to batch blur multiple frames of images in a video to render a blur effect in the video.
[0046] The blur scale in step 01 is an evaluation of the degree of blur. For example, the blur scale is divided into three scales: high, medium, and low. The blur scales correspond to changes from more blurry to clearer from high to low, respectively, and can be selected according to the user's blur strategy. In some embodiments, the blur scale is positively correlated with the depth of field. In the blur effect achieved by the lens, the greater the depth of field and the farther the focus point, the higher the degree of blur. Therefore, the blur scale can be determined according to the depth of field of the image to be processed, and the blur strategy can be intelligently formulated. After determining the blur scale, the weight matrix used for blur filtering can be further determined according to steps 02 and 03.
[0047] Please combine Figure 2 The weight matrix in step 03 refers to the weight set obtained by setting weights for each pixel in the image to be processed. The weight matrix is used to filter the pixel values of the image to be processed to render a blur effect. Figure 2 In the example, the weight matrix includes nine weight units, each weight unit has a corresponding weight, and the set of weight units is the weight matrix. The weight coordinates in step 02 are the coordinates of the weight unit in the weight matrix. For example, the weight coordinates of the weight unit in the 1st row and 1st column are (1, 1). The weight corresponding to the weight coordinate refers to the weight of the weight unit located at the weight coordinate. For example, the weight corresponding to the weight coordinate (1, 1) refers to the weight of the weight unit in the 1st row and 1st column.
[0048] In some embodiments, the image to be processed can be input into a trained convolutional neural network model for blurring, and the weight matrix is the convolution kernel used to filter the image to be processed. Using the convolutional neural network model to perform matrix calculations on the weight matrix and the pixels of the image to be processed is only an example of an implementation method, and any matrix calculation method can be applied to the image processing method of this application.
[0049] See also Figure 1 and Figure 2, the process of calculating the convolution in sequence along the preset direction in step 04 is the filtering process. Among them, the convolution of the weight matrix and the pixel value specifically refers to the value obtained by multiplying the weight of each weight unit in the weight matrix with the pixel value of the corresponding pixel within the overlapping range of the image to be processed and the weight matrix and adding them together. Taking the blurring process using a convolutional neural network as an example, assuming that the weight matrix Z0 is a 3*3 matrix, including nine weight units with weights Q1, Q2, Q3, Q4, Q5, Q6, Q7, Q8, and Q9, P5 is the pixel value of a certain coordinate point (x5, y5) in the image to be processed, and the pixel values of its adjacent coordinate points within the size of the weight matrix (that is, the adjacent coordinate points within the range of the 3*3 matrix) are P1, P2, P3, P4, P6, P7, P8, and P9 respectively. After filtering the image to be processed using the weight matrix, the pixel value of the coordinate point (x5, y5) in the generated first image is:
[0050] Q1*P1+Q2*P2+Q3*P3+Q4*P4+Q5*P5+Q6*P6+Q7*P7+Q8*P8+Q9*P9. Similarly, the first image can be generated according to the pixel values of each pixel in the image to be processed after filtering.
[0051] The preset direction may be along the row direction, the column direction, or a combination of row and column directions. The combination of row and column directions may be along the row direction first and then along the column direction; or along the column direction first and then along the row direction; or along the row direction first and then along the column direction and then along the row direction again, which is not limited here.
[0052] See also Figure 3 , Figure 3 This is an application scenario where a certain weight matrix Z1 is used to filter the processed image D1. Figure 3 In the example, the image to be processed includes a pixel matrix of 5 rows and 15 columns, in which the pixel values of the pixels in the 3rd row and 8th column are 1, and the remaining pixel values are 0. The weight matrix Z1 is a matrix of 5 rows and 9 columns, in which the weights of the weight units in the rectangular range S1 formed by the four corner points of the 2nd row and 2nd column, the 2nd row and 8th column, the 4th row and 2nd column, and the 4th row and 8th column are all Q0, and the weights of the weight units outside the range S1 are all 0. The weight matrix Z1 is used to filter the image to be processed and the following is obtained: Figure 3 In the output image T0 shown in the figure, the pixel value in the rectangular range S2 formed by the four corner points of row 2, column 5, row 2, column 11, row 4, column 5, and row 4, column 11 is Q0, and the rest of the pixel values are 0. If each weight unit needs to occupy 8 bits of storage space, then the weight matrix Z1 needs to occupy 5*9*8 bits of storage space.
[0053] See also Figure 3 and Figure 4In order to save storage space, the image processing method of the present application does not store the complete weight matrix, but only stores the coordinates and weights of the weight units with non-zero weights in the weight matrix. In this way, since the number of weight coordinates with non-zero weights is less than the number of weight units in the weight matrix, the method of pre-storing weight coordinates and weights corresponding to the weight coordinates can reduce the storage space compared to storing the complete weight matrix. When blurring, step 03 is executed again to determine the weight matrix of the current filtering according to the blurring scale, weight coordinates and weights corresponding to the weight coordinates.
[0054] Furthermore, in order to further save storage space, in the image processing method of the present application, the weight matrix used in step 03 is a compressed weight matrix, and storing the compressed weight matrix only requires storing a smaller number of coordinates and weights. For the first image generated according to the compressed weight matrix, the image processing method of the present application uses step 05 to generate a second image according to the first image, so that the blurring effect of the second image reaches the blurring effect of filtering using the weight matrix before compression.
[0055] For example, Figure 3 The weight matrix Z1 in the filter is used to filter the processed image D1 and then output the image T0. Figure 4 As shown, the gradient calculation along the row direction of the weight matrix Z1 is performed, specifically, the weight of the next weight unit in each row of weight units is subtracted from the weight of the previous weight unit to obtain the weight matrix Z2. In the weight matrix Z2, the weights of the three weight units in the 2nd row, 3rd row, and 4th row of the 2nd column are Q0, and the weights of the three weight units in the 2nd row, 3rd row, and 4th row of the 9th column are -Q0. In this way, the weight matrix Z2 reduces the number of weight units with non-zero weights compared with the weight matrix Z1, that is, the "compression" of the matrix size is achieved. For the weight matrix Z2, only the coordinates and weights of the weight units with non-zero weights are stored, that is, only 6 coordinates and corresponding weights need to be stored, wherein the storage of each weight coordinate and the weight corresponding to the weight coordinate is equivalent to the storage of a weight unit with a weight, and the storage space occupied by storing 6 coordinates and corresponding weights is 6*8bit. Compared with the storage space of 5*9*8bit required to store the weight matrix Z1, the storage space occupied is greatly saved.
[0056] like Figure 4 As shown, when blurring is performed, if the size of the weight matrix corresponding to the blurring scale is the same as the size of the weight matrix Z2, the method of step 03 can be performed to determine the weight matrix Z2 according to the stored 6 coordinates and the corresponding weights. For example, according to the blurring scale, it is determined that a matrix with 5 rows and 9 columns needs to be used for filtering. In the matrix with 5 rows and 9 columns, the weights of the 6 pre-stored coordinates are set to the corresponding pre-stored weights, and the weights of the remaining coordinates are all 0, and the weight matrix Z2 can be determined.
[0057] Please combine Figure 3 and Figure 4 , using the determined weight matrix Z2 to filter the image to be processed, it can generate Figure 4 In the first image T1 shown, the pixel values of the 2nd row and 5th column, the 3rd row and 5th column, and the 4th row and 5th column are Q0, the pixel values of the 2nd row and 11th column, the 3rd row and 11th column, and the 4th row and 11th column are -Q0, and the remaining pixel values are 0. However, the first image T1 is not the blurring effect of the target, and the blurring effect of the target is the blurring effect of the image T0. In other words, the blurring effect of the target is to filter the image to be processed using the weight matrix Z2 "compressed" by the weight matrix Z1, and finally obtain the same blurring effect as the image T0 generated by filtering with the weight matrix Z1 that is not "compressed", thereby reducing the storage space occupied while ensuring the blurring effect.
[0058] Step 05 of the image processing method of the present application further processes the first image T1 to obtain the second image T2, so that the second image T2 achieves the same blurring effect as the image T0. Figure 4 , the weight matrix Z2 is a matrix obtained by performing gradient calculation along the row direction on the basis of the weight matrix Z1. Therefore, the weights of the first image T1 are "compressed" along the row direction compared with the image T0. The weights of the first image T1 can be expanded along the row direction using the method of step 05 to achieve the filtering effect of the weight matrix Z1. Specifically, the weight matrix Z2 is a matrix obtained by subtracting the weight of the previous weight unit from the weight of the next weight unit in each row of the weight matrix Z1, so that the first image T1 obtained by filtering using the weight matrix Z2 is compared with the image T0 obtained by filtering using the weight matrix Z1. There is "compression" of pixel values along the row direction. Using the following formula 1, the pixel values of the first image T1 can be "expanded" row by row to obtain the second image T2 with the same blurring effect as the image T0.
[0059] Formula 1:
[0060] Wherein, T2(x,y) is the pixel value of the pixel with coordinates (x,y) in the second image T2, and T1(x,y) is the pixel value of the pixel with coordinates (x,y) in the first image T1. Figure 4 As shown, the pixel value of each pixel of the second image T2 is equal to the pixel value of the corresponding position of the image T0.
[0061] Please combine Figure 5 If the weight matrix Z1 is “compressed” along the column direction to obtain the weight matrix Z3, the pixel values of the first image T1′ obtained by filtering the weight matrix Z3 can be “expanded” using the following formula 2 to obtain the second image T2′ with the same blurring effect as the image T0.
[0062] Formula 2:
[0063] Wherein, T2'(x, y) is the pixel value of the pixel with coordinates (x, y) in the second image T2', and T1'(x, y) is the pixel value of the pixel with coordinates (x, y) in the first image T1'.
[0064] That is, if the preset direction is the row direction, the weight matrix used in step 03 is a matrix obtained by "compressing" along the row direction, then in step 04 and step 05, convolution is calculated in sequence along the row direction to obtain the first image, and the pixel value accumulation is calculated along the row direction using formula 1 to obtain the second image. If the preset direction is the column direction, the weight matrix used in step 03 is a matrix obtained by "compressing" along the column direction, then in step 04 and step 05, convolution is calculated in sequence along the column direction to obtain the first image, and the pixel value accumulation is calculated along the column direction using formula 2 to obtain the second image.
[0065] If the preset direction is first along the row direction and then along the column direction, the weight matrix used in step 03 is a matrix obtained by "compressing" along the row direction first and then along the column direction. In step 04, the convolution is calculated in the row direction and the column direction in turn to obtain the first image. In step 05, the pixel values are first "expanded" along the column direction using formula 2, and then the pixel values are "expanded" along the row direction using formula 1 to obtain the second image, that is, the pixel matrix is "expanded" in the reverse order of the "compression" of the weight matrix. Similarly, if the preset direction is first along the column direction and then along the row direction, the weight matrix used in step 03 is a matrix obtained by "compressing" along the column direction first and then along the row direction. Accordingly, in step 05, the pixel values are first "expanded" along the row direction using formula 1, and then the pixel values are "expanded" along the column direction using formula 2 to obtain the second image.
[0066] Please combine Figure 3 and Figure 4 , for the same image to be processed to achieve the same blurring effect, the image processing method of the present application only needs to store the coordinates and weights of the weight units with non-zero weights in the weight matrix Z2, and the storage space required is greatly reduced compared with the weight units of the weight matrix Z1. From a hardware perspective, the number of register elements required for storage can be reduced, which can save hardware costs on the one hand, and reduce hardware area on the other hand, which is conducive to the miniaturization of the electronic device 100.
[0067] In summary, the image processing method of the embodiment of the present application only stores weight coordinates whose weights are not zero. When blur filtering is required, the weight matrix used in filtering is determined according to the weight coordinates whose weights are not zero and the corresponding weights, so as to reduce the storage space occupied. Among them, the determined weight matrix can be a matrix of weight gradient compression along a preset direction, so as to further reduce the number of stored weight coordinates. Accordingly, after the first image is obtained by filtering using the weight matrix, the pixel values of the first image are sequentially calculated along the preset direction to add up the pixel values, and the gradient compression of the first image along the preset direction can be expanded to obtain the second image, so that the blurring effect of the second image is the same as the blurring effect of the image obtained by filtering using the uncompressed weight matrix, so that the image processing method of the present application saves the required storage space on the basis of ensuring the blurring quality.
[0068] See also Figure 6 The present application also provides an image processing device 10, which includes an input module 11, a selection module 12, a matrix acquisition module 13, a first processing module 14 and a second processing module 15. Figure 1 , the input module 11 is used to execute the method in step 01, the selection module 12 is used to execute the method in step 02, the matrix acquisition module 13 is used to execute the method in step 03, the first processing module 14 is used to execute the method in step 04, and the second processing module 15 is used to execute the method in step 05. That is, the input module 11 is used to obtain the image to be processed and the blur scale. The selection module 12 is used to obtain the pre-stored weight coordinates and the weights corresponding to the weight coordinates according to the blur scale, and the weights corresponding to the weight coordinates are not zero. The matrix acquisition module 13 is used to determine the weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates. The first processing module 14 is used to calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generate a first image according to the convolution. The second processing module 15 is used to calculate the cumulative sum of the pixel values of the pixels of the first image in sequence along a preset direction, and generate a second image according to the cumulative sum of the pixel values.
[0069] See also Figure 7The present application also provides an electronic device 100, which includes a housing 40 and a processor 30. The processor 30 is housed in the housing 40, and is used to execute the image processing method in the above steps 01, 02, 03, 04, and 05. That is, the processor 30 can be used to: obtain an image to be processed and a blur scale; obtain pre-stored weight coordinates and weights corresponding to the weight coordinates according to the blur scale; determine a weight matrix according to the blur scale, the weight coordinates, and the weights corresponding to the weight coordinates; calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generate a first image according to the convolution; calculate the cumulative sum of pixel values for the pixels of the first image in sequence along a preset direction, and generate a second image according to the cumulative sum of pixel values.
[0070] The electronic device 100 includes a mobile phone, a tablet computer, a laptop computer, a smart watch, a head-mounted display device, a smart wearable device, a smart robot, smart furniture, a smart wearable device, a digital camera, a drone, an unmanned vehicle, etc., which are not limited here. In some embodiments, the electronic device 100 also includes a register, which is used to store weight coordinates and weights corresponding to the weight coordinates.
[0071] See also Figure 8 In some embodiments, the image processing method further comprises:
[0072] 06: Construct the corresponding weight matrix according to the blur scale; and
[0073] 07: Stores the weight coordinates of non-zero weights in the weight matrix.
[0074] Please combine Figure 6 In some embodiments, the image processing device 10 further includes a construction module 16 and a storage module 17. The construction module 16 is used to execute the method in step 06, and the storage module 17 is used to execute the method in step 07. That is, the construction module 16 is used to construct a corresponding weight matrix according to the blurring scale. The storage module 17 is used to store weight coordinates whose weight values in the weight matrix are not zero.
[0075] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the method in step 06 and step 07. That is, the processor 30 may also be used to construct a corresponding weight matrix according to the blurring scale; and store weight coordinates whose weight values in the weight matrix are not zero.
[0076] Please combine Figure 6 and Figure 7The component weight matrix and storage weight coordinates of step 06 and step 07 can be obtained by executing the image processing device 10 or the processor 30 of the electronic device 100 of the present application, or after executing step 06 and step 07 in other devices to obtain weight coordinates with non-zero weights, the weight coordinates are transmitted to the register of the electronic device 100 for storage. In other words, the five steps of step 01, step 02, step 03, step 04 and step 05 and the two steps of step 06 and step 07 can be executed by the same electronic device 100, or they can be executed by different electronic devices 100. Please combine Fig. 9 For example, the first electronic device 101 is only used to execute step 06 and step 07, and the second electronic device 102 is used to execute step 01, step 02, step 03, step 04 and step 05. The weight coordinates stored in the first electronic device 101 can be transmitted to the second electronic device 102 as pre-stored weight coordinates called by the second electronic device 102 to execute step 02.
[0077] See also Fig.10 In some implementations, 06: constructing a corresponding weight matrix according to the blurring scale, including:
[0078] 061: Get the preset blur matrix corresponding to the blur scale;
[0079] 062: performing gradient calculation on the weight units of the blur matrix in sequence along a preset direction to obtain weight interpolation; and
[0080] 063: Generate a weight matrix based on weight interpolation.
[0081] Please combine Figure 6 In some embodiments, the construction module 16 is further used to execute the methods in step 061, step 062, and step 063. That is, the construction module 16 is further used to obtain a preset blur matrix corresponding to the blur scale; perform gradient calculations on the weight units of the blur matrix in sequence along a preset direction to obtain weight interpolation; and generate a weight matrix according to the weight interpolation.
[0082] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the methods in step 061, step 062, and step 063. That is, the processor 30 may also be used to obtain a preset blur matrix corresponding to the blur scale; perform gradient calculations on the weight units of the blur matrix in sequence along a preset direction to obtain weight interpolation; and generate a weight matrix according to the weight interpolation.
[0083] In some embodiments, the blur scale includes multiple blur scales, each blur scale corresponds to a preset blur matrix. For example, the blur scale includes a high scale, a medium scale, and a low scale, and the preset blur matrix includes a matrix Zj1, a matrix Zj2, and a matrix Zj3, and the sizes of the matrix Zj1, the matrix Zj2, and the matrix Zj3 decrease in sequence.
[0084] In one embodiment, the blur scale can be determined according to the user's input. For example, the blur scale corresponding to the user input "strong blur" is a high scale, and the blur scale corresponding to the user input "weak blur" is a low scale. In another embodiment, the blur scale can be determined according to the depth of field of the image to be processed. For example, the image processing method also includes: obtaining the depth of field of the image to be processed; and determining the blur scale according to the depth of field of the image to be processed. The depth of field includes shallow depth of field, medium depth of field, and far depth of field, which correspond to low scale, medium scale, and high scale, respectively.
[0085] Please combine Figure 3 and Figure 4 , the preset blur matrix is the filter matrix corresponding to the target blur effect. In the previous example, the weight matrix Z1 is the preset blur matrix, and the weight matrix Z2 is the weight matrix generated based on the weight matrix Z1. Among them, the gradient calculation in step 062 is to take the weight interpolation, specifically, the difference between the weight of the next weight unit and the weight of the previous weight unit in the weight unit along the preset direction is used as the weight interpolation of the next weight unit. The weight matrix generated according to weight interpolation in step 062 is a weight matrix obtained by replacing the weight of each weight unit with the weight interpolation of the weight unit.
[0086] Please combine Fig.11 In some embodiments, the target blur effect is to achieve a bokeh effect, also known as a bokeh effect. The bokeh effect refers to the effect of blurring the area outside the focus in photography. At present, a better algorithm for simulating bokeh effect in software is the diffusion algorithm. The principle of optical bokeh imaging is that the object point is not in focus but on multiple image points, resulting in blurring caused by the superposition of the object points on multiple image points. Fig.11 As shown in FIG. 1 , the principle of the diffusion algorithm is to diffuse the result of the central pixel to other pixels within the blur range, which can better simulate the principle of optical bokeh imaging. Among them, the central pixel can be selected as any pixel in the image to be processed. In some application scenarios, the pixel where the light spot in the image to be processed is identified as the central pixel for bokeh effect processing.
[0087] Please combine Figure 3 and Fig.11 , Figure 3This is an example of using the weight matrix Z1 to diffuse the pixel results of the 3rd row and 8th column in the image to be processed to the size range of the weight matrix Z1, that is, to the neighborhood range of 5 rows and 9 columns based on the diffusion algorithm. The effect after diffusion is presented as image T0. In other words, for blurring the image to be processed to the blur scale of image T0, the corresponding preset blur matrix is the weight matrix Z1.
[0088] Please combine Figure 3 and Figure 4 , the following example illustrates a method for generating a weight matrix Z2 using a preset blur matrix (weight matrix Z1). Assume that the preset direction is the row direction. For the weight units in the first row of the weight matrix Z1, their weights are all 0, so the difference between the weight of the next weight unit in the row direction and the weight of the previous weight unit is 0, that is, the weight interpolation of the first row is all 0. For the weight unit in the 2nd row and 1st column of the weight matrix Z1, its own weight is 0, and there is no previous weight unit in the row direction, so it is regarded as the weight of the previous weight unit is 0, so the weight interpolation of the weight unit in the 2nd row and 1st column is 0. For the weight unit in the 2nd row and 2nd column of the weight matrix Z1, its own weight is Q0, and the weight of the previous weight unit in the row direction is 0, so the weight interpolation of the weight unit in the 2nd row and 2nd column is Q0. For the weight unit in the 2nd row and 3rd column of the weight matrix Z1, its own weight is Q0, and the weight of the previous weight unit in the row direction is Q0, so the weight interpolation of the weight unit in the 2nd row and 3rd column is 0. Similarly, the interpolated weight values of the weight units in the 2nd row and 4th column to the 2nd row and 8th column are all 0. For the weight unit in the 2nd row and 9th column of the weight matrix Z1, its own weight value is 0, and the weight value of the previous weight unit along the row direction is Q0, so the interpolated weight value of the weight unit in the 2nd row and 9th column is -Q0. The weight matrix Z2 can be generated by calculating the interpolated weight values of other rows of the weight matrix Z1 using the same calculation method.
[0089] Please combine Figure 3 and Figure 4, for the first image T1 obtained using the weight matrix Z2, since the weight matrix Z2 is obtained by performing gradient calculation along the row direction of the weight matrix Z1, the cumulative sum is calculated for each pixel of the first image T1 in turn using the formula, and the pixel value at the corresponding position is replaced by the cumulative sum to generate the second image. For the pixel in the 2nd row and 1st column of the first image T1, its own pixel value is 0, and the pixel is the first pixel in the row, so the cumulative sum of the pixels in the 2nd row and 1st column is 0. For the pixel in the 2nd row and 2nd column of the first image T1, its pixel value is 0, and its front column pixels along the row direction only include the pixels in the 2nd row and 1st column with a pixel value of 0, so the cumulative sum of the pixels in the 2nd row and 2nd column is 0. For the pixel in the 2nd row and 5th column of the first image T1, its own pixel value is Q0, and the pixel values of its front column pixels along the row direction are all 0, so the cumulative sum of the pixels in the 2nd row and 5th column is 0+0+0+0+Q0=Q0. For the pixel in the 2nd row and 6th column of the first image T1, its own pixel value is 0, and the pixel values of the front column pixels along the row direction are all 0 except the pixel value of the 2nd row and 5th column, which is Q0. Therefore, the cumulative sum of the pixels in the 2nd row and 6th column is 0+0+0+0+Q0+0=Q0. For the pixel in the 2nd row and 12th column of the first image T1, its own pixel value is -Q0, and the pixel values of the front column pixels along the row direction are all 0 except the pixel value of the 2nd row and 5th column, which is Q0. Therefore, the cumulative sum of the pixels in the 2nd row and 6th column is Q0-Q0=0. The second image T2 can be generated by calculating the cumulative sum of other rows of the first image T1 according to the same calculation method. Comparing image T0 and second image T2, the pixel values of each pixel are the same, that is, image T0 and second image T2 achieve the same bokeh effect.
[0090] See also Fig.12 In some embodiments, the method in step 062 and step 063 may be performed multiple times to further reduce the number of weight interpolations with non-zero weights. Accordingly, in the process of constructing the weight matrix, each time the gradient calculation in step 062 is performed once, the corresponding number of calculation accumulations in step 05 need to be performed.
[0091] For example, Fig.12 The weights of the five weight units of the preset blur matrix Z3 are 0, 1, 2, 3, and 4 respectively. After a gradient calculation is performed on the blur matrix Z3 along the row direction, the weight interpolation of the row is 0, 1, 1, 1, and 1 in sequence. If the weight matrix Z4 is generated according to the current weight interpolation, the number of non-zero weights in the generated weight matrix Z4 is not reduced compared to the initial blur matrix Z3. After continuing to perform a gradient calculation on the above weight interpolation along the row direction, the new weight interpolation is 0, 1, 0, 0, and 0 in sequence. The weight matrix Z5 is generated according to the new weight interpolation. The number of non-zero weights in the weight matrix Z5 is reduced by 3 compared to the initial blur matrix Z3.
[0092] Please combine Fig.12 and Fig.13 , the image T3 is obtained by filtering the processed image D2 using the blur matrix Z3, and the first image T5 is obtained by filtering using the weight matrix Z5. Since the weight matrix Z5 is obtained by performing two gradient calculations on the blur matrix Z3, it is necessary to perform two pixel value accumulation and calculations on the first image T5 to achieve the blurring effect of the image T3. That is, firstly, the pixel value accumulation and calculation of the image T5 is performed once in the row direction to obtain the image T6, and then the pixel value accumulation and calculation of the image T6 is performed once in the row direction to obtain the image T7. The image T7 is the second image finally generated, and the blurring effect of the second image T7 is the same as that of the image T3.
[0093] Similarly, if the weight matrix Z5 is a matrix obtained by performing two gradient calculations along the column direction based on the initial blur matrix Z3, then for the first image T5 obtained by filtering with the weight matrix Z5, it is necessary to perform two pixel value accumulation and calculations along the column direction to generate the second image. If the weight matrix Z5 is a matrix obtained by performing one gradient calculation along the row direction and then one gradient calculation along the column direction based on the initial blur matrix Z3, then for the first image T5 obtained by filtering with the weight matrix Z5, it is necessary to perform one pixel value accumulation and calculation along the column direction and then one pixel value accumulation and calculation along the row direction to generate the second image.
[0094] See also Fig.14 In some embodiments, when constructing the weight matrix in step 062 and step 063, the weights of the adjacent weight units of the weight unit with a larger weight in the weight matrix can be extended, that is, a certain weight is assigned to the adjacent weight units of the weight unit with a larger weight, so that the blur transition is smoother and the bokeh spots generated at different positions are more consistent. Fig.14 As shown, 06: Construct the corresponding weight matrix according to the blur scale, and also include:
[0095] 064: Filter each weight interpolation according to the preset weight threshold to select high weight interpolation;
[0096] 065: Improving the value of adjacent weighted interpolation values of high weighted interpolation along a preset direction; and
[0097] 066: Use the increased weight interpolation to replace the corresponding weight unit in the weight matrix.
[0098] Please combine Figure 6In some embodiments, the construction module 16 is also used to execute the methods in step 064, step 065, and step 066. That is, the construction module 16 is also used to screen each weight interpolation according to a preset weight threshold, screen out high-weight interpolation; increase the value of the adjacent weight interpolation of the high-weight interpolation along the preset direction; and replace the corresponding weight unit in the weight matrix with the increased weight interpolation.
[0099] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the methods in step 064, step 065, and step 066. That is, the processor 30 may also be used to screen each weight interpolation according to a preset weight threshold, screen out high-weight interpolation; increase the value of adjacent weight interpolation of the high-weight interpolation along a preset direction; and replace the corresponding weight unit in the weight matrix with the increased weight interpolation.
[0100] In one embodiment, the weight value range of the weight interpolation is [0, 255], the preset weight threshold is 200, and the weight interpolation with a weight greater than or equal to 200 is a high weight interpolation. The method of increasing the value of the adjacent weight interpolation is to take a predetermined proportion of the high weight interpolation and assign it to the adjacent weight interpolation. Fig.15 , for example, the preset direction is the row direction, and the preset ratio value is 10%. Fig.15 The weight interpolation with a medium weight of 210 is determined as a high weight interpolation, which corresponds to the weight matrix of the 1st row and 2nd column of the weight matrix Z15. For the high weight interpolation of the 1st row and 2nd column, the increment corresponding to its predetermined proportional value is 21, and the next weight interpolation along the row direction is increased by 21, so that the weight interpolation becomes 211, that is, the weight value of the weight unit of the 1st row and 3rd column of the weight matrix Z15 becomes 211. When executing step 07, the weight coordinates are stored according to the weight matrix after interpolation replacement.
[0101] The preset weight threshold is not limited to 200 in the above embodiment, and a suitable weight threshold can be selected according to the application scenario. For example, the weight threshold can also be 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, etc., which are not listed here one by one. The predetermined ratio value is not limited to 10% in the above embodiment, and a suitable predetermined ratio value can be selected according to the application scenario. For example, the predetermined ratio value can also be 5%, 6%, 7%, 8%, 9%, 11%, 12%, 13%, 14%, 15%, etc., which are not listed here one by one.
[0102] In some embodiments, the values of adjacent weight interpolations of high weight interpolations can also be increased by assigning fixed values to adjacent weight interpolations. For example, the assigned fixed values can be 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, etc., which are not listed here one by one.
[0103] See also Fig.16 In some implementations, the pre-stored weight coordinates may be directly weight-expanded to make the blur transition smoother. The image processing method further includes:
[0104] 08: Adjust the weight corresponding to the weight coordinate; and
[0105] 09: Store the adjusted weights.
[0106] Wherein, step 08 also includes:
[0107] 081: Filter the weight corresponding to each weight coordinate according to the preset weight threshold, and filter out high-weight coordinates; and
[0108] 082: Increase the weights of adjacent weighted coordinates of high-weighted coordinates along the preset direction.
[0109] Please combine Figure 6 In some embodiments, the image processing device 10 further includes an adjustment module 18, which is used to execute the methods in step 08, step 081, and step 082. The storage module 17 is also used to execute the method in step 09. That is, the adjustment module 18 is used to adjust the weight corresponding to the weight coordinate; filter the weight corresponding to each weight coordinate according to a preset weight threshold, filter out the high-weight coordinate; and increase the weight corresponding to the adjacent weight coordinate of the high-weight coordinate along the preset direction. The storage module 17 is also used to store the adjusted weight.
[0110] Please combine Figure 7 In some embodiments, the processor 30 can also be used to execute the methods in step 08, step 09, step 081, and step 082. That is, the processor 30 can also be used to screen each weight interpolation according to a preset weight threshold, screen out high-weight interpolation; improve the value of the adjacent weight interpolation along the preset direction of the high-weight interpolation; and replace the corresponding weight unit in the weight matrix with the improved weight interpolation. Adjust the weight corresponding to the weight coordinate; store the adjusted weight; screen the weight corresponding to each weight coordinate according to the preset weight threshold, screen out high-weight coordinates; and improve the weight corresponding to the adjacent weight coordinates along the preset direction of the high-weight coordinate.
[0111] Please combine Fig.15 and Fig.17, the method for selecting high-weight coordinates can be the same as the method for selecting high-weight interpolation, that is, selecting based on the weight threshold. The method for increasing the weights corresponding to adjacent weight coordinates can also be the same as the method for increasing the values of adjacent weight interpolation, that is, assigning a predetermined proportion value or a fixed value based on the weight of the high-weight coordinate. Fig.15 After the interpolation weights are increased, the Fig.16 After the weight corresponding to the weight coordinate is increased in the embodiment, the weight of the weight coordinate finally stored is the same, the only difference is that the method of step 064, step 065, and step 066 is to perform smoothing when constructing the weight matrix, and the weight of the weight coordinate stored in the register of the electronic device 100 has been smoothed and does not occupy the computing resources of the electronic device 100; the method of step 081 and step 082 is to adjust the weight of the weight coordinate stored in the register of the electronic device 100, and the electronic device 100 can choose whether to perform smoothing according to the application scenario.
[0112] See also Fig.18 In some implementations, step 08: adjusting the weight corresponding to the weight coordinate, further includes:
[0113] 083: Adjust the weight corresponding to each weight coordinate to the Nth power of 2, where N is a natural number.
[0114] Please combine Fig.19 , step 04: sequentially calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed along a preset direction, and generating a first image according to the convolution, further comprising:
[0115] 041: Calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence by using a shift operation along a preset direction.
[0116] Please combine Figure 6 In some embodiments, the adjustment module 18 is further used to execute the method in step 083. The first processing module 14 is further used to execute the method in step 041. That is, the adjustment module 18 is further used to adjust the weight corresponding to each weight coordinate to a power of two, N being a natural number. The first processing module 14 is further used to sequentially calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed by using a shift operation along a preset direction.
[0117] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the method in step 083 and step 41. That is, the processor 30 may also be used to adjust the weight corresponding to each weight coordinate to a power of two, where N is a natural number; and to sequentially calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed by using a shift operation along a preset direction.
[0118] The convolution calculation in step 04 requires multiplication of each weight unit in the weight matrix with each pixel in the image to be processed. The multiplication calculation is often implemented by a multiplier, which requires a large area. The multiplication operation between any number and a power of 2 can be changed to a shift operation, which is implemented by an adder, greatly reducing the area of the computing hardware while ensuring the same calculation results. The weights involved in the convolution calculation in step 04 come from the weights corresponding to the pre-stored weight coordinates. Therefore, by adjusting the weights corresponding to the weight coordinates to the Nth power of two through step 083, the application conditions of the shift calculation can be met, so as to perform the convolution of the weight matrix and the pixel value using the shift operation method in step 041. Among them, the positive and negative values before and after the weight adjustment remain unchanged.
[0119] In one embodiment, the weight corresponding to the weight coordinate is adjusted to a power of two, and the approximation method can be used for adjustment. For example, if the weight corresponding to a certain weight coordinate is 15, and the power of two closest to the weight is 16, then the weight corresponding to the weight coordinate is adjusted to 16. For another example, if the weight corresponding to a certain weight coordinate is -30, and the power of two closest to the weight is -32, then the weight corresponding to the weight coordinate is adjusted to -32.
[0120] In some embodiments, when constructing a weight matrix, a weight matrix in which the weights of the weight units with non-zero weights are all N-powers of two can be constructed, and the weight coordinates and corresponding weights of the weight units with non-zero weights of the weight matrix are used as pre-stored weights. For example, the weight matrix constructed in step 06 is already a weight matrix in which the weights are all N-powers of two. It is also possible to adjust the non-zero weights in the weight matrix that are not N-powers of two to N-powers of two before executing step 07 to store the weight coordinates and corresponding weights of the weight matrix with non-zero weights, and then execute step 07 so that the weights corresponding to the stored weight coordinates are all N-powers of two. In this way, the weight coordinates stored in the register of the electronic device 100 are already weight coordinates in which the weights are all N-powers of two, and there is no need to perform the adjustment of step 083, which can improve the efficiency of the blurring process.
[0121] See also Fig. 20 In some embodiments, a set of pre-stored weight coordinates and weight values corresponding to the weight coordinates correspond to a preset matrix. In one embodiment, the number of blur scales is the same as the number of blur scales corresponding to the number of weight coordinates stored, so that each blur scale corresponds to a set of weight coordinates. When blurring, a weight matrix is determined from a set of weight coordinates corresponding to the blur scale according to the input blur scale, and filtering is performed using the weight matrix.
[0122] For example, the blur scale includes the first scale, the second scale, the third scale, the fourth scale, the fifth scale, the sixth scale, the seventh scale, the eighth scale, the ninth scale and the tenth scale with blur degrees from small to large. According to these blur scales, weight matrices Zg1, weight matrix Zg2, weight matrix Zg3, weight matrix Zg4, weight matrix Zg5, weight matrix Zg6, weight matrix Zg7, weight matrix Zg8, weight matrix Zg9 and weight matrix Zg10 can be constructed from small to large sizes respectively; the weight coordinates whose weights in the above weight matrices are not zero are stored respectively, and 10 weight coordinate groups are obtained: weight coordinate group g1, weight coordinate group g2, weight coordinate group g3, weight coordinate group g4, weight coordinate group g5, weight coordinate group g6, weight coordinate group g7, weight coordinate group g8, weight coordinate group g9 and weight coordinate group g10, and the above weight matrices are respectively used as preset matrices corresponding to the above weight coordinate groups. When performing blur processing, if the input blur scale is the first scale, a weight matrix Zg1 is determined according to the pre-stored weight coordinate group g1, and the weight matrix Zg1 is used for subsequent filtering processing.
[0123] Please combine Fig.21 In some implementations, to further reduce storage space usage, weighted coordinate groups whose number is smaller than the blur scale may be pre-stored. Fig.21 As shown, step 03: determining a weight matrix according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates, including:
[0124] 031: Check whether the size of the preset matrix corresponding to the pre-stored weight coordinates matches the target blur scale;
[0125] 032: When the size of the preset matrix matches the target blur scale, determine the preset matrix as the weight matrix; and
[0126] 033: When the size of the preset matrix does not match the target blur scale, interpolation processing is performed on the preset matrix to obtain an interpolation matrix whose size matches the target blur scale, and the interpolation matrix is determined as a weight matrix.
[0127] Please combine Figure 6 In some embodiments, the matrix acquisition module 13 is further used to execute the methods in step 031, step 032, and step 033. That is, the matrix acquisition module 13 is further used to check whether the size of the preset matrix corresponding to the pre-stored weight coordinates matches the blur scale; if the size of the preset matrix matches the blur scale, the preset matrix is determined as the weight matrix; and if the size of the preset matrix does not match the blur scale, the preset matrix is interpolated to obtain an interpolation matrix whose size matches the blur scale, and the interpolation matrix is determined as the weight matrix.
[0128] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the methods in step 031, step 032, and step 033. That is, the processor 30 may also be used to check whether the size of the preset matrix corresponding to the pre-stored weight coordinates matches the blur scale; if the size of the preset matrix matches the blur scale, the preset matrix is determined as the weight matrix; and if the size of the preset matrix does not match the blur scale, the preset matrix is interpolated to obtain an interpolation matrix whose size matches the blur scale, and the interpolation matrix is determined as the weight matrix.
[0129] The target blur scale refers to a blur scale used for current image processing among multiple blur scales.
[0130] See also Fig. 20 In one embodiment, the pre-stored set of weight coordinates is a set of coordinates with non-zero weights derived from the weight matrix constructed in step 06 and stored in step 07, and the preset matrix corresponding to the weight coordinate group is the weight matrix constructed in step 06. For example, the weight coordinate group g1 is derived from the weight matrix Zg1, and the weight matrix Zg1 is the preset matrix corresponding to the weight coordinate group g1.
[0131] See also Fig. 22 In another embodiment, the preset matrix corresponding to the weighted coordinate group is determined by the weighted coordinate group. Assume that the preset matrix is a matrix with M rows and N columns, and the coordinates in the weighted coordinate group are G(Xi, Yj), where the coordinates of the largest row are G(Xmax, Yj), and the coordinates of the largest column are G(Xi, Ymax), then the number of rows in the preset matrix is Preset the number of matrix columns For example, Fig. 22 In the weight coordinate group g2, Xmax=3, Ymax=3, then N=3, M=3, the preset matrix corresponding to the weight coordinate group g2 is a matrix of 3 rows and 3 columns, and the coordinates of the weight coordinate group g2 are all coordinates with non-zero weights. After filling the coordinates and weights of the weight coordinate group g2 into the matrix of 3 rows and 3 columns in sequence, set the weights of the remaining coordinates without weights to 0, and you can get the preset matrix Zg2 corresponding to the weight coordinate group g2. Fig. 22 In the weighted coordinate group g3, Xmax=3, Ymax=4, then N=3, M=5. Similarly, after filling the coordinates and weights of the weighted coordinate group g3 into a matrix of 3 rows and 5 columns in sequence, set the weights of the remaining coordinates without weights to 0, and you can get the preset matrix Zg3 corresponding to the weighted coordinate group g3.
[0132] See also Fig.23In one embodiment, among the first scale, the second scale, the third scale, the fourth scale, the fifth scale, the sixth scale, the seventh scale, the eighth scale, the ninth scale and the tenth scale with blurring degrees from small to large, the commonly used blurring scales are the first scale, the second scale and the sixth scale. Then, when storing the weighted coordinate groups, only the weighted coordinate groups g1, g2 and g6 may be stored to save storage space and reduce the area of storage hardware.
[0133] For example, when the target blur scale is the first scale, the size of the preset matrix corresponding to the weight coordinate group g1 in the pre-stored weight coordinate group matches the first size, that is, the size of the weight matrix Zg1 matches the first size, then according to step 032, the weight matrix Zg1 is determined as the weight matrix for subsequent filtering processing.
[0134] For example, when the target blur scale is the third scale, in the pre-stored weight coordinate groups, the sizes of the preset matrices corresponding to the weight coordinate groups g1, g2, and g6 do not match the third scale. Among them, the size of the preset matrix Zg2 corresponding to the weight coordinate group g2 is closest to the matching size of the third scale. According to step 033, the preset matrix Zg2 is interpolated to obtain the interpolation matrix Zg3 whose size matches the blur scale, and the interpolation matrix Zg31 is determined as the weight matrix for subsequent filtering processing.
[0135] In certain embodiments, the proximity principle is used to determine the preset matrix that is closest to the target blur scale. For example, the size of the matching matrix of the target blur scale is M rows and N columns, and the size of the preset matrix corresponding to the pre-stored weight coordinate group is M(k) rows and N(k) columns, k is the standard serial number, for example, the size of the preset matrix Zg2 corresponding to the weight coordinate group g2 is M(2) rows and N(2) columns. If M(k)=M and N(k)=N of a preset matrix, then it is considered that the preset matrix matches the target blur scale. For example, the target is the second scale, and the blur effect of the second scale requires a matrix of 2 rows and 2 columns for filtering. The size of the matching matrix of the second scale is 3 rows and 3 columns. In the pre-stored weight coordinate group, the size of the preset matrix Zg2 corresponding to the weight coordinate group g2 is also 3 rows and 3 columns, then the preset matrix Zg2 matches the second scale.
[0136] If none of the preset matrices corresponding to the pre-stored weight coordinate groups match the target blur scale, set L(k) = |M(k)-M| + |N(k)-N|, and determine the preset matrix corresponding to the smallest L(k) as the preset matrix closest to the target blur scale.
[0137] For example, the target is the third scale, and the size of the matching matrix of the third scale is 3 rows and 5 columns. The pre-stored weight coordinate groups include weight coordinate group g1, weight coordinate group g2, and weight coordinate group g6. The preset matrix Zg1 corresponding to weight coordinate group g1 is 1 row and 1 column, L(1)=6; the preset matrix Zg2 corresponding to weight coordinate group g2 is 3 rows and 3 columns, L(2)=2; the preset matrix Zg6 corresponding to weight coordinate group g6 is 7 rows and 7 columns, L(6)=6, where L(2) is the minimum value. Therefore, the preset matrix Zg2 is determined as the preset matrix closest to the target blur scale. When executing step 033, the preset matrix Zg2 is interpolated.
[0138] See also Fig.24 In some implementations, step 033: interpolating the preset matrix to obtain an interpolation matrix whose size matches the target blur scale includes:
[0139] 0331: Determine the size of the interpolation matrix according to the blur scale;
[0140] 0332: According to the size of the interpolation matrix, interpolate the pre-stored weight coordinates and the weights corresponding to the weight coordinates to obtain multiple interpolation units; and
[0141] 0333: Generate an interpolation matrix based on the interpolation unit.
[0142] Please combine Figure 6 In some embodiments, the matrix acquisition module 13 is further used to execute the methods in step 0331, step 0332, and step 0333. That is, the matrix acquisition module 13 is further used to determine the size of the interpolation matrix according to the blur scale; perform interpolation processing on the pre-stored weight coordinates and the weights corresponding to the weight coordinates according to the size of the interpolation matrix to obtain multiple interpolation units; and generate an interpolation matrix according to the interpolation units.
[0143] Please combine Figure 7 In some embodiments, the processor 30 may also be used to execute the methods in step 0331, step 0332, and step 0333. That is, the processor 30 may also be used to determine the size of the interpolation matrix according to the blur scale; perform interpolation processing on the pre-stored weight coordinates and the weights corresponding to the weight coordinates according to the size of the interpolation matrix to obtain multiple interpolation units; and generate an interpolation matrix according to the interpolation units.
[0144] Please combine Fig.25For example, the matrix size of the target blur scale matching is 3 rows and 5 columns. In the pre-stored weight coordinate group, the preset matrix Zg2 corresponding to the weight coordinate group g2 is the preset matrix with the closest size, and the preset matrix Zg2 is a 3-row 3-column matrix. The interpolation processing of the preset matrix Zg2 includes expanding the preset matrix Zg2 to 3 rows and 5 columns, and interpolating the weights of the weight units of the preset matrix Zg2 to the newly expanded 2 columns to obtain the interpolation units. The interpolation method includes the proximity interpolation method, the linear interpolation method, etc., which are not limited here. For example, the linear interpolation method is used to obtain the interpolation unit, and the weight of the interpolation unit in the 1st row and 4th column after expansion is obtained by interpolating the weights of the 1st row and 1st column, the 1st row and 2nd column, and the 1st row and 3rd column of the preset matrix Zg2, and the weight of the interpolation unit in the 1st row and 5th column after expansion is obtained by interpolating the weights of the 1st row and 2nd column, the 1st row and 3rd column of the preset matrix Zg2 and the weights of the interpolation unit in the 1st row and 4th column. By analogy, the interpolation units of the 4th and 5th columns are expanded, and the interpolation matrix Zg3 can be obtained by combining the weight units of the original preset matrix Zg2 and the expanded interpolation units of the 4th and 5th columns.
[0145] Fig.25 An embodiment is an embodiment of interpolating a small-sized preset matrix to obtain a large-sized interpolation matrix. If the matrix size matched by the target blur scale is smaller than the size of the preset matrix closest in size, the large-sized preset matrix can be interpolated into a small-sized interpolation matrix. For example, a preset matrix of 7 rows and 7 columns is interpolated to obtain an interpolation matrix of 5 rows and 7 columns. There is no limitation here.
[0146] In some embodiments, the preset matrix corresponding to the pre-stored weight coordinate group is a small-size matrix. For example, the pre-stored weight coordinates are G(Xi, Yj), where the coordinates of the largest row are G(Xmax, Yj), and the coordinates of the largest column are G(Xi, Ymax), Xmax≤3, Ymax≤3. In this way, the number of weight coordinates to be saved is small, which can greatly reduce the storage space occupied. In the case where the target blur scale does not match the preset matrix corresponding to the pre-stored weight coordinates, the preset matrix closest to the target blur scale can be interpolated to obtain a weight matrix that matches the target blur scale.
[0147] In yet another embodiment, weighted coordinate groups with high usage frequencies may be pre-stored based on statistical usage frequencies.
[0148] In another embodiment, weighted coordinate groups can be pre-stored at uniform intervals according to the number of blur scales. For example, the number of blur scales is 20, and the number of pre-stored weighted coordinate groups is 4 groups, each weighted coordinate group is used for blur processing of 5 blur scales, and the size of the preset matrix corresponding to each weighted coordinate group matches at least one blur scale in the corresponding 5 blur scales. For example, the 1st to 5th blur scales correspond to the weighted coordinate group g1, the 2nd to 10th blur scales correspond to the weighted coordinate group g2, the 11th to 15th blur scales correspond to the weighted coordinate group g3, and the 16th to 20th blur scales correspond to the weighted coordinate group g4. If the target blur scale is the 12th blur scale, it corresponds to the weighted coordinate group g3, and then whether to perform interpolation processing on the preset matrix is determined based on whether the size of the preset matrix corresponding to the weighted coordinate group g3 matches the 12th blur scale. In this way, when performing interpolation processing, the size of the preset matrix as the source of the interpolation matrix is closer to the size of the interpolation matrix, making the interpolation effect smoother.
[0149] See also Fig.26 The present application also provides a non-volatile computer-readable storage medium 300 including a computer program 301. When the computer program 301 is executed by one or more processors 30, the processor 30 can execute the image processing method of any of the above embodiments.
[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, a technician in the field may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are contradictory.
[0151] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0152] Although the embodiments of the present application have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations on the present application. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present application.
Claims
1. An image processing method, characterized in that: include: Get the image to be processed and the blur scale; Acquire pre-stored weight coordinates and weight values corresponding to the weight coordinates according to the blurring scale, wherein the weight values corresponding to the weight coordinates are not zero; Determine a weight matrix according to the blurring scale, the weight coordinates and the weights corresponding to the weight coordinates; Calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence along a preset direction, and generating a first image according to the convolution; and The pixel value accumulation sum is calculated sequentially for the pixels of the first image along the preset direction, and the second image is generated according to the pixel value accumulation sum.
2. The image processing method according to claim 1, characterized in that: The image processing method further comprises: Constructing the corresponding weight matrix according to the blurring scale; and The weight coordinates of the weights in the weight matrix that are not zero are stored.
3. The image processing method according to claim 2, characterized in that: The step of constructing the corresponding weight matrix according to the blurring scale includes: Obtaining a preset blur matrix corresponding to the blur scale; Performing gradient calculation on the weight units of the blur matrix in sequence along the preset direction to obtain weight interpolation; and The weight matrix is generated according to the weight interpolation.
4. The image processing method according to claim 3, characterized in that: The step of constructing the corresponding weight matrix according to the blurring scale further includes: Screening each of the weighted interpolations according to a preset weight threshold to screen out high-weighted interpolations; Increasing the value of the adjacent weighted interpolation of the high weighted interpolation along the preset direction; and The corresponding weight units in the weight matrix are replaced by interpolation using the increased weights.
5. The image processing method according to claim 1, characterized in that: The image processing method further comprises: Adjusting the weight corresponding to the weight coordinate; and The adjusted weights are stored.
6. The image processing method according to claim 5, characterized in that: The adjusting the weight corresponding to the weight coordinate includes: The weight corresponding to each weighted coordinate is screened according to a preset weight threshold to screen out high-weighted coordinates; and Increase the weight corresponding to the adjacent weight coordinates of the high-weight coordinate along the preset direction.
7. The image processing method according to any one of claims 5 or 6, characterized in that: The step of sequentially calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed along a preset direction includes: Calculating the convolution of the weight matrix and the pixel value of each pixel in the image to be processed in sequence by using a shift operation along a preset direction; The adjusting the weight corresponding to the weight coordinate includes: The weight corresponding to each of the weight coordinates is adjusted to a power of two, where N is a natural number.
8. The image processing method according to claim 1, characterized in that: The weight corresponding to the pre-stored weight coordinate is a power of two, where N is a natural number, and the convolution of the weight matrix and the pixel value of each pixel in the image to be processed is calculated in sequence along a preset direction, including: The convolution of the weight matrix and the pixel value of each pixel in the image to be processed is calculated in sequence by using a shift operation along a preset direction.
9. The image processing method according to claim 1, characterized in that: The pre-stored weight coordinates include a plurality of groups, each group of the weight coordinates corresponds to a preset matrix, the number of the pre-stored weight coordinates is less than the number of blur scales, and the weight matrix is determined according to the blur scale, the weight coordinates and the weights corresponding to the weight coordinates, including: Checking whether the size of the preset matrix corresponding to the pre-stored weight coordinates matches the target blur scale; In a case where the size of the preset matrix matches the target blur scale, determining the preset matrix as the weight matrix; and In the case that the size of the preset matrix does not match the target blur scale, an interpolation process is performed on the preset matrix to obtain an interpolation matrix whose size matches the target blur scale, and the interpolation matrix is determined as the weight matrix.
10. The image processing method according to claim 9, characterized in that: The step of performing interpolation processing on the size of the preset matrix to obtain an interpolation matrix includes: Determining the size of the interpolation matrix according to the blur scale; According to the size of the interpolation matrix, interpolate the pre-stored weight coordinates and the weights corresponding to the weight coordinates to obtain a plurality of interpolation units; and The interpolation matrix is generated according to the interpolation unit.
11. The image processing method according to any one of claims 1, 3 or 9, characterized in that: The preset direction includes any one of a row direction, a column direction or a row-column combination direction.
12. An image processing device, characterized in that: The image processing device comprises: An input module, used to obtain the image to be processed and the blur scale; A selection module, used for acquiring pre-stored weight coordinates and weight values corresponding to the weight coordinates according to the blurring scale, wherein the weight values corresponding to the weight coordinates are not zero; A matrix acquisition module, used to determine a weight matrix according to the blurring scale, the weight coordinates and the weights corresponding to the weight coordinates; A first processing module, configured to sequentially calculate the convolution of the weight matrix and the pixel value of each pixel in the image to be processed along a preset direction, and generate a first image according to the convolution; and The second processing module is used to calculate the cumulative sum of pixel values of the pixels of the first image in sequence along the preset direction, and generate a second image according to the cumulative sum of pixel values.
13. An electronic device, characterized in that: The electronic device comprises: housing; and A processor, wherein the processor is housed in the housing, and the processor is used to execute the image processing method according to any one of claims 1 to 11.
14. A non-volatile computer-readable storage medium containing a computer program, comprising one or more processors, and when the computer program is executed by the one or more processors, the processors implement instructions of the image processing method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and storage medium
CN113177890A
Image enhancement processing method and apparatus thereof, program, and data recording medium
JP2003309723A
Dilated convolution using systolic array
US20200410036A1
Using difference kernels for image filtering
US7359576B1
Convolution operation circuit and method, neural network accelerator and electronic device
WO2023030061A1
