Image processing method, device, storage medium and terminal

By acquiring terminal motion information, the fuzzy convolution kernel is generated for de-fuzzing and the details of the image are enhanced, which solves the problem of poor repairing of blurred images in the prior art, and significantly improves the clarity and repair effect of the image.

CN114708166BActive Publication Date: 2024-12-03GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210397863.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-08
Publication Date
2024-12-03
Estimated Expiration
2042-04-08

AI Technical Summary

Technical Problem

The prior art uses poor blur repair effect when processing blurred images caused by movement or jitter, and cannot effectively restore image information.

Method used

By obtaining the motion information of the terminal within the preset time period when shooting the original image, a fuzzy convolution kernel is generated, the original image is defuzzed, and the repaired image is enhanced to improve the image clarity.

Benefits of technology

Effectively restore image information, improve the clarity and blur repair effect of blurred images, and improve the blurred image repair effect caused by movement or jitter.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114708166B_ABST
    Figure CN114708166B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an image processing method, device, storage medium and terminal, the method comprising: obtaining the motion information of the terminal from the starting point of the preset time period to the end point of the preset time period within a preset time period of shooting the original image, generating a fuzzy convolution kernel corresponding to the original image based on the motion information, deblurring the original image based on the fuzzy convolution kernel to obtain a repaired image corresponding to the original image, performing detail enhancement processing on the repaired image to obtain a target image corresponding to the original image. The embodiment of the present application deblurs the original image through the motion information of the terminal in the process of shooting the original image, which can effectively restore the information in the original image, and then performs corresponding detail enhancement processing on the deblurred image, which can improve the clarity of the image, thereby improving the blur repair effect of the blurred image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and terminal. Background Art

[0002] Nowadays, smart portable terminals such as mobile phones and tablets have become indispensable tools in users' daily lives. With the substantial improvement of mobile phone pixels and image quality, more and more users choose to use mobile phones to take pictures, and even some mobile phones used for professional photography can replace ordinary cameras to take pictures. On mobile phones, you can not only take pictures, but also perform some image processing operations, such as blurring the background, adjusting colors, adjusting filters, graffiti, adding text, cutting out pictures, etc. The increasing number of image processing operations brings users a good user experience. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, computer storage medium and terminal, which can improve the clarity of deblurred images, and further improve the blur restoration effect of blurred images. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Acquire motion information of the terminal within a preset time period of shooting the original image, and generate a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period;

[0006] Deblurring the original image based on the blur convolution kernel to obtain a restored image corresponding to the original image;

[0007] Perform detail enhancement processing on the repaired image to obtain a target image corresponding to the original image.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0009] An information acquisition module, used to acquire motion information of the terminal within a preset time period of shooting the original image, and generate a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period;

[0010] A fuzzy calculation module, used for performing a deblurring process on the original image based on the fuzzy convolution kernel to obtain a restored image corresponding to the original image;

[0011] The enhancement processing module is used to perform detail enhancement processing on the repaired image to obtain a target image corresponding to the original image.

[0012] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium has a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0013] In a fourth aspect, an embodiment of the present application provides a terminal, which may include: a memory and a processor; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the memory and executing the above-mentioned method steps.

[0014] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0015] In an embodiment of the present application, firstly, the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period within the preset time period of shooting the original image is obtained, and then a fuzzy convolution kernel corresponding to the original image is generated based on the motion information, and the fuzzy convolution kernel is used to deblur the original image to obtain a repaired image, and finally the repaired image is subjected to detail enhancement processing to obtain a target image corresponding to the original image. The embodiment of the present application deblurs the original image through the motion information of the terminal in the process of shooting the original image, and can effectively restore the information in the original image, and then the corresponding detail enhancement processing is performed on the deblurred image, which can improve the clarity of the image, and thus can improve the blur repair effect of the blurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a scene schematic diagram of an image processing method provided in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a spatial coordinate system provided in an embodiment of the present application;

[0020] Figure 4 is a flowchart of another image processing method provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a blur convolution kernel provided in an embodiment of the present application;

[0022] Figure 6 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0023] Figure 7 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0025] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0026] When users use mobile phones to take photos, the images taken are often blurred due to movement or shaking. Although the anti-shake shooting function has been introduced in mobile phones, the problem of blurred images taken due to movement or shaking is still unavoidable. In the related art, when blur repair is performed on blurred images taken due to movement, a blur kernel is usually used to perform an inverse filtering operation to obtain a repaired image. However, the use of a blur kernel for blur repair has a poor repair effect on blurred images.

[0027] The present application is described in detail below with reference to specific embodiments.

[0028] See also Figure 1 , which is a scene diagram of an image processing method provided in an embodiment of the present application.

[0029] like Figure 1 As shown, the application scenario of the embodiment of the present application may be a scenario in which a user performs a deblurring operation on an image on a terminal. Figure 1 In the figure, the terminal shown is a mobile phone, and the interface displayed on the screen of the mobile phone can be an operation interface in the camera software or an operation interface in the photo editing software. The operation interface displays a picture to be operated, and the picture to be operated can be a picture stored on the mobile phone. There are many operation functions below the picture to be operated, such as color adjustment, deblurring, filter, editing, cutout and other functions. When the user selects the deblurring function, the terminal in the embodiment of the present application can perform the following operation steps: obtain the motion information of the terminal within a preset time period of shooting the original image, generate a fuzzy convolution kernel corresponding to the original image based on the motion information, the motion information is the motion information of the terminal from the starting point of the preset time period to the end point of the preset time period, deblur the original image based on the fuzzy convolution kernel, obtain the repaired image corresponding to the original image, perform detail enhancement on the repaired image, and obtain the target image corresponding to the original image. Therefore, the embodiment of the present application deblurs the original image through the motion information of the terminal within a preset time period when the original image is captured, which can effectively restore the information in the image, and then perform corresponding detail enhancement processing on the deblurred image to improve the clarity of the image, thereby improving the blur repair effect of the blurred image.

[0030] In the following method embodiments, for the sake of convenience, only the execution subject of each step is introduced and described as the terminal.

[0031] See also Figure 2 , is a flow chart of an image processing method provided in an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps:

[0032] S201, obtaining motion information of the terminal within a preset time period of shooting an original image, and generating a blur convolution kernel corresponding to the original image based on the motion information.

[0033] It is understandable that the preset time period may be a time period between a time point before the shooting moment and another time point after the shooting moment. The shooting moment refers to the moment when the terminal receives the shooting instruction and then performs the shooting action. The shooting instruction may be an instruction formed by the user pressing the shooting button in the application, or a voice shooting instruction input by the user to the terminal, or an instruction formed by the user pressing a physical button on the terminal for performing the shooting action. For example, a time point before the shooting moment may be a time point 0.5 seconds or 1 second earlier than the shooting moment. Similarly, another time point after the shooting moment may be a time point 0.5 seconds or 1 second later than the shooting moment.

[0034] Motion information refers to the motion information of the terminal from the starting point of the preset time period to the end point of the preset time period, that is, when the terminal moves from the spatial position corresponding to the starting point of the preset time period to another spatial position corresponding to the end point of the preset time period, the position change between the two spatial positions, and the position change may include the angle change and the distance change. Among them, the spatial position corresponding to the starting point of the preset time period can be understood as the spatial position of the terminal at the moment when it is at the starting point of the preset time period; the other spatial position corresponding to the end point of the preset time period can be understood as the spatial position of the terminal at the moment when it is at the end point of the preset time period. For example, see Figure 3 In the two-dimensional space coordinate system shown, at the starting point of the preset time period, the terminal is located at point A in the above coordinate system, and at the end point of the preset time period, the terminal is located at point B in the above coordinate system. Then, the movement from point A to point B is reflected in the coordinate system as a vector from point A to point B, that is, vector AB. The motion information can be the position change from point A to point B, and the position change from point A to point B can be reflected in the coordinate system as the modulus and direction angle of vector AB. The modulus can correspond to the distance change between point A and point B, and the direction angle can correspond to the angle change between point A and point B. In the embodiment of the present application, the distance refers to the straight-line distance between point A and point B.

[0035] The blur convolution kernel refers to a two-dimensional matrix used to deblur an image.

[0036] In some embodiments, the motion information of the terminal moving from the starting point of the preset time period to the preset time period can be calculated by the terminal when shooting the original image and stored in the image information of the original image. When the original image is deblurred, the pre-stored motion information can be directly obtained from the image information of the original image. Further, the fuzzy convolution kernel corresponding to the original image is generated based on the motion information, which can be understood as first determining the motion angle and motion distance based on the motion information, and then calculating the fuzzy convolution kernel based on the motion angle and motion distance.

[0037] Among them, determining the motion angle and motion distance based on the motion information can be understood as, after the motion information is obtained, which part of the information in the motion information is the motion angle and which part of the information is the motion distance can be determined according to the preset storage rules. Specifically, the preset storage rules can be to divide the motion information into two parts of information, which can include the first half of the information and the second half of the information, the first half of the information can be the motion angle, and the second half of the information can be the motion distance, or the first half of the information can be the motion distance, and the second half of the information can be the motion angle. Further, the width and height values ​​of the blurred convolution kernel can be calculated based on the motion distance and the motion angle. Furthermore, knowing the width and height values ​​of the blurred convolution kernel, the coordinate values ​​of each element in the blurred convolution kernel can be determined. For example, the width value of the fuzzy convolution kernel is 3, the height value is 4, the fuzzy convolution kernel can be a 4*3 two-dimensional matrix, that is, a two-dimensional matrix with 4 rows and 3 columns, and then the coordinate value of the element in the first row and first column of the fuzzy convolution kernel can be (1, 1), the coordinate value of the element in the first row and second column can be (1, 3), the coordinate value of the element in the first row and third column can be (1, 3), and so on, the coordinate value of each element in the fuzzy convolution kernel can be determined. Further, the corresponding weight of each element can be calculated using a preset calculation formula, and the parameters involved in the preset calculation formula can include two or more of the coordinate value of the element (such as x value or y value), movement distance and movement angle. Therefore, if the coordinate value of each element and the weight of each element are known, the fuzzy convolution kernel can be obtained.

[0038] S202: Deblurring the original image based on the blur convolution kernel to obtain a restored image corresponding to the original image.

[0039] In some embodiments, a linear deblurring model can be used to deblur the original image, and a restored image of the original image can be obtained. Specifically, the linear deblurring model can be: B(x,y)=p(x,y)*f(x,y)+n(x,y), in which B(x,y) represents the pixel value of each pixel in the original image in the embodiment of the present application, p(x,y) represents the weight of each element in the blur convolution kernel in the embodiment of the present application, f(x,y) represents the pixel value of each pixel in the restored image in the embodiment of the present application, and n(x,y) represents the noise information of each pixel. The above formula is used to obtain the repaired image, and the inverse operation can be performed based on the above formula. Since B(x,y) and p(x,y) are known, if the noise information (n(x,y)) is known, the repaired image f(x,y) can be obtained; if the noise information (n(x,y)) is unknown, a low-pass filter can be used to denoise the original image B(x,y) to obtain the denoised image H(x,y), and then n(x,y)=0 can be made, and H(x,y) can be used to replace B(x,y) and substitute it into the above model, and the corresponding inverse operation can be performed to obtain the repaired image f(x,y).

[0040] S203: Perform detail enhancement processing on the repaired image to obtain a target image corresponding to the original image.

[0041] In some embodiments, the high-frequency information may be used to perform detail enhancement processing on the repaired image, and a target image corresponding to the original image may be obtained. Specifically, the high-frequency information corresponding to the original image may be calculated first, and then the high-frequency information may be superimposed on the repaired image, so that the edge details of the repaired image may be enhanced, and then the target image may be obtained, that is, the image obtained by deblurring the original image and superimposing the high-frequency information.

[0042] In an embodiment of the present application, firstly, the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period within the preset time period of shooting the original image is obtained, and then a fuzzy convolution kernel corresponding to the original image is generated based on the motion information, and the fuzzy convolution kernel is used to deblur the original image to obtain a repaired image, and finally the repaired image is subjected to detail enhancement processing to obtain a target image corresponding to the original image. The embodiment of the present application deblurs the original image through the motion information formed by the terminal in the process of shooting the original image, and can effectively restore the information in the image, and then the corresponding detail enhancement processing is performed on the deblurred image, which can improve the clarity of the image, and thus can improve the blur repair effect of the blurred image.

[0043] See also Figure 4 , is a flow chart of an image processing method provided in an embodiment of the present application. Figure 4As shown, the method of the embodiment of the present application may include the following steps:

[0044] S401, decoding the code stream information of the original image photographed by the terminal to obtain image information corresponding to the original image.

[0045] It can be understood that the code stream information may refer to the compressed information stored in the terminal after the original image captured by the terminal is compressed in an image format. Commonly used image formats may be BMP format, PCX format, TIF format, GIF format, JPEG format, etc. These file formats generally use compression technology to compress image data, and then during the transmission or storage process of the image, the compressed image data is transmitted or stored. Some marker codes can usually be defined in the image format, and these marker codes can be used to distinguish and identify specific information in the image. For example, in the JPEG format, there may be a marker code SOI, and its corresponding marker code is 0xFFD8, indicating the beginning of the image; there may be a marker code EOI, and its corresponding marker code is 0xFFD9, indicating the end of the image.

[0046] In some embodiments, the code stream information of the original image captured by the terminal is decoded to obtain the image information corresponding to the original image. It can be understood that the compressed information of the original image stored in a certain image format is decoded and processed, and then the image information is obtained. Specifically, the decoding process is explained by taking the original image stored in JPEG format as an example. It should be noted that there are many ways to store image data in JPEG format. Here, the JPEG File Interchange Format (JPEG File Interchange Format, JFIF) is used for explanation. JPEG files (*.jpg) can be mainly divided into two parts: a marker code and compressed data. The marker code can be composed of two bytes. The first byte can be a fixed value 0xFF, and the second byte can be set to different values ​​according to different meanings. After a complete two-byte marker code, it can be the compressed data corresponding to the marker code, which records various information about the image. Commonly used marker code names may include SOI, APP0, DQT, SOF0, EOI, etc., and in the file, the marker code appears in the form of a marker code, for example, the marker code of SOI is 0xFFD8, that is, if the data 0xFFD8 appears in the JPEG file, it means that there is a marker code SOI here, and the marker code indicates the beginning of the image. According to the above-mentioned method of storing data in the JPEG file, the relevant information can be read out one by one to prepare for the decoding process. In the decoding process, the format of the JPEG file can be determined first; then, the color component can be internally decoded, which can be understood as the process of searching the Huffman tree; then, the DC variable value obtained by decoding in the above steps can be differentially encoded; then, inverse quantization processing, inverse Zig-zag encoding processing, interlaced positive and negative correction processing, inverse discrete cosine transform processing, YCrCb to RGB conversion processing, etc. can be performed to complete the process of decoding the code stream information of the original image, and the original data corresponding to the compressed data in the JPEG file can be obtained. The image information corresponding to the original image in the embodiment of the present application may include these original data, and the position information of these original data in the JPEG file.

[0047] S402: Use information at a preset position in the image information as movement information of the terminal within a preset time period when the terminal captures the original image.

[0048] It is understandable that for the partial explanation of the motion information, please refer to the description in S201. The following is an explanation based on the above description. The known motion information can be understood as the terminal from Figure 3When point A in the spatial coordinate system shown moves to point B in the spatial coordinate system, the position change between point A and point B. In fact, before the terminal stores the motion information in the image information, the motion information can be calculated based on the respective position information of point A and point B, and then the motion information can be stored in a preset position in the image information.

[0049] Specifically, in daily life, the position of the terminal (center of gravity) is actually the position in the three-dimensional space coordinate system. The accelerometer in the terminal can measure the position of the terminal in the three-dimensional space coordinate system in real time, so the position of the terminal in the three-dimensional space coordinate system can be mapped to the two-dimensional space coordinate system. For example, at the starting point of the preset time period, the position of the terminal in the three-dimensional space coordinate system at this time can be the position of point A', and at the end of the preset time period, the position of the terminal in the three-dimensional space coordinate system at this time can be the position of point B'. Point A can be obtained by mapping point A' in the three-dimensional space coordinate system to the two-dimensional space coordinate system, and point B can be obtained by mapping point B' in the three-dimensional space coordinate system to the two-dimensional space coordinate system. Furthermore, the coordinate values ​​of point A (Ax, Ay) and point B (Bx, By) can be calculated. Wherein, m is the movement angle, and n is the movement distance. Further, m and n can be stored at a preset position in the image information. Therefore, after obtaining the decoded image information, the information stored at the preset position in the image information can be directly obtained, and then the information stored at the preset position is used as the movement information of the terminal within the preset time of shooting the original image.

[0050] Optionally, the information at a preset position in the image information is used as the motion information of the terminal within a preset time period of shooting the original image. The specific implementation method may be: determine the position of the image end marker code in the image information, and use the information of the preset number of bytes stored after the position of the image end marker code as the motion information of the terminal within the preset time period of shooting the original image. In the embodiment of the present application, the JPEG file is used as an example for explanation. Since the JPEG file includes a marker code and compressed data, after decoding the JPEG file, the marker code and the original data can be obtained. Some commonly used marker codes include SOI, APP0, DQT, SOF0, EOI, etc. The position of the preset number of bytes after the position of the image end marker code (i.e., EOI) can be used as the preset position. The marker code of the image end marker code in the JPEG file is 0xFFD9. For example, the position of the 4 bytes after the marker code 0xFFD9 can be used as the preset position. The first 2 bytes of the 4 bytes can be used to store the motion angle, and the last 2 bytes can be used to store the motion distance, or the first 2 bytes of the 4 bytes can be used to store the motion distance, and the last 2 bytes can be used to store the motion angle. Therefore, after obtaining the image information of the original image, the original data stored in the 4 bytes after the mark code 0xFFD9 can be used as motion information.

[0051] S403, determining a motion angle and a motion distance based on the motion information, and calculating the motion angle and the motion distance to obtain a width and a height of a blur convolution kernel.

[0052] It can be understood that the blurred convolution kernel refers to a two-dimensional matrix, the width of the blurred convolution kernel can be understood as the number of columns of the two-dimensional matrix, and the height of the blurred convolution kernel can be understood as the number of rows of the two-dimensional matrix.

[0053] In some embodiments, after the motion information (the original data stored in 4 bytes) is acquired by S402, the motion angle and the motion distance can be determined in the motion information according to the preset storage rules, that is, the motion angle and the motion distance can be determined in the 4-byte original data according to the preset storage rules. For example, the preset storage rule may be that the original data stored in the first 2 bytes of the 4-byte original data is the motion angle, and the original data stored in the last 2 bytes is the motion distance. Alternatively, the preset storage rule may be that the original data stored in the first 2 bytes of the 4-byte original data is the motion distance, and the original data stored in the last 2 bytes is the motion angle. Further, the width and height of the blurred convolution kernel can be calculated. Specifically, following the description in S302, assuming that m represents the motion angle, n represents the motion distance, w represents the width of the blurred convolution kernel, and h represents the height of the blurred convolution kernel, the width and height of the blurred convolution kernel can be calculated. h=n×sinm+1. It should be noted that the above formulas for calculating w and h are applicable to the case where cosm is not 0 and sinm is not 0. When cosm is 0, the width of the blurred convolution kernel can be 1 and the height can be n; when sinm is 0, the width of the blurred convolution kernel can be n and the height can be 1.

[0054] S404: Determine the coordinate information of each element in the blurred convolution kernel based on the width and height of the blurred convolution kernel.

[0055] In some embodiments, if the width and height of the blur convolution kernel are known, that is, the number of rows and columns of the two-dimensional matrix is ​​known, then the coordinate information of each element in the two-dimensional matrix can be determined. Assuming that the coordinate information of each element in the two-dimensional matrix is ​​represented by (x, y), the number of rows of the two-dimensional matrix is ​​3, and the number of columns is 4, it can be known that the coordinate information of the element in the first row and first column of the two-dimensional matrix is ​​(1,1), the coordinate information of the element in the first row and second column is (1,2), the coordinate information of the element in the first row and third column is (1,3), the coordinate information of the element in the first row and fourth column is (1,4), and so on, the coordinate information of each element can be obtained.

[0056] S405, obtaining weights corresponding to the elements based on the coordinate information, the movement angles and the movement distances, and obtaining blurred convolution kernels based on the weights corresponding to the elements and the coordinate information.

[0057] In some embodiments, when the x value and y value of the coordinate information (x, y), the movement angle m, and the movement distance n are known, p(x, y) can be used to represent the weights corresponding to each element. The x and y in p(x, y) are the x value and y value in the coordinate information (x, y), respectively. It can be calculated that: when cosm>0, p(x, y)=1-|x·cosm-y·sinm|; when cosm<0, p(x, y)=1-|(nx)·cosm-(ny)·sinm|; when cosm=0, p(x, y)=1-y·sinm; when sinm=0, p(x, y)=1-x·cosm; wherein, when p(x, y)<0, p(x, y)=0. Furthermore, if the coordinate information of each element in the blurred convolution kernel is known, and the weight of each element in the blurred convolution kernel is also known, the blurred convolution kernel can be obtained. For example, see Figure 5 The blurred convolution kernel shown is exemplified by the element with coordinate information (2,2). The element with coordinate information (2,2) represents the element in the 2nd row and 2nd column of the blurred convolution kernel. It can be seen that the weight corresponding to the element is p(2,2)=3.

[0058] S406, performing denoising processing on the original image to obtain a denoised image, performing Wiener filtering on the denoised image and the blur convolution kernel to obtain a restored image corresponding to the original image.

[0059] It can be understood that the original image is processed accordingly and then the repaired image is obtained. The original image can be processed accordingly based on the Wiener filter prototype to obtain the repaired image. The Wiener filter prototype can be understood as using the deconvwnr() function to process the original image, the fuzzy convolution kernel and the additive noise. Specifically, assuming that the repaired image is represented by f(x,y), the function value of f(x,y) represents the pixel value of each pixel, and x and y represent the coordinate values ​​of the pixel in the image respectively. The calculation formula of f(x,y) can be: f(x,y)=deconvwnr(B(x,y), p(x,y), n(x,y)). Among them, the original image is represented by B(x,y), the function value of B(x,y) represents the pixel value of each pixel, and x and y represent the coordinate values ​​of the pixel in the image respectively; the weight of each element in the fuzzy convolution kernel can be represented by p(x,y); n(x,y) represents the additive noise. This calculation formula is mainly obtained based on the linear deblurring model B(x,y)=p(x,y)*f(x,y)+n(x,y). In this linear deblurring model, the meanings of p(x,y) and n(x,y) can be found in the above description. f(x,y) represents the initial image, which can be understood as a clear image. Then B(x,y) can be a blurred image obtained by deblurring the initial image using the linear deblurring model. Therefore, the initial image in the linear deblurring model is equivalent to the repaired image in the embodiment of the present application, and the blurred image in the linear deblurring model is equivalent to the original image in the embodiment of the present application. Then, by performing the corresponding inverse operation according to the linear deblurring model, the repaired image in the embodiment of the present application can be obtained, and the above formula f(x,y)=deconvwnr(B(x,y), p(x,y), n(x,y)) can be used to calculate the repaired image f(x,y).

[0060] In some embodiments, since the additive noise is unknown and the value of n(x,y) cannot be obtained, the original image can be denoised to obtain a denoised image. Specifically, a low-pass filter can be used to denoise the original image B(x,y) to filter out the noise in the original image B(x,y). It is assumed that after filtering, a denoised image H(x,y) can be obtained. Further, the denoised image and the blurred convolution kernel are subjected to Wiener filtering to obtain a restored image corresponding to the original image. Specifically, B(x,y) in deconvwnr(B(x,y), p(x,y), n(x,y)) can be replaced with H(x,y), and n(x,y)=0, and then the deconvwnr() function is operated to obtain the restored image f(x,y).

[0061] S407, performing Gaussian filtering on the repaired image using a preset Gaussian kernel to obtain a smoothed image corresponding to the original image.

[0062] In some embodiments, a preset Gaussian kernel is used to perform Gaussian filtering on the repaired image. It can be understood that the preset Gaussian kernel is used to scan the repaired image, and the weighted mean of the pixels in the neighborhood determined by the preset Gaussian kernel can be used to replace the pixel value of the central pixel of the preset Gaussian kernel. Therefore, each pixel can obtain a new pixel value respectively, and these new pixel values ​​can be the pixel values ​​of each pixel in the smoothed image, so that a smoothed image can be obtained. In an embodiment of the present application, the preset Gaussian kernel can be a two-dimensional 3*3 Gaussian kernel.

[0063] S408, determining the first pixel value of each pixel point in the original image in the repaired image, determining the second pixel value of each pixel point in the smoothed image, and calculating the pixel difference between the first pixel value and the second pixel value of each pixel point.

[0064] In some embodiments, the pixel value of each pixel in the original image in the restored image can be determined, which can be referred to as the first pixel value. The pixel value of each pixel in the original image in the smoothed image can be determined, which can be referred to as the second pixel value. Further, the difference between the first pixel value and the second pixel value of each pixel can be calculated, and the difference can be referred to as the pixel difference. It can be understood that the difference between the pixel value of the pixel in the restored image and the pixel value of the pixel in the smoothed image can be understood as high-frequency information, which can be understood as the high-frequency information lost in the restored image.

[0065] S409, respectively calculating the sum of the first pixel value and the pixel difference of each pixel point, and taking the sum of each pixel point as the target pixel value of each pixel point, wherein the target pixel value of each pixel point is the pixel value of each pixel point in the target image.

[0066] In some embodiments, based on S408, the first pixel value of each pixel point is known, the pixel difference value of each pixel point is known, and the sum of the first pixel value of each pixel point and the pixel difference value can be calculated. Further, the sum of each pixel point can be used as the target pixel value of each pixel point. Then, the target pixel value of each pixel point is known, and these target pixel values ​​can be in an image at the same time. The image can be called a target image, that is, a target image can be obtained, in which the pixel value of each pixel point is the target pixel value of each pixel point. It can be understood that adding the pixel difference value and the pixel value of the pixel point in the repaired image can be understood as compensating for the high-frequency information lost in the repaired image. The high-frequency information in the image refers to the detail information in the image, which can usually be the detail information of the edge. Therefore, the embodiment of the present application can compensate for the detail information of the edge in the repaired image, that is, the detail information of the edge in the repaired image can be enhanced, and then the effect of improving the clarity of the deblurred image can be achieved.

[0067] In an embodiment of the present application, first, the image information is obtained by decoding the code stream information of the original image, and then the motion information of the terminal can be obtained. Since the motion information can be pre-stored in a preset position in the image information, when it is needed, the terminal does not need to calculate the motion information again, and can directly decode and obtain it, which can save the time of subsequent image processing; then, the motion angle and motion distance can be determined according to the motion information, and then when calculating the blurred convolution kernel according to the motion angle and motion distance, a comprehensive blurred convolution kernel calculation method is designed taking into account the cosine value or sine value of the special angle, which can improve the accuracy of the blurred convolution kernel; then, the original image is deblurred based on the blurred convolution kernel and additive noise to obtain a repaired image; finally, the high-frequency information (pixel difference) lost in the repaired image is calculated, and the high-frequency information is superimposed on the repaired image. The high-frequency information is usually the detail information of the edge. Therefore, the detail information of the edge in the repaired image can be enhanced, the clarity of the repaired image can be improved, and the blur repair effect of the blurred image can be improved.

[0068] See also Figure 6 , is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 600 can be implemented as all or part of the terminal through software, hardware, or a combination of both. The device 600 includes:

[0069] The information acquisition module 610 acquires the motion information of the terminal within a preset time period of shooting the original image, and generates a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period;

[0070] A fuzzy calculation module 620, configured to perform a deblurring process on the original image based on the fuzzy convolution kernel to obtain a restored image corresponding to the original image;

[0071] The enhancement processing module 630 is used to perform detail enhancement processing on the repaired image to obtain a target image corresponding to the original image.

[0072] Optionally, the information acquisition module 610 includes:

[0073] A first acquisition unit, configured to decode code stream information of an original image captured by a terminal to obtain image information corresponding to the original image;

[0074] The second acquiring unit is configured to use information at a preset position in the image information as motion information of the terminal within a preset time period when the terminal captures the original image.

[0075] Optionally, the second acquiring unit includes:

[0076] The acquisition subunit is used to determine the position of the image end marker code in the image information, and use the preset number of bytes of information stored after the position of the image end marker code as the movement information of the terminal within a preset time period of shooting the original image.

[0077] Optionally, the information acquisition module 610 includes:

[0078] A first calculation unit, configured to determine a motion angle and a motion distance based on the motion information, and calculate the motion angle and the motion distance to obtain a width and a height of a blur convolution kernel;

[0079] A second calculation unit, used to determine the coordinate information of each element in the blurred convolution kernel based on the width and height of the blurred convolution kernel;

[0080] The third calculation unit is used to obtain the weights corresponding to the elements based on the coordinate information, the movement angle and the movement distance, and to obtain the blurred convolution kernel based on the weights corresponding to the elements and the coordinate information.

[0081] Optionally, the fuzzy calculation module 620 includes:

[0082] A denoising processing unit, used for performing denoising on the original image to obtain a denoised image;

[0083] The blur processing unit is used to perform Wiener filtering on the denoised image and the blur convolution kernel to obtain a restored image corresponding to the original image.

[0084] Optionally, the enhanced processing module 630 includes:

[0085] A first enhancement unit, configured to perform smoothing on the repaired image to obtain a smoothed image corresponding to the original image;

[0086] The second enhancement unit is used to perform superposition processing on the repaired image based on the smoothed image to obtain a target image corresponding to the original image.

[0087] Optionally, the first enhancement unit includes:

[0088] The filtering unit is used to perform Gaussian filtering on the repaired image using a preset Gaussian kernel to obtain a smoothed image corresponding to the original image.

[0089] Optionally, the second enhancement unit includes:

[0090] A difference calculation unit, used to determine the first pixel value of each pixel in the original image in the restored image, determine the second pixel value of each pixel in the smoothed image, and calculate the pixel difference between the first pixel value and the second pixel value of each pixel;

[0091] The sum calculation unit is used to calculate the sum between the first pixel value and the pixel difference of each pixel point, and use the sum of each pixel point as the target pixel value of each pixel point, and the target pixel value of each pixel point is the pixel value of each pixel point in the target image.

[0092] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 7 As shown, the terminal 700 may include: at least one processor 701 , at least one network interface 704 , a user interface 703 , a memory 705 , a display screen component 706 , and at least one communication bus 702 .

[0093] The communication bus 702 is used to realize the connection and communication between these components.

[0094] The user interface 703 may include a display screen (Display) and a camera (Camera), and the optional user interface 703 may also include a standard wired interface and a wireless interface.

[0095] The network interface 704 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0096] Among them, the processor 701 may include one or more processing cores. The processor 701 uses various interfaces and lines to connect various parts in the entire terminal 700, and executes various functions and processes data of the terminal 700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 705, and calling data stored in the memory 705. Optionally, the processor 701 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 701 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 701, and it can be implemented separately through a chip.

[0097] Among them, the memory 705 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 705 includes a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 705 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 705 may optionally be at least one storage device located away from the aforementioned processor 701. As Figure 7 As shown, the memory 705 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a program of an image processing method.

[0098] exist Figure 7In the terminal 700 shown, the user interface 703 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 701 can be used to call the program of the image processing method stored in the memory 705, and specifically perform the following operations:

[0099] Acquire motion information of the terminal within a preset time period of shooting the original image, and generate a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period;

[0100] Deblurring the original image based on the blur convolution kernel to obtain a restored image corresponding to the original image;

[0101] Perform detail enhancement processing on the repaired image to obtain a target image corresponding to the original image.

[0102] In one embodiment, when the processor 701 executes the step of acquiring the motion information of the terminal within a preset time period of shooting the original image, and generating a blurred convolution kernel corresponding to the original image based on the motion information, the following operations are specifically performed:

[0103] Decoding the code stream information of the original image captured by the terminal to obtain image information corresponding to the original image;

[0104] The information at the preset position in the image information is used as the motion information of the terminal within the preset time period of shooting the original image.

[0105] In one embodiment, when the processor 701 executes the step of using the information located at a preset position in the image information as the motion information of the terminal within a preset time period when the terminal shoots the original image, the processor 701 specifically performs the following operations:

[0106] The position of the image end mark code is determined in the image information, and information of a preset number of bytes stored after the position of the image end mark code is used as the motion information of the terminal within a preset time period of shooting the original image.

[0107] In one embodiment, when the processor 701 executes the step of generating a blur convolution kernel corresponding to the original image based on the motion information, the processor 701 specifically performs the following operations:

[0108] Determine a motion angle and a motion distance based on the motion information, and calculate the motion angle and the motion distance to obtain a width and a height of a blur convolution kernel;

[0109] Determining coordinate information of each element in the blurred convolution kernel based on the width and height of the blurred convolution kernel;

[0110] The weights corresponding to the elements are obtained based on the coordinate information, the movement angle and the movement distance, and the blurred convolution kernel is obtained based on the weights corresponding to the elements and the coordinate information.

[0111] In one embodiment, when the processor 701 performs the step of deblurring the original image based on the blur convolution kernel to obtain a repaired image corresponding to the original image, the following operations are specifically performed:

[0112] Performing denoising processing on the original image to obtain a denoised image;

[0113] The denoised image and the blurred convolution kernel are subjected to Wiener filtering to obtain a restored image corresponding to the original image.

[0114] In one embodiment, when the processor 701 performs the step of performing detail enhancement processing on the repaired image to obtain a target image corresponding to the original image, the following operations are specifically performed:

[0115] Smoothing the repaired image to obtain a smoothed image corresponding to the original image;

[0116] The repaired image is overlaid based on the smoothed image to obtain a target image corresponding to the original image.

[0117] In one embodiment, when the processor 701 performs the step of smoothing the repaired image to obtain a smoothed image corresponding to the original image, the following operations are specifically performed:

[0118] The restored image is Gaussian filtered using a preset Gaussian kernel to obtain a smoothed image corresponding to the original image.

[0119] In one embodiment, when the processor 701 performs the step of performing superposition processing on the repaired image based on the smoothed image to obtain a target image corresponding to the original image, the processor 701 specifically performs the following operations:

[0120] Determine the first pixel value of each pixel point in the original image in the repaired image, determine the second pixel value of each pixel point in the smoothed image, and calculate the pixel difference between the first pixel value and the second pixel value of each pixel point;

[0121] The sum of the first pixel value and the pixel difference of each pixel point is calculated respectively, and the sum of the pixel points is used as the target pixel value of each pixel point respectively. The target pixel value of each pixel point is the pixel value of each pixel point in the target image.

[0122] In addition, those skilled in the art will appreciate that the structure of the terminal 700 shown in the above figures does not limit the terminal 700, and the user terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the terminal 700 also includes components such as a radio frequency circuit, an audio circuit, a WiFi component, a power supply, and a Bluetooth component, which will not be described in detail here.

[0123] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the image processing method described in the above embodiments.

[0124] An embodiment of the present application further provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image processing method described in the above embodiments.

[0125] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access. The above is only an optional embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire motion information of the terminal within a preset time period of shooting the original image, and generate a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period; Deblurring the original image based on the blur convolution kernel to obtain a restored image corresponding to the original image; Performing detail enhancement processing on the repaired image to obtain a target image corresponding to the original image; Among them, the detail enhancement processing of the repaired image to obtain the target image corresponding to the original image includes: using a preset Gaussian kernel to perform Gaussian filtering on the repaired image to obtain a smoothed image corresponding to the original image; determining the first pixel value of each pixel point in the original image in the repaired image, determining the second pixel value of each pixel point in the smoothed image, and calculating the pixel difference between the first pixel value and the second pixel value of each pixel point; calculating the sum of the first pixel value and the pixel difference of each pixel point, and using the sum of each pixel point as the target pixel value of each pixel point, and the target pixel value of each pixel point is the pixel value of each pixel point in the target image.

2. The method according to claim 1, characterized in that The acquiring the motion information of the terminal within a preset time period of shooting the original image by the terminal includes: Decoding the code stream information of the original image captured by the terminal to obtain image information corresponding to the original image; The information at the preset position in the image information is used as the motion information of the terminal within the preset time period of shooting the original image.

3. The method according to claim 2, characterized in that The using the information at the preset position in the image information as the motion information of the terminal within the preset time period when the terminal shoots the original image includes: The position of the image end mark code is determined in the image information, and information of a preset number of bytes stored after the position of the image end mark code is used as the motion information of the terminal within a preset time period of shooting the original image.

4. The method according to any one of claims 1 to 3, characterized in that: The generating a blurred convolution kernel corresponding to the original image based on the motion information includes: Determine a motion angle and a motion distance based on the motion information, and calculate the motion angle and the motion distance to obtain a width and a height of a blur convolution kernel; Determining coordinate information of each element in the blurred convolution kernel based on the width and height of the blurred convolution kernel; The weights corresponding to the elements are obtained based on the coordinate information, the movement angle and the movement distance, and the blurred convolution kernel is obtained based on the weights corresponding to the elements and the coordinate information.

5. The method according to any one of claims 1 to 3, characterized in that: The deblurring the original image based on the blur convolution kernel to obtain a repaired image corresponding to the original image includes: Performing denoising processing on the original image to obtain a denoised image; The denoised image and the blurred convolution kernel are subjected to Wiener filtering to obtain a restored image corresponding to the original image.

6. The method according to any one of claims 1 to 3, characterized in that: The performing detail enhancement processing on the repaired image to obtain a target image corresponding to the original image includes: Smoothing the repaired image to obtain a smoothed image corresponding to the original image; The repaired image is overlaid based on the smoothed image to obtain a target image corresponding to the original image.

7. The method according to claim 6, characterized in that The step of smoothing the repaired image to obtain a smoothed image corresponding to the original image includes: The restored image is Gaussian filtered using a preset Gaussian kernel to obtain a smoothed image corresponding to the original image.

8. An image processing device, characterized in that: The device comprises: An information acquisition module, used to acquire motion information of the terminal within a preset time period of shooting the original image, and generate a blurred convolution kernel corresponding to the original image based on the motion information; the motion information is the motion information of the terminal pointing from the starting point of the preset time period to the end point of the preset time period; A fuzzy calculation module, used for performing a deblurring process on the original image based on the fuzzy convolution kernel to obtain a restored image corresponding to the original image; An enhancement processing module, used for performing detail enhancement processing on the repaired image to obtain a target image corresponding to the original image; Among them, the enhancement processing module is specifically used to: use a preset Gaussian kernel to perform Gaussian filtering on the repaired image to obtain a smoothed image corresponding to the original image; determine the first pixel value of each pixel point in the original image in the repaired image, determine the second pixel value of each pixel point in the smoothed image, and calculate the pixel difference between the first pixel value and the second pixel value of each pixel point; calculate the sum of the first pixel value and the pixel difference of each pixel point, and use the sum of each pixel point as the target pixel value of each pixel point, and the target pixel value of each pixel point is the pixel value of each pixel point in the target image.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps as claimed in any one of claims 1 to 7.

10. A terminal, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Coal mine low-illumination blurred image restoration method

    CN110415193A

  • Image deblurring method and device, electronic equipment and storage medium

    CN113409209A

  • Video image processing method and device

    CN113992847A