Image processing method, module, device, and storage medium

Abstract

The embodiments of the present application disclose an image processing method and a module, a device, and a storage medium; wherein the method includes: analyzing a first image to be compressed to obtain a shooting clarity; wherein the first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image; according to the shooting clarity, determining a compression rate of the image to be compressed; according to the compression rate, compressing the image to be compressed to obtain a second image.

Description

Technical Field

[0001] The embodiments of the present application relate to image processing technology, and are related to but not limited to image processing methods and modules, devices, and storage media. Background Art

[0002] With the development of photography technology, photography technology has been integrated into more and more portable mobile devices, which has greatly promoted the popularization of photography technology. The photography function of mobile phones has greatly enriched people's daily life and work. However, the images taken take up a lot of storage space in mobile phones. Summary of the invention

[0003] In view of this, the image processing method, apparatus, processor, device, and storage medium provided in the embodiments of the present application can compress the image size while ensuring the user's requirements for image clarity, thereby saving storage resources. The image processing method, module, device, and storage medium provided in the embodiments of the present application are implemented as follows:

[0004] The image processing method provided by the embodiment of the present application includes: analyzing a first image to be compressed to obtain a shooting clarity; wherein the first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image; determining a compression rate of the image to be compressed according to the shooting clarity; and compressing the image to be compressed according to the compression rate to obtain a second image.

[0005] The image processing module provided by the embodiment of the present application includes: a front-end image signal processing PreISP circuit, which is used to analyze a first image to be compressed to obtain the shooting clarity; wherein the first image is a RAW image or an image in the quality improvement processing stage of the RAW image; an image signal processor (Image Signal Processing, ISP) circuit, which is used to determine the compression rate of the image to be compressed according to the shooting clarity; and compress the image to be compressed according to the compression rate to obtain a second image.

[0006] An electronic device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0007] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0008] In an embodiment of the present application, the electronic device performs a clarity analysis based on a RAW image or an image in the quality improvement processing stage of the RAW image to obtain shooting clarity; since the RAW image is unprocessed raw data, and the image in the quality improvement processing stage of the RAW image also includes the raw data of the RAW image, both types of images retain all the information of the image. It can be seen that the shooting clarity obtained based on these images is more accurate, so that a more accurate compression rate can be obtained based on the accurate shooting clarity, so that the size of the second image finally compressed is more in line with the actual image quality. In this way, on the one hand, the storage resources occupied by the image can be saved; on the other hand, the bandwidth consumption of the image during transmission can be reduced, and the transmission efficiency of the image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0010] Figure 1A A schematic diagram of the implementation flow of the image processing method provided in the embodiment of the present application;

[0011] Figure 1B A schematic diagram of the implementation process of the RAW image quality improvement process provided in the embodiment of the present application;

[0012] Figure 2 A schematic diagram of the implementation flow of the training method of the target neural network of the embodiment of the present application;

[0013] Figure 3 A schematic diagram of the implementation flow of the image processing method according to an embodiment of the present application;

[0014] Figure 4 A schematic diagram of a mapping relationship between a compression rate to be compressed and a shooting definition in an embodiment of the present application;

[0015] Figure 5 This is a schematic diagram for comparing images taken at different sensitivities;

[0016] Figure 6 A schematic diagram of a process for acquiring a training data set of a neural network (NN) according to an embodiment of the present application;

[0017] Figure 7 A schematic diagram of the NN network training process and use process of an embodiment of the present application;

[0018] Figure 8 It is a schematic diagram of the structure of a neural network;

[0019] Fig. 9It is a schematic diagram of the process of compressing a RAW image into a JPEG (Joint Photographic Experts Group) image;

[0020] Fig.10 It is a schematic diagram of the relationship between the shooting clarity and the compression rate to be compressed;

[0021] Fig.11 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0022] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0023] Fig.13 This is a schematic diagram of the structure of the image processing module according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0028] The embodiment of the present application provides an image processing method, which is applied to an electronic device. During the implementation process, the electronic device can be various types of devices with image processing capabilities, for example, the electronic device can include a mobile phone, a tablet computer, a drone, a robot, a wearable device, a vehicle-mounted device, a desktop computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television or a sensor device, etc. The function implemented by the method can be implemented by calling a program code by a processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0029] Figure 1A A schematic diagram of the implementation flow of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 1A As shown, the method may include the following steps 101 to 103:

[0030] Step 101 , analyzing a first image to be compressed to obtain shooting clarity; wherein the first image is a RAW image or an image in a quality improvement processing stage of the RAW image.

[0031] It can be understood that the RAW image is the raw data of the light source signal captured by the image sensor such as Complementary Metal Oxide Semiconductor (CMOS) or Charge-coupled Device (CCD) and converted into a digital signal. Since the image has not been processed by post-compression, all the information of the image is retained to the maximum extent. In this way, the analysis of the shooting clarity based on the image is more accurate than the analysis of the shooting clarity based on the compressed image, which lays a solid foundation for the subsequent more accurate determination of the compression rate of the image.

[0032] The image in the RAW image quality improvement processing stage may be any image in this stage. Figure 1BAs shown, the quality improvement process includes at least one of the following: bad pixel correction, white balance, noise reduction, demosaicing, color correction, tone mapping, sharpening, and gamma correction. The steps from bad pixel correction to gamma correction are all for correcting and enhancing the RAW image. Any intermediate image of this stage is used to analyze the shooting clarity, that is, the image generated at any stage before the YUV image is generated, can be used as the basis for analyzing the shooting clarity, and the obtained analysis results are relatively accurate. This is because these images basically include all the detail information of the image. The less the detail information of the image is lost, the more accurate the shooting clarity obtained based on the image, and accordingly, the compression rate obtained based on the shooting clarity is more consistent with the quality of the image itself.

[0033] In some embodiments, the clarity of the first image can be evaluated by an energy gradient function or an entropy function; for example, the first image is first interpolated to obtain a red, green, and blue (RGB) image, and then the clarity of the first image is evaluated by an energy gradient function or an entropy function, and the clarity of the first image is used as the shooting clarity.

[0034] In other embodiments, the analysis of the shooting clarity can also be achieved through step 301 of the following embodiment, that is, the first image is input into a pre-trained target neural network, so that the target neural network obtains the shooting clarity by analyzing the first image. Regarding the training process of the target neural network, in some embodiments, it can be achieved through steps 201 to 203 of the following embodiment.

[0035] Step 102, determining a compression rate of the image to be compressed according to the shooting definition;

[0036] In some embodiments, the above steps can be implemented through steps 302 to 306 of the following embodiments, which will not be described again to avoid repetition.

[0037] Step 103: compress the image to be compressed according to the compression rate to obtain a second image.

[0038] In some embodiments, the electronic device may directly display the second image.

[0039] In an embodiment of the present application, the electronic device performs a clarity analysis based on a RAW image or an image in the quality improvement processing stage of the RAW image to obtain shooting clarity; since the RAW image is unprocessed raw data, and the image in the quality improvement processing stage of the RAW image also includes the raw data of the RAW image, both types of images retain all the information of the image. It can be seen that the shooting clarity obtained based on these images is more accurate, so that a more accurate compression rate can be obtained based on the accurate shooting clarity, so that the size of the second image finally compressed is more in line with the actual image quality. In this way, on the one hand, the storage resources occupied by the image can be saved; on the other hand, the bandwidth consumption of the image during transmission can be reduced, and the transmission efficiency of the image can be improved.

[0040] Before providing an embodiment of an image processing method, the present application embodiment first provides a method for training a target neural network. Figure 2 Schematic diagram of the implementation process of the training method of the target neural network in the embodiment of the present application, such as Figure 2 As shown, the training process of the target neural network includes the following steps 201 to 203:

[0041] Step 201, interpolating each sample first image to obtain a corresponding sample RGB image;

[0042] The sample first image is obtained by pre-capturing the scene. These sample first images can be obtained by capturing different scenes, and the shooting clarity is varied. It should be noted that the sample first image is consistent with the type of the first image described in step 101. The sample first image can be a sample RAW image or an image in the quality improvement processing stage of the sample RAW image.

[0043] Generally speaking, each pixel of the first image only includes a part of the spectrum, so it is necessary to realize the RGB value of each pixel by interpolation. In order to convert the first image into an RGB image, it is necessary to fill the missing two colors by interpolation. There are many ways of interpolation, such as neighborhood interpolation, linear interpolation or 3×3 interpolation. The advantage of interpolation is that it lays a foundation for determining the clarity of the image through the following step 202 without losing the detailed information of the first image.

[0044] Step 202, determining the clarity of the sample RGB image according to the grayscale values ​​of the pixels of the sample RGB image;

[0045] In some embodiments, the clarity of the sample RGB image can be determined according to the energy gradient function shown in the following equation (1):

[0046]

[0047] Among them, x represents the horizontal coordinate of the pixel, y represents the vertical coordinate of the pixel, and f(x,y) represents the pixel value at the pixel coordinates (x,y).

[0048] In addition, the entropy function can also achieve the same function. The entropy function based on statistical features is an important indicator to measure the richness of image information. According to information theory, the amount of information of an image f is measured by the information entropy D(f) of the image. The definition of D(f) is shown in the following formula (2):

[0049]

[0050] Among them, p i is the probability of a pixel with gray value i appearing in the image, and L is the total number of gray levels (usually 256). According to Shannon information theory, the maximum amount of information occurs when entropy is the largest. Applying this principle to the focusing process, the larger D(f) is, the clearer the image is.

[0051] Step 203, taking the clarity of each of the sample RGB images as the sample clarity of the corresponding sample first image, and using each of the sample shooting clarity and the corresponding sample first image to train the initial neural network to obtain the target neural network.

[0052] In this way, after obtaining the target neural network, in the following embodiment, the target neural network can be used to analyze the first image to obtain the shooting clarity, without converting the first image into an RGB image through interpolation processing, and without calculating the RGB image according to the above formula (1) or formula (2). In this way, the shooting clarity can be quickly obtained, thereby enhancing the real-time performance of shooting.

[0053] It should be noted that in the embodiments of the present application, there is no limitation on the structure of the initial neural network, and the initial neural network can be any network.

[0054] Based on this, the embodiment of the present application further provides an image processing method. Figure 3 Schematic diagram of the implementation process of the image processing method of the embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 301 to 307:

[0055] Step 301, inputting a first image of an image to be compressed into a pre-trained target neural network, so that the target neural network extracts image features of the first image, and analyzes the image features to obtain the shooting definition;

[0056] The first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image;

[0057] It can be understood that the clarity of this shooting can be obtained by inputting the first image into the target neural network. Compared with first interpolating the first image to obtain an RGB image and then calculating the clarity of the RGB image through an energy gradient function or an entropy function, the former can obtain the clarity of this shooting more quickly, thereby enhancing the real-time performance of the shooting.

[0058] Step 302, determining the relationship between the shooting clarity and the upper limit and lower limit of the preset clarity range; if the shooting clarity is less than the lower limit, executing step 303; if the shooting clarity is greater than the upper limit, executing step 304; if the shooting clarity belongs to the preset clarity range, executing step 305;

[0059] Step 303, using the preset maximum compression rate as the compression rate to be compressed, and then proceeding to step 307;

[0060] It is understandable that if the shooting clarity is less than the lower limit of the preset clarity range, using a compression rate greater than the maximum to-be-compressed image to compress the image may cause the image to be over-compressed, so that the second image obtained is seriously distorted, thereby seriously affecting the user's visual experience. In order to avoid this situation, in the embodiment of the present application, the maximum compression rate is set, and even if the shooting clarity is very small (for example, less than the lower limit), a value greater than the maximum compression rate will not be used as the compression rate of the image to be compressed.

[0061] Step 304, using the preset minimum compression rate as the to-be-compressed rate, and then proceeding to step 307;

[0062] It is understandable that if the shooting clarity is greater than the upper limit of the preset clarity range, it means that the shooting clarity is very high. At this time, if a value less than the minimum compression rate is used to compress the image to be compressed, it will not bring a significant improvement to the user's visual experience, because the clarity of the image is sufficient to meet the user's visual needs. On the contrary, if the compression rate is too small, it will cause the compressed image size to be too large. In order to avoid this situation, in an embodiment of the present application, if the shooting clarity is greater than the upper limit of the preset clarity range, the compression rate of the image to be compressed is set to the minimum compression rate, so that the storage space can be reasonably saved while ensuring that the user's visual experience is not reduced.

[0063] Step 305, obtaining an objective function; wherein the objective function is used to characterize a mapping relationship between shooting clarity and a compression rate to be compressed, and the mapping relationship is negatively correlated;

[0064] The so-called negative correlation means that the greater the shooting clarity, the smaller the compression rate; conversely, the smaller the shooting clarity, the greater the compression rate.

[0065] In some embodiments, the electronic device can determine the scene type of the image to be compressed; select a target function that matches the scene type from a plurality of different candidate functions; in this way, the compressed second image matches the shooting clarity requirement of the actual scene; wherein each of the candidate functions is used to characterize a different mapping relationship between the compression rate to be compressed and the shooting clarity.

[0066] For example, Figure 4 The mapping relationship between the compression rate and the shooting definition is shown. Figure 4 As shown, when the shooting clarity belongs to the preset clarity range, that is, greater than or equal to the lower limit value of the clarity range and less than or equal to the upper limit value of the clarity range, the electronic device can select one as the target function from the candidate functions represented by curves 401, 402, 403 and 404 according to the scene type of the image to be compressed. When the shooting clarity is less than the lower limit value of the clarity range, the compression rate to be compressed will not be higher than the preset maximum compression rate; when the shooting clarity is greater than the upper limit value of the clarity range, the compression rate to be compressed will not be lower than the preset minimum compression rate.

[0067] In some embodiments, the candidate function may be obtained by fitting in advance through a plurality of typical mapping relationships between shooting sharpnesses and to-be-compressed rates.

[0068] Step 306: Map the shooting definition to the to-be-compressed rate according to the mapping relationship represented by the objective function.

[0069] Step 307: compress the image to be compressed according to the compression rate to obtain a second image.

[0070] At present, the pixel of mobile phones is getting higher and higher, but the quality of the image is often not proportional to the pixel value. This is because: under the limitation of mobile phone cameras, the images taken often contain too much redundant information, which not only cannot increase the fineness of the image, but also takes up a lot of storage space in the mobile phone.

[0071] For example, Figure 5 As shown, image 501 and image 502 are two images taken by two mobile phones in the same scene with default parameters, wherein: image 501 has a size of 2.12 megabytes (MB), a dimension of (4608×2592), and a sensitivity (ISO) of 1250; image 502 has a size of 3.7 MB, a dimension of (4000×3000), and an ISO of 640;

[0072] It can be seen that, while the shooting quality (ie, sensitivity) of image 502 is significantly lower than that of image 501 , it occupies a much larger space than image 501 .

[0073] Image 503 is an image selected from the Internet, and its size is about 1MB, but from the human eye's perception, image 503 is not obviously more blurred than image 501 and image 502. This shows that there is considerable information redundancy in the images taken by the mobile phone.

[0074] As mentioned above, the disadvantages of information redundancy are obvious:

[0075] On the one hand, it takes up a lot of space on the phone. Images of 2MB to 4MB take up too much space in the storage. After long-term use, the number of photos in the phone increases, which greatly occupies the storage space of the phone.

[0076] On the other hand, it increases the consumption of image transmission. Images that occupy a large amount of space must either be compressed by third-party applications (Application, APP) or transmitted as "original images". When using third-party APP compression, the default compression rate often causes the image to be over-compressed and the image details are completely lost; when the original image is transmitted, a large amount of bandwidth will be consumed; and when the original image is compressed and transmitted, the transmitted image does not have the corresponding image quality (details), resulting in a poor user experience.

[0077] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0078] In the embodiment of the present application, in order to solve the above problems, it is proposed that after the front-end chip of the camera module obtains data from the image sensor (senor), it analyzes and processes the original (RAW) image, obtains the information and size ratio of the image according to the actual information of the image, and sets different compression rates according to the values, thereby reducing the damage to the image information caused by multiple compressions and format conversions; in addition, reducing the size of the RAW image in the early stage can also effectively reduce the pressure caused by subsequent format conversion and transmission of the image, thereby reducing the burden on the application (AP) side in disguise. Compared with converting the RAW image into a JPG image and then compressing it, it can save steps and calculations.

[0079] It can be understood that RAW images are the original data that the image sensor of the camera module converts the captured light source signal into a digital signal. RAW images are in an unprocessed and uncompressed format. RAW images can be conceptualized as "raw image encoding data" or more figuratively called "digital negatives."

[0080] The technical solution provided in the embodiment of the present application includes the following steps, namely Step 1 to Step 4:

[0081] Step 1, RAW image clarity analysis:

[0082] The advantage of using RAW images to analyze clarity is that RAW files are almost unprocessed information obtained directly from the CCD or CMOS, without any post-processing, and can maximize the acquisition of all image information. RAW files do not have white balance settings, but the actual data has not been changed.

[0083] Compared to compressing the already compressed JPEG format information, it can reduce the number of compressions and effectively save relevant information. Understandably, a bitmap file can no longer be restored to the clarity of a bitmap after being compressed by JPEG lossy compression. Details are permanently lost after compression. Moreover, the compression loss of JPEG is cumulative. For example, when converting from an sRGB JPEG image to an aRGB JPEG image, the details will be less.

[0084] In some embodiments, the RAW image can be converted into an RGB image through interpolation processing, and then the clarity of the RGB image is judged according to the energy gradient function or the entropy function. Among them, the energy gradient function is more suitable for real-time evaluation of image clarity, and the definition of the function is shown in the following formula (3):

[0085]

[0086] Among them, x represents the horizontal coordinate of the pixel, y represents the vertical coordinate of the pixel, and f(x,y) represents the pixel value at the pixel coordinates (x,y).

[0087] In addition, the entropy function can also achieve the same function. The entropy function based on statistical features is an important indicator to measure the richness of image information. According to information theory, the amount of information of an image f is measured by the information entropy D(f) of the image. The definition of D(f) is shown in the following formula (4):

[0088]

[0089] Among them, p i is the probability of a pixel with gray value i appearing in the image, and L is the total number of gray levels (usually 256). According to Shannon information theory, the maximum amount of information occurs when entropy is the largest. Applying this principle to the focusing process, the larger D(f) is, the clearer the image is.

[0090] like Figure 6As shown in the figure, without additional processing, the RAW image is converted into an RGB image through interpolation. The clarity of the RGB image is consistent with that of the RAW image. Therefore, the clarity of the generated RGB image can be used to reflect the clarity of the RAW image. Based on this, multiple images can be taken in the original way, and the clarity of each image can be calculated to obtain the [RAW, clarity] sequence, which is used as the training data set of the NN network in the subsequent steps.

[0091] Step 2, training based on convolutional neural network:

[0092] The NN network in the front-end chip PreISP is trained using the multiple RAW images obtained in Step 1 and the set of sharpness of each RAW image. The advantage is that the use of dedicated processing in PreISP can reduce power consumption while achieving the same function, and the sharpness judgment before ISP can effectively guide the subsequent compression steps. In some embodiments, such as Figure 7 As shown, Step 2 can be implemented by following steps 1 to 3:

[0093] Step 1: Get the original RAW images and the definition sets of these images:

[0094] The image can be acquired through the data acquisition and processing of the mobile phone camera. Assume that n RAW images are obtained, which are recorded as: p1...p n The RGB image obtained by interpolating the RAW image is used as the data reflecting the clarity. Similarly, n RGB images are obtained and recorded as: p'1...p' n , using the formula in Step 1 to obtain the corresponding clarity data: q'1...q' n . So we get the training data [RAW, clarity] n ;

[0095] Step 2: Neural network construction and training:

[0096] The construction of the neural network requires training on an existing set of data, and converting the [RAW, clarity] obtained in step 1 into n As training data for iterative training, the training network is Figure 8 As shown, a neural network architecture 81 is selected, which includes but is not limited to the AlexNet network structure, and the details of the model are not repeated here.

[0097] Step 3, clarity identification:

[0098] After the training is completed, in actual judgment, the RAW image acquired by the sensor is directly input into the trained convolutional neural network to obtain the shooting clarity.

[0099] Step 4. Select the corresponding compression rate according to the shooting clarity.

[0100] like Fig. 9 As shown, the NN network in PreISP is used to obtain a RAW image from the image sensor, and the RAW image is input into the NN network. The output data is the clarity data of the image (i.e., the clarity of this shot); the ISP determines the compression rate of the RAW image according to the clarity data, and then compresses the RAW image using the compression rate. For example, after the RAW image is converted into a YUV image, the compression rate when the YUV image is converted into JPEG encoding is used to reduce the final JPEG image size, and the image is displayed at the application layer.

[0101] In some embodiments, Fig.10 As shown in , the shooting clarity is negatively correlated with the compression rate. However, in order to avoid the problem of over-compression of the image and the problem of weak compression of the image with higher clarity, as shown in Figure 4 As shown, a maximum compression rate is set, that is, when the image is too blurry (that is, the sharpness is lower than the lower limit begin), the corresponding compression rate will not be higher than the maximum compression rate max; and a minimum compression rate is set. When the shooting sharpness is higher than the upper limit end, the minimum compression rate is maintained, that is, the existing JPG image size (about 2 to 4MB) is guaranteed. When the shooting sharpness is in the range of [begin, end], the actual compression rate is between [min, max], as shown in formula (5):

[0102]

[0103] T (sharpness) is selected based on the scene. By default, it is a straight line between the two ends. When there is a preference for clarity or compression rate in some scenes, you can make a trade-off between the two. By selecting fixed points and using spline interpolation to generate T (sharpness), the following nodes are set:

[0104] x:a=begin<x1<x2<...<x m =b

[0105] y:y0=Max,y1,y2,...,y m =Min

[0106] The spline curve S(x) is a piecewise defined formula. Given m+1 data points, there are m intervals in total, and the cubic spline equation satisfies the following conditions:

[0107] a. In each segment interval [x i ,x i+1 ](i=0,1,…,m-1,x is increasing), S(x)=S i (x) is always a cubic polynomial;

[0108] b.Satisfy S(x i )=y i (i=0,1,…,m);

[0109] cS(x), derivative S'(x), and second-order derivative S″(x) are all continuous in the interval [a,b], that is, the S(x) curve is smooth, so the m cubic polynomials can be written piecewise as:

[0110] S i (x) = a i +b i (xx i )+c i (xx i ) 2 +d i (xx i ) 3 , i=0,1,…,m-1;

[0111] Where ai, bi, ci, and di represent 4m unknown coefficients. By obtaining the correlation coefficients, T (sharpness) can be calculated segment by segment, which will not be repeated here.

[0112] Among the images taken by mobile phones, there are often some images whose quality is not very clear, but their image size exceeds 1MB. By utilizing the technical solution provided in the embodiments of the present application, it is possible to effectively reduce invalid pixels in the image and reduce the space occupied by the image to a level consistent with the image quality, thereby effectively saving mobile phone space and improving user experience.

[0113] In an embodiment of the present application, in view of the phenomenon that images taken by mobile phones contain a lot of redundant information, a method of reducing the image size by deleting invalid information based on the actual clarity of the captured image and retaining valid information is proposed, that is, compression based on image clarity.

[0114] In the embodiment of the present application, the processing is performed in advance in the processing stage and directly inside the PreISP chip. The RAW image generated by the front-end sensor is directly analyzed and judged (RAW image is the image information directly generated by the sensor, which can maximize the retention of image detail information), and the NN network is used to judge the redundancy of image information, thereby guiding the ISP to directly generate a JPG image with a larger information density per unit size, effectively reducing the space occupied by the mobile phone image while ensuring the image quality.

[0115] In some embodiments, the training of NN networks, where different networks have different training effects, can be further subdivided;

[0116] In some embodiments, the degree of compression in different scenarios can be customized according to user preferences;

[0117] In some embodiments, the compression aspect can be further expanded to perform compression schemes of different degrees according to the importance of different contents in the same scene. In other words, in the same image, different regions have different clarity, and compression is performed in a targeted manner according to the clarity of each region.

[0118] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the modules included and the units included in the modules, and can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0119] Fig.11 Schematic diagram of the structure of the image processing device according to the embodiment of the present application. Fig.11 As shown, the device 110 includes:

[0120] An analysis module 111 is used to analyze a first image of the image to be compressed to obtain shooting clarity; wherein the first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image;

[0121] A determination module 112, configured to determine a compression rate of the image to be compressed according to the shooting definition;

[0122] The compression module 113 is used to compress the image to be compressed according to the compression rate to obtain a second image.

[0123] In some embodiments, the analysis module 111 is used to: input the first image into a pre-trained target neural network so that the target neural network extracts image features of the first image, and analyzes the image features to obtain the shooting clarity.

[0124] In some embodiments, the training process of the target neural network includes: performing interpolation processing on each sample first image to obtain a corresponding sample RGB image; determining the clarity of the sample RGB image based on the grayscale value of the pixels of the sample RGB image; using the clarity of each of the sample RGB images as the sample clarity of the corresponding sample first image, and using each of the sample shooting clarity and the corresponding sample first image to train the initial neural network to obtain the target neural network.

[0125] In some embodiments, the determination module 112 is used to: when the shooting clarity is less than the lower limit of the preset clarity range, use the preset maximum compression rate as the compression rate to be compressed; when the shooting clarity is greater than the upper limit of the preset clarity range, use the preset minimum compression rate as the compression rate to be compressed.

[0126] In some embodiments, the determination module 112 is also used to: obtain an objective function when the shooting clarity belongs to the preset clarity range; wherein the objective function is used to characterize the mapping relationship between the shooting clarity and the compression rate, and the mapping relationship is negatively correlated; and map the shooting clarity to the compression rate according to the mapping relationship represented by the objective function.

[0127] In some embodiments, the determination module 112 is used to: determine the scene type of the image to be compressed; select a target function that matches the captured scene type from a plurality of different candidate functions; wherein each of the candidate functions is used to characterize a different mapping relationship between the compression rate to be compressed and the capture clarity.

[0128] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0129] It should be noted that in the embodiments of this application Fig.11The division of modules in the image processing device shown is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0130] It should be noted that in the embodiment of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0131] An embodiment of the present application provides an electronic device, Fig.12 Schematic diagram of the hardware entity of the electronic device of the embodiment of the present application, such as Fig.12 As shown, the electronic device 120 includes a memory 121 and a processor 122, wherein the memory 121 stores a computer program that can be run on the processor 122, and the processor 122 implements the steps in the method provided in the above embodiment when executing the program.

[0132] It should be noted that the memory 121 is configured to store instructions and applications executable by the processor 122, and can also cache data to be processed or processed by the processor 122 and various modules in the electronic device 120 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0133] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0134] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0135] The present application embodiment provides an image processing module, such as Fig.13 As shown, the image processing module 13 includes a PreISP circuit 131 and an ISP circuit 132, wherein:

[0136] The PreISP circuit 131 is used to analyze a first image of the image to be compressed to obtain shooting clarity; wherein the first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image;

[0137] The ISP circuit 132 is used to determine the compression rate of the image to be compressed according to the shooting clarity; and compress the image to be compressed according to the compression rate to obtain a second image.

[0138] It should be noted here that the description of the above device, storage medium, computer program and image processing module embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device, storage medium, computer program and image processing module embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0139] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0140] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0141] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0143] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0144] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0145] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0146] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0147] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0148] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0149] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0150] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Analyze a first image of the image to be compressed to obtain shooting clarity; wherein the first image is a RAW image of the image to be compressed or an image in a quality improvement processing stage of the RAW image; the image in the quality improvement processing stage of the RAW image is an image after the RAW image is corrected and enhanced; Determining a compression rate of the image to be compressed according to the shooting definition; According to the compression rate, compressing the image to be compressed to obtain a second image; The step of determining the compression rate of the first image according to the shooting definition includes: When the shooting definition is less than a lower limit of a preset definition range, a preset maximum compression rate is used as the to-be-compressed rate; When the shooting definition is greater than the upper limit of the preset definition range, a preset minimum compression rate is used as the to-be-compressed rate.

2. The method according to claim 1, characterized in that The analyzing the first image of the image to be compressed to obtain the shooting definition includes: The first image is input into a pre-trained target neural network so that the target neural network extracts image features of the first image and analyzes the image features to obtain the shooting clarity.

3. The method according to claim 2, characterized in that The training process of the target neural network includes: Perform interpolation processing on each sample first image to obtain the corresponding sample red, green and blue RGB image; Determining the clarity of the sample RGB image according to the grayscale values ​​of the pixels of the sample RGB image; The clarity of each of the sample RGB images is used as the sample shooting clarity of the corresponding sample first image, and the initial neural network is trained using each of the sample shooting clarity and the corresponding sample first image to obtain the target neural network.

4. The method according to claim 1, characterized in that The step of determining the compression rate of the image to be compressed according to the shooting definition further includes: When the shooting definition belongs to the preset definition range, obtaining an objective function; wherein the objective function is used to characterize a mapping relationship between the shooting definition and the compression rate to be compressed, and the mapping relationship is negatively correlated; The shooting definition is mapped to the to-be-compressed rate according to the mapping relationship represented by the objective function.

5. The method according to claim 4, characterized in that The obtaining of the objective function comprises: Determining the scene type of the image to be compressed; A target function matching the scene type is selected from a plurality of different candidate functions; wherein each of the candidate functions is used to characterize a different mapping relationship between a compression rate to be compressed and a shooting definition.

6. An image processing module, characterized in that: include: A front-end image signal processing PreISP circuit is used to analyze a first image of the compressed image to obtain shooting clarity; wherein the first image is a RAW image or an image in a quality improvement processing stage of the RAW image; the image in the quality improvement processing stage of the RAW image is an image after the RAW image is corrected and enhanced; An image signal processor ISP circuit is used to determine the compression rate of the image to be compressed according to the shooting clarity; and compress the image to be compressed according to the compression rate to obtain a second image; wherein, determining the compression rate of the first image according to the shooting clarity includes: when the shooting clarity is less than the lower limit of a preset clarity range, taking the preset maximum compression rate as the compression rate; when the shooting clarity is greater than the upper limit of the preset clarity range, taking the preset minimum compression rate as the compression rate.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images