Image processing methods, processors, processing devices, software products, and storage media

Abstract

This application relates to the field of image processing technology, and provides an image processing method, processor, processing device, program product, and storage medium. The image processing method includes: based on an image to be processed, using a neural network model to predict the interpolation direction information corresponding to a target pixel in the image to be processed; all pixels in the image to be processed contain K categories of color values, each pixel contains only one category of color values, and the other K-1 categories of color values ​​for each pixel are to be calculated; the target pixel is the pixel in the image to be processed whose target category color value is to be calculated; for each pixel in the image to be processed, the other K-1 categories of color values ​​for that pixel are interpolated to obtain a result image; for each target category color value of each target pixel, interpolation calculation is performed according to the interpolation direction information corresponding to that target category of the pixel. This image processing method can obtain a result image with fewer false colors and higher resolution.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image processing method, processor, processing device, program product, and storage medium. Background Technology

[0002] Currently, most image sensors can only sense light intensity, not color. To obtain color images, color filters need to be installed on the pixels of the image sensor. For example, to obtain an RGB image, filters for R (red), G (green), and B (blue) colors are required. However, installing three color filters on the same pixel is too costly. Therefore, a common practice is to install only one color filter on each pixel, and to arrange the various color filters on the pixels of the image sensor in a fixed pattern, alternating between them.

[0003] Reference Figure 1 , Figure 1 In the upper sub-image, the small cuboid is divided into two layers. The lower layer is all gray, representing the pixels of the image sensor. The upper layer has three colors: black, gray, and white, representing the R, G, and B color filters, respectively. These filters are arranged in a way called the Bayer pattern, that is, four filters per group. In each group, there are two G filters on one diagonal and one R filter and one B filter on the other diagonal.

[0004] After processing in the above way, although the number of filters is reduced, each pixel in the raw image generated by the image sensor (called a RAW image) contains only one color value. Visually, the raw image contains a lot of "mosaic". (Continue to refer to...) Figure 1 , Figure 1 The middle sub-image is a 5×5 local area of ​​the RAW image, where each pixel contains only R, G, or B values, and the color values ​​are arranged in a Bayer array.

[0005] However, in color images typically used for display or other processing, each pixel contains multiple color values. For example, each pixel in an RGB image contains R, G, and B color values. Therefore, interpolation algorithms are needed to "fill in" the missing color values ​​for each pixel, eliminating the "mosaic" effect in the RAW image, thus facilitating subsequent image use. Such interpolation algorithms are generally called demosaic algorithms. (Continue to refer to...) Figure 1 , Figure 1 The sub-image below is the RGB image obtained after performing a demosaic algorithm on the RAW image. The pixel in the upper left corner originally only contained the R value, but now the G and B values ​​have been calculated through interpolation. The same applies to the pixels in other positions.

[0006] Traditional demosaic algorithms typically perform interpolation calculations based on manually defined rules. However, these rules are ineffective in handling complex image content, often resulting in incorrect color values. This leads to problems such as false color and reduced resolution in the resulting color image, which is particularly noticeable in high-frequency texture areas of the image. Summary of the Invention

[0007] The purpose of this application is to provide an image processing method, processor, processing device, program product, and storage medium to improve the above-mentioned technical problems.

[0008] To achieve the above objectives, this application provides the following technical solution:

[0009] In a first aspect, embodiments of this application provide an image processing method, comprising: predicting interpolation direction information corresponding to a target pixel in the image to be processed using a neural network model based on an image to be processed; wherein, all pixels in the image to be processed contain color values ​​of K categories, where K is an integer greater than 1, each pixel contains only one category of color values, and the color values ​​of the other K-1 categories of each pixel are to be calculated, the target pixel is the pixel in the image to be processed whose target category color value is to be calculated, the target category is at least one of the K categories, and each target category corresponds to one interpolation direction information; for each pixel in the image to be processed, interpolating the color values ​​of the other K-1 categories of the pixel based on the color values ​​of the pixels in its neighborhood, to obtain a result image; wherein, for the color value of each target category of each target pixel, interpolation calculation is performed based on the interpolation direction information corresponding to the target category of the pixel.

[0010] The image to be processed in the above image processing method can be a RAW image generated by an image sensor, and the resulting image can be a color image used for device display or other processing. Therefore, this method can be regarded as an improved scheme of the de-mosaic algorithm. The advantages of this method are:

[0011] First, it utilizes a neural network model to predict interpolation direction information. The neural network model can be regarded as a complex interpolation rule determined based on training data. Its expressive power is significantly stronger than that of manually set interpolation rules, and it has good generalization ability, which is conducive to improving interpolation accuracy.

[0012] Secondly, the aforementioned neural network model predicts the interpolation direction rather than directly predicting the color value. For digital images, the interpolation direction is very limited (e.g., horizontal, vertical, diagonal directions). Therefore, this model can be regarded as a classification model rather than a regression model. Relatively speaking, classification models are more likely to achieve higher prediction accuracy and are less likely to output unexpected prediction results.

[0013] In summary, the image processing method described above can produce images with fewer false colors and higher resolution, which naturally helps to improve the processing effect of subsequent applications on the resulting images.

[0014] In one implementation of the first aspect, the interpolation direction information includes the probability corresponding to each potential interpolation direction. The step of performing interpolation calculation based on the interpolation direction information corresponding to the target category of the pixel includes: determining the potential interpolation direction with the highest probability as the actual interpolation direction, and performing interpolation calculation based on the actual interpolation direction to obtain the color value of the target category of the pixel; and / or, performing interpolation calculation in each potential interpolation direction to obtain multiple initial color values, and performing a weighted summation of the multiple initial color values ​​with the probability corresponding to each potential interpolation direction as the weight to obtain the color value of the target category of the pixel.

[0015] The above implementation presents two methods for utilizing interpolation direction information: First, select the optimal (from the prediction results) interpolation direction from all possible directions; this method has high interpolation efficiency. Second, perform interpolation in all possible directions and then take a weighted average of the interpolation results; this method has high interpolation accuracy. Of course, these two methods can also be combined to give the interpolation process greater flexibility: for example, some target pixel color values ​​can be interpolated using method one, while others can be interpolated using method two; or, for example, some target category color values ​​can be interpolated using method one, while others can be interpolated using method two.

[0016] In one implementation of the first aspect, the step of predicting the interpolation direction information corresponding to the target pixel in the image to be processed using a neural network model based on the image to be processed includes: obtaining attribute information of the image to be processed; determining the neural network model corresponding to the image to be processed from a plurality of candidate neural network models based on the attribute information; and predicting the interpolation direction information based on the image to be processed using the corresponding neural network model.

[0017] The same neural network model can be used to process all the images to be processed. This approach is relatively simple in logic, but the model may not be the best match for some of the images to be processed. In the above approach, multiple alternative neural network models are pre-set, and the most suitable model is assigned to the image to be processed based on its attribute information for predicting the interpolation direction information, which helps to improve the prediction accuracy.

[0018] In one implementation of the first aspect, the attribute information includes at least one of the following: the arrangement of color values ​​in the image to be processed, the photosensitivity of the image to be processed, and the brightness of the image to be processed.

[0019] In one implementation of the first aspect, the number of pixels in the image to be processed that contain the color value of the target category is greater than the number of pixels that contain the color value of any non-target category, and when performing interpolation calculations on the pixels of the image to be processed, the color value of the target category of the target pixel is calculated first.

[0020] For example, in an image to be processed using a Bayer array, the number of G pixels (pixels containing only G values) is twice that of R pixels (pixels containing only R values) and B pixels (pixels containing only B values). In this case, the G value can be used as the color value of the target category. A neural network model is used to predict the interpolation direction information at R and B pixels (G values ​​need to be calculated for both R and B pixels), and the G value to be interpolated is calculated based on the interpolation direction information. For the interpolation calculation of R and B values, the neural network can be used to predict the interpolation direction information. Furthermore, during the interpolation calculation, the G value can be calculated first, followed by the R and B values ​​(the calculation results of the G value may be used when calculating the R and B values).

[0021] Because the human eye is more sensitive to green than red and blue, the image being processed contains more G pixels (including the actual acquired G values), and the interpolation accuracy required for G values ​​is higher than that for R and B values. Therefore, a neural network model can be specifically used to improve the interpolation accuracy of G values, while the interpolation of R and B values ​​can be performed without a neural network model to maintain interpolation efficiency. Furthermore, according to the interpolation order above, the interpolation accuracy of R and B values ​​can also be guaranteed because they can utilize the high-precision interpolation results of G values.

[0022] In one implementation of the first aspect, the color values ​​of the K categories are R value, G value and B value, the color value of the target category is G value, the target pixel is the pixel containing R value and the pixel containing B value in the image to be processed, and the color values ​​in the image to be processed are arranged in a Bayer array manner.

[0023] The Bayer array is currently the most widely used RGB value arrangement method. The image processing method of this application can effectively process images of the Bayer array, indicating that it has high practical value.

[0024] Secondly, embodiments of this application provide an image signal processor, comprising: a front-end module for performing a RAW image processing algorithm on a first RAW image generated by an image sensor to obtain a second RAW image; an intermediate module for performing interpolation calculations on the second RAW image using a method provided in the first aspect or any possible implementation of the first aspect to obtain a first color image; and a back-end module for performing a color image processing algorithm on the first color image to obtain a second color image.

[0025] Thirdly, embodiments of this application provide an image processing apparatus, including: a neural network calculation module, an interpolation calculation module, a storage module, and a control module; wherein, the neural network calculation module is used to predict, based on an image to be processed, the interpolation direction information corresponding to a target pixel in the image to be processed using a neural network model; wherein, all pixels in the image to be processed contain color values ​​of K categories, where K is an integer greater than 1, each pixel contains only one category of color values, and the other K-1 categories of color values ​​for each pixel are to be calculated, the target pixel is the pixel in the image to be processed whose target category of color value is to be calculated, and the target category is at least one of the K categories, each Each target category corresponds to one interpolation direction information; the interpolation calculation module is used to calculate the color values ​​of the pixel in the other K-1 categories for each pixel of the image to be processed, based on the color values ​​of the pixels in its neighborhood, to obtain the result image; wherein, for the color value of each target category of each target pixel, interpolation calculation is performed based on the interpolation direction information corresponding to that target category of the pixel; the storage module is used to store the image to be processed, the interpolation direction information, the result image, the neural network model, and the driver program; the control module is used to execute the driver program, which drives the neural network calculation module and the interpolation calculation module to perform calculations during runtime.

[0026] Fourthly, embodiments of this application provide an image processing apparatus, including a storage module and a processing module. The storage module stores computer program instructions, which are read and executed by the processing module to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0027] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0028] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This illustrates the working principle of the Bayer array and the demosaic algorithm;

[0031] Figure 2 This illustrates an application scenario of the image processing method provided in an embodiment of this application;

[0032] Figure 3 The flowchart of an image processing method provided in an embodiment of this application is shown;

[0033] Figure 4 This illustrates a method for interpolating the G value of an R pixel;

[0034] Figure 5 This demonstrates a method for interpolating the R and B values ​​of a G pixel;

[0035] Figure 6 This illustrates a method for interpolating the B value of pixel R and the R value of pixel B;

[0036] Figure 7 The structure of an image processing apparatus provided in an embodiment of this application is shown;

[0037] Figure 8 The structure of another image processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0038] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0039] The image processing method in this application embodiment is, in general, a demosaic method based on a neural network model that interpolates a single-channel RAW image to obtain a multi-channel color image. The color image obtained by this method has high quality and is significantly better than the processing results of traditional demosaic algorithms for RAW images.

[0040] Figure 2 This illustrates an application scenario of the image processing method implemented in this application. (Refer to...) Figure 2 The image sensor of the camera outputs an image in RAW format, which we can call the first RAW image. The first RAW image is a single-channel image, where each pixel contains only one color value. For example, the first RAW image could be... Figure 1 The middle sub-graph.

[0041] The first RAW image is of poor quality and the color value of each pixel is incomplete. Therefore, it usually needs to be processed by an Image Signal Processor (ISP) before it can be used for display or other operations.

[0042] An ISP can be functionally divided into three parts: the ISP front-end module, the ISP middle module, and the ISP back-end module. The ISP can be implemented using a dedicated chip.

[0043] The ISP front-end module is used to perform RAW image processing algorithms on the first RAW image to obtain the second RAW image. The first RAW image may be a photo or video frame captured by an image sensor. The RAW image processing algorithm here may be noise reduction, brightness adjustment, white balance adjustment, etc.

[0044] The ISP intermediate module is used to execute the image processing method proposed in the embodiments of this application (details described below), and to perform interpolation calculations on the second RAW image to obtain a first color image. The first color image is a multi-channel image, and each pixel contains complete multiple color values. For example, the first color image can be... Figure 1 The sub-diagram below can be simply summarized as follows: the ISP intermediate module implements the de-mosaic function.

[0045] The ISP post-module is used to perform color image processing algorithms on the first color image to obtain a second color image, which can be used for display or further processing. The color image processing algorithms here can include noise reduction, brightness adjustment, color adjustment, etc.

[0046] for Figure 2 The following issues also need to be explained:

[0047] Firstly, the ISP front-end module and the ISP back-end module may perform some similar operations, such as noise reduction, but the objects they process are different, so the noise reduction algorithms they use may also be different.

[0048] Secondly, in Figure 2 In this embodiment, the ISP front-end module, ISP middle module, and ISP back-end module are all hardware modules. Specifically, the ISP middle module is formed by hardware-based implementation of the image processing method described in this application (hardware-based implementation can be understood as directly describing the method steps using hardware circuitry). However, in some alternatives, the ISP middle module may also be implemented purely in software or a combination of software and hardware. For example, the original de-mosaic algorithm in a certain ISP may be disabled, and the image processing method described in this application may be implemented externally in software. After de-mosaicing, the processing structure is then sent back to the ISP back-end module for further processing.

[0049] Third, although the ISP intermediate module is part of the ISP, it does not mean that when it executes the image processing method in the embodiments of this application, the module will only use the internal hardware and software resources of the ISP. For example, the ISP intermediate module may use storage resources, but in the design of some ISPs, storage resources are shared with other processing chips of the device (e.g., central processing unit, CPU), and the ISP does not contain storage resources.

[0050] Fourth, the image processing methods in the embodiments of this application are not limited to applications in... Figure 2 In the illustrated scenario, for example, an existing RGB image is first sampled and restored to a single-channel RAW image. Then, the image processing method described in this embodiment is applied to the restored RAW image, and the resulting image is compared with the existing RGB image to test the interpolation effect of the method. This process can be executed purely in software on a computer, without the need for... Figure 2 The same method will be deployed in the ISP.

[0051] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0052] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] The terms “first,” “second,” etc., are used only to distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance, nor as requiring or implying any such actual relationship or order between these entities or operations.

[0054] Figure 3 This application illustrates a flowchart of an image processing method provided in an embodiment. This method may be implemented in software, hardware, or a combination of both. The entity executing this method can also be of various types, such as... Figure 2 The ISP middleware module in the middleware, Figure 7 Image processing device in Figure 8Image processing devices, etc. (In the introduction...) Figure 3 At this time, it is advisable to temporarily ignore the differences in details between different implementation methods and focus on explaining the functions achieved by the method steps.

[0055] Reference Figure 3 The method includes:

[0056] Step S110: Based on the image to be processed, use a neural network model to predict the interpolation direction information corresponding to the target pixels in the image to be processed.

[0057] Step S120: For each pixel in the image to be processed, interpolate the color values ​​of the pixel in the other K-1 categories based on the color values ​​of its neighboring pixels to obtain the result image. Specifically, for the color value of each target category of each target pixel, interpolation is performed based on the interpolation direction information corresponding to that target category.

[0058] The following is a detailed explanation of steps S110 and S120:

[0059] The image to be processed in step S110 is a single-channel image, where each pixel contains only one category of color value. The total number of pixels in the image to be processed contains K categories of color values ​​(K is an integer greater than 1). By executing steps S110 and S120, the K-1 categories of color values ​​that each pixel of the image to be processed does not contain are calculated. In this way, the image to be processed is converted into a color image with K channels, which is the result image in step S120.

[0060] The arrangement of color values ​​in the image to be processed is not limited, but in order to facilitate interpolation calculations, various color values ​​can be arranged in a fixed pattern to ensure that a pixel containing any color value has pixels containing the same color value distributed in its neighborhood.

[0061] For example, the image to be processed can be Figure 2 The second RAW image in the image to be processed has K categories of color values, namely R, G, and B values ​​(K=3). Let's simply refer to the pixels in the image containing R, G, and B values ​​as R pixels, G pixels, and B pixels, respectively. The G and B values ​​of R pixels, the R and B values ​​of G pixels, and the R and G values ​​of B pixels need to be calculated. If these color values ​​are all calculated using steps S110 and S120, a result image containing the R, G, and B channels can be obtained, such as... Figure 2 The first RGB image in the image.

[0062] In this example, the R, G, and B values ​​in the image to be processed can be arranged in a Bayer array (sometimes abbreviated as RGGB), such as... Figure 1The middle sub-image shows what is referred to hereafter as the image to be processed using a Bayer array. The Bayer array is currently the most widely used RGB value arrangement in RAW images. The image processing method of this application embodiment can effectively process images using a Bayer array, thus possessing high practical value.

[0063] It is understandable that the color values ​​in the image to be processed are not necessarily RGB values, and the arrangement of RGB values ​​in the image to be processed is not limited to the Bayer array. For example, it may also be R value, Y value (yellow) and B value (arranged in RYYB), or R value, W value (white) and B value (arranged in RWWB), etc. That is, the image processing method of this application embodiment has high universality.

[0064] The K color value categories in the image to be processed can be divided into two categories: target categories and non-target categories. The target category is at least one of the K color value categories, and the target category can be pre-defined. Furthermore, the pixels in the image to be processed can also be divided into two categories: target pixels and non-target pixels. Target pixels are the pixels in the image to be processed whose color values ​​are to be calculated for the target category, and all pixels in the image to be processed other than target pixels are non-target pixels.

[0065] For example, for an image to be processed using a Bayer array, if the target category is green, then the non-target categories are red and blue. In this case, the target pixels are the pixels in the image to be processed whose G value is to be calculated, namely R pixels and B pixels, and the non-target pixels are G pixels. If the target category is red, green, and blue, then there are no non-target categories. In this case, the target pixels are all the pixels in the image to be processed, and there are no non-target pixels.

[0066] For each target pixel and each target category's color value, the neural network model in step S110 predicts an interpolation direction information. In step S120, this interpolation direction information is used to calculate the target pixel's color value for that target category. Here, interpolation direction information generally refers to information related to the interpolation direction used in the interpolation calculation; further examples will be provided later.

[0067] For color values ​​of non-target categories of target pixels, or color values ​​of non-target pixels, the neural network model will not be used to predict interpolation direction information. Note that this does not mean that these color values ​​will not use interpolation direction information during calculation in step S120; it simply means that even if interpolation direction information is used, the source of this information is not predicted by the neural network model.

[0068] For example, for the image to be processed by the Bayer array, if the target category is green, the neural network model in step S110 will predict an interpolation direction for the G value for each R pixel and B pixel in the image to be processed; if the target category is red, green and blue, the neural network model in step S110 will predict three interpolation direction information for the R value, G value and B value for each pixel in the image to be processed.

[0069] The neural network model in step S110 takes at least the image to be processed as input and outputs at least interpolation direction information as output. The specific type of neural network model is not limited; for example, it can be a convolutional neural network, a recurrent neural network, an artificial neural network, or a combination thereof. The specific structure of the neural network model is also not limited; taking a convolutional neural network as an example, it can contain several convolutional layers, and may also contain pooling layers, fully connected layers, and other structures. The neural network model has been trained before executing step S110; possible training methods will be described later.

[0070] It should be noted that the image to be processed may undergo some format conversion before being input into the neural network model to facilitate network processing. For example, for the image to be processed in a Bayer array, its size is H×W (height×width). Before being input into the convolutional neural network, it may be converted into a tensor of size H / 2×W / 2×4. That is, this tensor has 4 channels of size H / 2×W / 2, namely the R channel, two G channels and the B channel. Each channel contains only one corresponding color value (in the Bayer array, every 4 color values ​​form a small unit, so such conversion can be easily achieved by sampling the image to be processed).

[0071] For a target pixel with a specific target category color value, the corresponding interpolation direction information can include the probability corresponding to each potential interpolation direction, and the sum of these probabilities is 1. For example, if there are S potential interpolation directions (S is an integer greater than 1), then each interpolation direction information includes the probability corresponding to S potential interpolation directions, and the sum of these probabilities is 1. A potential interpolation direction can be understood as the interpolation direction that might be used when performing interpolation calculations on the target pixel's color value of that target category in step S120, and the probability corresponding to a potential interpolation direction can be understood as the probability that a potential interpolation direction is used as the actual interpolation direction. Of course, in some alternative solutions, the interpolation direction information can also include a coefficient corresponding to each potential interpolation direction, which can represent the probability corresponding to each potential interpolation direction to a certain extent, but the sum of these coefficients is not necessarily 1.

[0072] For example, for an image to be processed using a Bayer array, if the target category is green and there are four potential interpolation directions, the interpolation direction information output by the convolutional neural network can be a tensor of size H / 2 × W / 2 × 8. Of the eight channels of this tensor, four channels represent the probabilities corresponding to the potential interpolation directions of pixels R (there are H / 2 × W / 2 pixels in total, and each R pixel corresponds to four potential interpolation directions, therefore the total probabilities of the potential interpolation directions for pixels R are H / 2 × W / 2 × 4). The other four channels represent the probabilities corresponding to the potential interpolation directions of pixels B. Furthermore, for a pixel R at coordinates (0,0) in the image to be processed, its four probability values ​​in the interpolation direction information might be (0.5, 0.2, 0.2, 0.1), summing to 1; for a pixel R at coordinates (0,2), its four probability values ​​in the interpolation direction information might be (0.5, 0.4, 0.1, 0), summing to 1; and so on.

[0073] In step S120, for each pixel of the image to be processed, in addition to the color value it already contains, interpolation is performed to calculate the color values ​​of its other K-1 categories, resulting in the final image. For example, for pixel R, interpolation is performed to calculate the G and B values; for pixel G, interpolation is performed to calculate the R and B values; and for pixel B, interpolation is performed to calculate the R and G values.

[0074] The specific method of interpolation is to select some or all of the pixels in the neighborhood of the current pixel (referring to the pixel currently being processed, which can be any pixel in the image to be processed) and perform a weighted sum of their color values. Regarding the interpolation process, the following points should be noted:

[0075] Firstly, a neighborhood represents a small region centered on the current pixel. Its specific shape and size are not limited; the following text primarily uses a 5×5 square neighborhood as an example. Figure 1 The middle sub-image is a 5×5 neighborhood of an R pixel (the R pixel is located at the center of the neighborhood).

[0076] Secondly, the order of interpolation calculation is not limited. It can be pixel-by-pixel: for each pixel, after calculating the color values ​​of the K-1 categories that need to be inserted, the calculation for the next pixel begins; or color category-by-color: for each category of color value, after completing the calculation at all pixels where the color value of that category needs to be inserted, the calculation for the color value of the next category begins; of course, other calculation orders may also be used, as shown in the examples below.

[0077] Third, during interpolation calculations, previously obtained interpolation results can be used (but are not always necessary). For example, if interpolation is performed sequentially by color category, when calculating the color value of the first category, only the color values ​​in the neighborhood of the current pixel in the image to be processed can be used; when calculating the color value of the second category, in addition to using the color values ​​in the neighborhood of the current pixel in the image to be processed, the newly calculated color values ​​of the first category in the neighborhood of the current pixel may also be used; and so on. For example, for the image to be processed in a Bayer array, if the G value of pixel R is calculated first, only the color values ​​in the 5×5 neighborhood of pixel R in the image to be processed can be used. After the G value is calculated, if the R value of pixel G is calculated, in addition to using the color values ​​in the 5×5 neighborhood of pixel G in the image to be processed, the previously calculated G value for pixel R can also be used.

[0078] Although the calculation method for each color value is similar (interpolation calculation based on the color values ​​of neighboring pixels), as mentioned before, for the target category color value of the target pixel, the interpolation direction information predicted by the neural network model must be combined during the interpolation calculation. For the non-target category color value of the target pixel, or the color value of a non-target pixel, since the neural network model does not predict the interpolation direction information for it, other calculation methods must be adopted, such as selecting the color value in the neighborhood for calculation according to established rules.

[0079] Furthermore, if the interpolation direction information includes the probability corresponding to each potential interpolation direction, then for the color value of any target pixel of any target category, there are at least two interpolation calculation methods:

[0080] (1) The potential interpolation direction with the highest probability is determined as the actual interpolation direction, and interpolation calculation is performed according to the actual interpolation direction to obtain the color value of the target pixel for the target category.

[0081] For example, if there are 4 potential interpolation directions with corresponding probabilities of (0.5, 0.2, 0.2, 0.1), the first direction (probability 0.5) can be determined as the actual interpolation direction. Within the neighborhood of the current pixel, the color value to be interpolated is calculated by weighting the color values ​​in the first direction.

[0082] (2) Interpolation calculation is performed in each potential interpolation direction to obtain multiple initial color values. The multiple initial color values ​​are weighted and summed using the probability corresponding to each potential interpolation direction as the weight to obtain the color value of the target category of the pixel.

[0083] For example, if there are four potential interpolation directions with corresponding probabilities of (0.5, 0.2, 0.2, 0.1), then in the neighborhood of the current pixel, the color values ​​in these four directions can be weighted and calculated (the weights can be pre-set) to obtain four initial color values ​​A1, A2, A3, and A4. Then, 0.5×A1 + 0.2×A2 + 0.2×A3 + 0.1×A4 is calculated to obtain the color value to be interpolated.

[0084] In summary: Method (1) selects the optimal (which appears optimal based on the prediction results but may not be the actual optimal) interpolation direction from all possible interpolation directions, resulting in higher interpolation efficiency; Method (2) performs interpolation in all possible interpolation directions and then takes a weighted average of the interpolation results, resulting in higher interpolation accuracy. Of course, these two methods can also be combined to give the interpolation process greater flexibility: for example, some target pixel color values ​​can be interpolated using Method 1, while others can be interpolated using Method 2; or, for example, some target category color values ​​can be interpolated using Method 1, while others can be interpolated using Method 2, and so on.

[0085] Below, we will use a practical example to illustrate the complete interpolation process:

[0086] For an image to be processed using a Bayer array, if the target category is green, interpolation can be performed in the following three steps to obtain the resulting image:

[0087] Step 1: For each R pixel or B pixel in the image to be processed, calculate the G value of the pixel by interpolation based on the color value of the pixels in its neighborhood and the corresponding interpolation direction information. The image obtained after interpolation is called the first intermediate image.

[0088] Figure 4 This demonstrates a method for interpolating the G value of R pixels within a 5×5 neighborhood of the image to be processed. Figure 4 It consists of 4 subgraphs, arranged from top to bottom and left to right, corresponding to 4 potential interpolation directions: top, right, bottom, and left. The numbers in the graph represent the preset weights.

[0089] For example, assuming the neural network model predicts probabilities of (0.5, 0.2, 0.2, 0.1) for these four directions, and using the interpolation method (1) described above, the actual interpolation direction can be determined to be upward. The coordinates of the center R pixel are marked as (0,0), and the formula for calculating the G value of the R pixel is:

[0090] G(0,0)=0.5×R(0,0)+1×G(0,-1)-0.5×R(0,-2).

[0091] If we follow the interpolation method (2) described above, the formula for calculating the G value of pixel R is:

[0092] G(0,0)=0.5×(0.5×R(0,0)+1×G(0,-1)-0.5×R(0,-2)) / / above

[0093] +0.2×(0.5×R(0,0)+1×G(1,0)-0.5×R(2,0)) / / right side

[0094] +0.2×(0.5×R(0,0)+1×G(0,1)-0.5×R(0,2)) / / below

[0095] +0.1×(0.5×R(0,0)+1×G(-1,0)-0.5×R(-2,0)) / / Left side

[0096] The calculation method for the G value of B pixels is similar to that of R pixels, and will not be repeated here.

[0097] Step 2: For each G pixel in the first intermediate image, interpolate the R and B values ​​of that pixel based on the color values ​​of its neighboring pixels. The resulting image is called the second intermediate image. Here, the G pixels in the first intermediate image refer to those pixels in the first intermediate image that correspond to the G pixels in the image to be processed.

[0098] Figure 5 This diagram illustrates a method for interpolating the R and B values ​​of pixel G within a 5×5 neighborhood of a first intermediate image. Since red and blue are not target categories, the values ​​can be calculated directly using the following formula, where the numbers in the diagram represent pixel indices:

[0099] R[1]=G[1]+0.5×(R[0]-G[0])+0.5×(R[2]-G[2])

[0100] B[1]=G[1]+0.5×(B[3]-G[3])+0.5×(B[4]-G[4])

[0101] In this formula, G[0], G[2], G[3], and G[4] are calculated in step 1. The 0.5 in the formula is a preset coefficient, and the difference in color values ​​represents the gradient information in the first intermediate image. Understandably, this formula may vary slightly depending on the position of the G pixel.

[0102] Step 3: For each R pixel in the second intermediate image, calculate the B value of the pixel by interpolation based on the color values ​​of its neighboring pixels; and for each B pixel in the second intermediate image, calculate the R value of the pixel by interpolation based on the color values ​​of its neighboring pixels; wherein, the R pixels in the second intermediate image refer to those pixels in the second intermediate image that correspond to the R pixels in the image to be processed, and the B pixels in the second intermediate image refer to those pixels in the second intermediate image that correspond to the B pixels in the image to be processed.

[0103] Figure 6 The diagram includes two sub-images. The left sub-image shows a method for interpolating the B value of a pixel (R) within a 5×5 neighborhood of the second intermediate image. The right sub-image shows another method for interpolating the R value of a pixel (B) within a 5×5 neighborhood of the second intermediate image. Since red and blue are not target categories, the calculation can be performed directly using the following formula, where the numbers in the diagram represent pixel indices:

[0104] B[0]=G[0]+0.25×(B[1]-G[1])+0.25×(B[2]-G[2])

[0105] +0.25×(B[3]-G[3])+0.25×(B[4]-G[4])

[0106] R[0]=G[0]+0.25×(R[1]-G[1])+0.25×(R[2]-G[2])

[0107] +0.25×(R[3]-G[3])+0.25×(R[4]-G[4])

[0108] Among them, G[1], G[2], G[3], and G[4] are all calculated in step 1. 0.25 in the formula is a preset coefficient, and the difference in color values ​​represents the gradient information in the second intermediate image.

[0109] In particular, if some pixels are located at the edge of the image to be processed, some color values ​​can be mirrored around the image to be processed before the interpolation process is performed so that these edge pixels can also have a 5×5 neighborhood.

[0110] It is understood that the formulas in the examples above are merely illustrative and should not be considered as limitations on the scope of protection of this application.

[0111] In summary, the image processing method described in the above embodiments can be considered an improvement on existing demosaic algorithms. The advantages of this method are:

[0112] First, it utilizes a neural network model to predict interpolation direction information. The neural network model can be regarded as a complex interpolation rule determined based on training data. Its expressive power is significantly stronger than the manually set interpolation rules in existing demosaic algorithms, and it has good generalization ability, which is conducive to improving interpolation accuracy.

[0113] Secondly, the neural network model in the above method predicts the interpolation direction rather than directly predicting the color value. For digital images, the interpolation direction is very limited (e.g., horizontal, vertical, diagonal directions). Therefore, this model can be regarded as a classification model rather than a regression model. Relatively speaking, classification models are more likely to achieve higher prediction accuracy and are less likely to output unexpected prediction results.

[0114] In summary, the image processing method described above can produce images with fewer false colors and higher resolution. The resulting images are also less prone to artifacts, and using these images for subsequent display or other processing can achieve better results.

[0115] In some implementations, the neural network model in step S110 can be trained using the following method (taking the trained model for processing images to be processed by the Bayer array as an example):

[0116] First, collect some RGB images as training samples.

[0117] Then, the RGB image is sampled according to the requirements of the Bayer array (the G value sampling rate is twice that of the R and B values), and it is restored to a single-channel RAW image. The RAW image is then input into the neural network model, and the corresponding result image is calculated according to the image processing method above.

[0118] Next, the gradient of the loss function corresponding to the neural network model is calculated. This loss function represents the difference between the resulting image and the RGB image. For example, the loss function can take the form of a function such as mean squared error.

[0119] Finally, the parameters of the neural network model are updated based on the gradient of the loss function until the model converges.

[0120] Based on the loss function settings in the above training method, it can be seen that the training process aims to reduce the difference between the resulting image and the RGB image, thereby helping the neural network model predict more accurate interpolation direction information. It is understandable that other implementations may also use additional loss functions.

[0121] Below, based on the above embodiments, we will continue to introduce the case of multi-neural network models:

[0122] While it's possible to use the same neural network model to process all images, this approach is logically simple. However, the content of the images varies greatly, and their acquisition environments differ significantly. A single model may not be optimally suited to every image. Therefore, some implementations pre-select multiple alternative neural network models and assign the most suitable model based on the image's attribute information for predicting the interpolation direction, thus improving prediction accuracy. The specific steps are as follows:

[0123] Step A: Obtain the attribute information of the image to be processed. Attribute information can be defined as information describing the content of the image to be processed or its acquisition environment. For example, attribute information includes, but is not limited to, at least one of the following: the arrangement of color values ​​in the image to be processed, the photosensitivity of the image to be processed, and the brightness of the image to be processed.

[0124] In this context, the color value arrangement of the image to be processed for a specific camera is generally known, for example, arranged according to a Bayer array. However, even if the Bayer array is known, the color value arrangement in different images to be processed may still vary to some extent. For example, for... Figure 1 The middle sub-image has units of 4 pixels arranged from top to bottom and left to right with color values ​​of RGGB. However, if the first row and first column of this sub-image are removed, the color values ​​in each unit become BGGR, resulting in two different color value arrangements. This is possible because some image sensors crop the generated RAW images, for example, by cropping one or more of the following methods: the first row, the first column, the last row, and the last column. These cropping operations will cause changes to the color value arrangement in the image being processed.

[0125] The sensitivity of the image to be processed, also known as the ISO information of the image, is a parameter that is usually set when the image sensor acquires the image.

[0126] The brightness of the image to be processed reflects the lighting conditions of the environment in which the image is located. It is usually expressed as the magnitude of the color values ​​in the image. For example, the average value of all color values ​​in the image to be processed can be defined as the brightness of the image.

[0127] The attribute information of the image to be processed can be extracted from the image or image file. For example, the brightness information of the image to be processed can be calculated directly from the color values ​​in the image; alternatively, it can be obtained from the image sensor, for example, in... Figure 2 In this process, the image sensor can transmit the photosensitivity information of the image to be processed to the ISP module; or, it can combine these two methods to obtain the attribute information of the image to be processed, and other methods are also possible.

[0128] Step B: Determine a neural network model corresponding to the image to be processed from multiple candidate neural network models based on the attribute information of the image to be processed.

[0129] Different neural network models correspond to different image attribute information, but different attribute information does not necessarily correspond to different neural network models. For the sake of simplicity, let's consider only the image sensitivity attribute information. One possible correspondence is as follows:

[0130] ISO80~ISO160----Model 1;

[0131] ISO200~ISO800----Model 2;

[0132] ISO 800 and above ---- Model 3

[0133] Each model corresponds to a different ISO range, but different ISO values ​​within the same ISO range all correspond to the same model.

[0134] Each candidate neural network model can be trained using images with specific attribute information. Once trained, the model will perform well in predicting interpolation direction information for images with that specific attribute information. For example, for Model 1 above, it can be trained using RGB images with ISO 80 to ISO 160 resolution. The training process can be found in the previous text.

[0135] After obtaining the attribute information of the image to be processed in step A, its corresponding neural network model can be determined according to the preset matching rules. For example, if the ISO of the image to be processed is 100, which falls within the range of 80 to 160, then its corresponding neural network model is Model 1.

[0136] Step C: Based on the image to be processed, the interpolation direction information is predicted using the corresponding neural network model.

[0137] Step C is similar to step S110 and will not be repeated.

[0138] Building upon the above embodiments, the following section will further discuss the selection of the target category and the corresponding interpolation order:

[0139] In some implementations, the number of pixels in the image to be processed containing color values ​​of the target category is greater than the number of pixels containing color values ​​of any non-target category. In other words, the target category can be selected as one or more categories with the most color values ​​in the image to be processed. Generally speaking, the more color values ​​of a certain category are present in the image to be processed, the more important that color is to the image. This is a consideration when generating the image to be processed. For example, when setting the filters of the image sensor, the number of filters for each color is already set according to the importance of the corresponding color (the more important the color, the more filters). Since a certain color is relatively important, it is used as the target category, and the interpolation direction information corresponding to this color value is accurately predicted by the neural network model so as to accurately calculate this color value, which naturally helps to improve the image quality of the result image.

[0140] Of course, to improve interpolation accuracy, it is not necessary to use all color categories as the target category. The reasons are as follows: First, interpolation calculations based on neural network models generally involve more computation than those without neural network models (e.g., based on some of the interpolation formulas given above), which may affect the efficiency of image de-mosaicing. Second, predicting interpolation direction information for different color categories may require different neural network models to achieve good results, which means training more models and consuming more storage and computing resources. Third, if the neural network model is implemented using a hardware module, predicting the interpolation direction information corresponding to different color categories may require inputting the image to be processed into that hardware module multiple times, which violates the principle of streaming processing (the same data will not enter a module repeatedly, but will only flow downstream), which may affect processing efficiency. If multiple hardware modules are set up to predict interpolation direction information in parallel, the cost is too high.

[0141] For example, in the image to be processed using a Bayer array, the number of G pixels is twice that of R pixels and B pixels, and the interpolation accuracy requirement for G values ​​also exceeds that for R and B values. This design is because the human eye is more sensitive to green than to red and blue; therefore, including more G pixels (containing the actual acquired G values) in the image to be processed is beneficial for improving the quality of the de-mosaiced image.

[0142] For such images to be processed, green can be used as the target category. A neural network model can be used to predict the interpolation direction information at pixels R and B, and the G value to be interpolated can be calculated based on the interpolation direction information. As for the interpolation calculation of R and B values, the neural network can be used to predict the interpolation direction information. Thus, both the interpolation accuracy of G value and the interpolation efficiency are guaranteed.

[0143] Furthermore, when performing interpolation calculations on pixels of the image to be processed, the color value of the target pixel's target category can be calculated first, followed by the color values ​​of other categories of the target pixel and the color values ​​of non-target pixels. The color value of the target pixel's target category is particularly important; by calculating it first using high-precision interpolation direction information, the calculation results of other color values ​​can also be used, ensuring the accuracy of other color value calculations.

[0144] For example, continuing with the previous example, during the interpolation calculation, the G value can be calculated first, and then the R and B values ​​can be calculated. The calculation result of the G value may be used when calculating the R and B values. Since the calculation accuracy of the G value is high, the calculation accuracy of the R and B values ​​can also be guaranteed.

[0145] Figure 7 The structure of an image processing apparatus 200 provided in an embodiment of this application is shown. (Refer to...) Figure 7 The image processing device 200 includes a control module 210, a storage module 220, an I / O port 230, a neural network calculation module 240, and an interpolation calculation module 250. These components are interconnected and communicate with each other via a communication bus 260 and / or other forms of connection mechanism (not shown).

[0146] The neural network calculation module 240 is used to predict the interpolation direction information corresponding to the target pixel in the image to be processed based on the image to be processed using a neural network model. The image to be processed contains color values ​​of K categories, where K is an integer greater than 1. Each pixel contains color values ​​of only one category, and the color values ​​of the other K-1 categories of each pixel are to be calculated. The target pixel is the pixel in the image to be processed whose color value of the target category is to be calculated. The target category is at least one of the K categories, and each target category corresponds to one interpolation direction information.

[0147] As can be seen, the neural network computing module 240 implements the function of step S110 (including any possible implementation), or in other words, the neural network computing module 240 is the hardware implementation of step S110. For example, the neural network computing module 240 can be a neural network processing unit (NPU) or an NPU array, and some NPUs have the built-in function of converting the image to be processed into tensors.

[0148] The interpolation calculation module 250 is used to calculate the color values ​​of another K-1 categories of each pixel in the image to be processed based on the color values ​​of the pixels in its neighborhood, and obtain the result image; wherein, for the color value of each target category of each target pixel, the interpolation calculation is performed based on the interpolation direction information corresponding to the target category of the pixel.

[0149] As can be seen, the interpolation calculation module 250 implements the function of step S120 (including any possible implementation), or in other words, the interpolation calculation module 250 is the hardware implementation of step S120. For example, the interpolation calculation module 250 can be a unit in the ISP specifically used for interpolation calculation.

[0150] I / O ports 230 include one or more, used for communication between the image processing device 200 and external devices, such as inputting the image to be processed from the ISP front-end module into the image processing device 200, and outputting the calculated image from the image processing device 200 to the ISP back-end module. I / O ports 230 can be MIPI transceivers, PCI-e controllers, DDR controllers, debug ports, or other components capable of communicating with an external bus.

[0151] Storage module 220 is used to store the image to be processed, interpolation direction information, the result image, the neural network model (parameters, structure), and the program instructions of the driver program. There can be one or more storage modules 220. Each storage module 220 can be a random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.; multiple storage modules 220 can be a combination of the above components. Storage module 220 may be internal to the ISP or shared by the ISP and other modules.

[0152] The control module 210 is used to execute the program instructions of the driver program. During runtime, the driver program drives the neural network calculation module 240 and the interpolation calculation module 250 to perform calculations. There can be one or more control modules 210. Each control module 210 can be a general-purpose processor such as a CPU or a microcontroller unit (MCU), or a special-purpose processor such as a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or it can be other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; multiple control modules 210 can be a combination of the above components.

[0153] One possible operating procedure of the image processing device 200 is as follows:

[0154] After the device containing the image processing device 200 is powered on, the neural network model is loaded from other storage spaces into the storage module 220 (if the neural network model is already in the storage module 220, there is no need to load it).

[0155] The image to be processed output by the ISP front-end module (or other modules) is input to the image processing device 200 through the IO port 230 and stored in the storage module 220.

[0156] The neural network calculation module 240 uses a neural network model to predict the interpolation direction information corresponding to the target pixels in the image to be processed based on the image to be processed, and the interpolation direction information is stored in the storage module 220.

[0157] The interpolation calculation module 250 performs interpolation calculations based on the image to be processed and in conjunction with the interpolation direction information to obtain the result image, which is then stored in the storage module 220.

[0158] The resulting image is output to the ISP post-module (or other modules) via IO port 230.

[0159] Several points need to be clarified regarding the above work process:

[0160] First, the entire process is completed under the control of the control module 210, which is not shown in the above steps for the sake of simplicity.

[0161] Secondly, the loading of the neural network model occurs at a different time than the steps described above; it only needs to be loaded before the neural network computing module 240 begins computation. Furthermore, the neural network model supports dynamic loading. For example, based on the attribute information of the image to be processed provided by the image sensor (this information can be sent to the image processing device 200 along with the image to be processed), a matching model is selected from the candidate neural network models and dynamically loaded into the storage module 220. If the storage module 220 already has other models, they can be replaced if storage space is limited.

[0162] Understandable. Figure 7 The structure shown is for illustrative purposes only; the image processing apparatus 200 may also include components that are more advanced than those shown. Figure 7 The more or fewer components shown, or having the same Figure 7 Different configurations are shown. The image processing device 200 can be a standalone electronic device or integrated into other electronic devices, such as mobile phones, cameras, camcorders, PCs, tablets, laptops, wearable devices, robots, etc.

[0163] It should also be pointed out that, Figure 7 The image processing device 200 in the middle is different from Figure 2 The ISP intermediate module is a part of the ISP, that is, the part related to the image processing method in the embodiments of this application. The image processing apparatus 200 may include ISP components (e.g., interpolation calculation module 250, IO port 230) or non-ISP components (e.g., neural network calculation module 240, control module 210). The image processing apparatus 200 is concerned with the components required to implement the image processing method in the embodiments of this application, and does not care whether the component belongs to the ISP.

[0164] Figure 8 This illustration shows a possible structure of the image processing apparatus 300 provided in an embodiment of this application. (Refer to...) Figure 8 The image processing apparatus 300 includes a processing module 310, a storage module 320, and an I / O port 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0165] The processing module 310 includes one or more, and each processing module 310 can be a general-purpose processor, including CPU, MCU, etc., or a special-purpose processor, including digital signal processor, DSP, ASIC, FPGA, etc., or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; multiple processing modules 310 can be a combination of the above components.

[0166] The storage module 320 includes one or more, and each storage module 220 can be RAM, ROM, PROM, EPROM, EEPROM, etc.; multiple storage modules 220 can be a combination of the above components.

[0167] Processing module 310 and other possible components can access storage module 320 to read and / or write data therein. In particular, storage module 320 can store one or more computer program instructions, which processing module 310 can read and run to implement the image processing method (including any possible implementation thereof) provided in the embodiments of this application.

[0168] I / O ports 330 include one or more, used for communication between the image processing device 300 and external devices, such as inputting the image to be processed from the ISP front-end module into the image processing device 300, and outputting the calculated image from the image processing device 300 to the ISP back-end module. I / O ports 330 can be MIPI transceivers, PCI-e controllers, DDR controllers, debug ports, or other components capable of communicating with an external bus.

[0169] Understandable. Figure 8 The structure shown is for illustrative purposes only; the image processing device 300 may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 Different configurations are shown. The image processing device 300 can be a standalone electronic device or integrated into other electronic devices, such as mobile phones, cameras, camcorders, PCs, tablets, laptops, servers, wearable devices, robots, etc.

[0170] It should also be pointed out that, Figure 8 The image processing device 300 in the middle is different from Figure 7 The image processing device 200 in this application mainly implements the image processing method of this embodiment in hardware (for example, using a hardware-based neural network computing module 240 to predict interpolation direction information). Although the image processing device 200 also involves a driver program, the driver program only plays a control and scheduling role, and the core function of the method is still implemented by hardware. The image processing device 300, on the other hand, mainly implements the image processing method of this embodiment in software. The steps of the method are all program instructions stored in the storage module 320, and the core function of the method is implemented by the processing module 310 executing these instructions.

[0171] Depending on the specific processing requirements, either image processing device 200 or image processing device 300 can be selected. For example, for processing a single photograph, either image processing device 300 or image processing device 200 can be used; for video processing, image processing device 200 can be used. Since image processing device 200 uses hardware to remove mosaic effects, it can effectively ensure that the video frame rate is not significantly affected.

[0172] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a processor to perform the image processing method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 7 The storage module 220 in the image processing device 200, or, Figure 8 The storage module 320 in the image processing device 300.

[0173] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the image processing method provided in this application.

[0174] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: Based on the image to be processed, the interpolation direction information corresponding to the target pixel in the image to be processed is predicted using a neural network model; wherein, all pixels in the image to be processed contain color values ​​of K categories, where K is an integer greater than 1, each pixel contains only one category of color values, and the other K-1 categories of color values ​​for each pixel are to be calculated, the target pixel is the pixel in the image to be processed whose target category of color value is to be calculated, the target category is at least one of the K categories, and each target category corresponds to one interpolation direction information; For each pixel of the image to be processed, the color values ​​of the pixel in the next K-1 categories are interpolated based on the color values ​​of the pixels in its neighborhood to obtain the result image; wherein, for the color value of each target category of each target pixel, the interpolation is performed based on the interpolation direction information corresponding to the target category of the pixel. The image to be processed is a single-channel image, and the resulting image is a color image with K channels obtained by converting the image to be processed.

2. The image processing method according to claim 1, characterized in that, The interpolation direction information includes the probability corresponding to each potential interpolation direction. The interpolation calculation based on the interpolation direction information corresponding to the target category of the pixel includes: The potential interpolation direction with the highest probability is determined as the actual interpolation direction, and interpolation calculation is performed based on the actual interpolation direction to obtain the color value of the target category for that pixel; and / or, Interpolation calculations are performed in each potential interpolation direction to obtain multiple initial color values. The multiple initial color values ​​are then weighted using the probability corresponding to each potential interpolation direction as the weight to obtain the color value of the target category for the pixel.

3. The image processing method according to claim 1 or 2, characterized in that, The step of predicting the interpolation direction information corresponding to the target pixels in the image to be processed using a neural network model, based on the image to be processed, includes: Obtain the attribute information of the image to be processed, and determine the neural network model corresponding to the image to be processed from multiple candidate neural network models based on the attribute information; Based on the image to be processed, the interpolation direction information is predicted using the corresponding neural network model.

4. The image processing method according to claim 3, characterized in that, The attribute information includes at least one of the following: the arrangement of color values ​​in the image to be processed, the photosensitivity of the image to be processed, and the brightness of the image to be processed.

5. The image processing method according to any one of claims 1-2 and 4, characterized in that, The number of pixels in the image to be processed that contain the color value of the target category is greater than the number of pixels that contain the color value of any non-target category, and when interpolating the pixels of the image to be processed, the color value of the target category of the target pixel is calculated first.

6. The image processing method according to claim 5, characterized in that, The color values ​​of the K categories are R, G, and B values, respectively. The color value of the target category is G. The target pixels are the pixels in the image to be processed that contain R values ​​and the pixels that contain B values. The color values ​​in the image to be processed are arranged in a Bayer array manner.

7. An image signal processor, characterized in that, include: The front-end module is used to perform RAW image processing algorithms on the first RAW image generated by the image sensor to obtain the second RAW image; An intermediate module is used to perform interpolation calculations on the second RAW image using the method described in any one of claims 1-6 to obtain a first color image; The post-processor module is used to perform a color image processing algorithm on the first color image to obtain a second color image.

8. An image processing apparatus, characterized in that, include: The system includes a neural network calculation module, an interpolation calculation module, a storage module, and a control module. The neural network calculation module is used to predict the interpolation direction information corresponding to the target pixel in the image to be processed using a neural network model based on the image to be processed; wherein, all pixels in the image to be processed contain color values ​​of K categories, where K is an integer greater than 1, each pixel contains only one category of color values, and the other K-1 categories of color values ​​for each pixel are to be calculated, the target pixel is the pixel in the image to be processed whose target category of color value is to be calculated, the target category is at least one of the K categories, and each target category corresponds to one interpolation direction information; The interpolation calculation module is used to calculate the color values ​​of another K-1 categories of each pixel in the image to be processed based on the color values ​​of its neighboring pixels, and obtain the result image; wherein, for the color value of each target category of each target pixel, the interpolation calculation is performed based on the interpolation direction information corresponding to the target category of the pixel. The storage module is used to store the image to be processed, the interpolation direction information, the result image, the neural network model, and the driver program; the image to be processed is a single-channel image, and the result image is a color image including K channels obtained by converting the image to be processed; The control module is used to execute the driver program, which drives the neural network calculation module and the interpolation calculation module to perform calculations during runtime.

9. An image processing apparatus, characterized in that, It includes a storage module and a processing module. The storage module stores computer program instructions. When the computer program instructions are read and executed by the processing module, the method of any one of claims 1-6 is performed.

10. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

Images