Image processing method, image processing device, storage medium, and electronic device

Abstract

This disclosure provides an image processing method, apparatus, storage medium, and electronic device, relating to the field of image and video processing technology. The method includes: acquiring an image to be processed; the image to be processed includes pixels to be filled; performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged according to a preset direction; the intermediate image does not include pixels to be filled; and optimizing the intermediate image using a pre-trained image optimization model to obtain a target image. This disclosure combines the advantages of both non-machine learning algorithms and machine learning, enabling the calculation of accurate pixel values ​​for missing pixels and improving image quality.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing technology, and in particular to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In image processing, it is often necessary to supplement missing pixel values. For example, in a Bayer image, the pixel values ​​of the R, G, and B channels are incomplete, and the pixel values ​​in each channel need to be supplemented to obtain an RGB image.

[0003] Most related technologies involve interpolation based on existing pixel values ​​in an image to calculate missing pixel values. However, the accuracy of pixel values ​​calculated through interpolation is difficult to guarantee, resulting in low quality of the interpolated image. Summary of the Invention

[0004] This disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device to at least partially solve the problem of low image quality after supplementing missing pixel values ​​in related technologies.

[0005] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring an image to be processed; the image to be processed including pixels to be filled; performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel having fixed parameters, the parameters in the fixed convolution kernel being arranged according to a preset direction; the intermediate image not including pixels to be filled; and optimizing the intermediate image using a pre-trained image optimization model to obtain a target image.

[0006] According to a second aspect of this disclosure, an image processing apparatus is provided, comprising: an image acquisition module configured to acquire an image to be processed; the image to be processed including pixels to be filled; a convolution processing module configured to perform convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel having fixed parameters, the parameters in the fixed convolution kernel being arranged according to a preset direction; the intermediate image not including pixels to be filled; and an optimization processing module configured to optimize the intermediate image using a pre-trained image optimization model to obtain a target image.

[0007] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image processing method of the first aspect described above and its possible implementations.

[0008] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of the first aspect and possible implementations thereof by executing the executable instructions.

[0009] The technical solution disclosed herein has the following beneficial effects:

[0010] On the one hand, this solution employs a fixed convolution kernel and an image optimization model to perform convolution and optimization processing on the image sequentially. This fills in the pixels to be filled in the image and achieves global image optimization. It combines the advantages of both non-machine learning algorithms and machine learning algorithms. Compared to interpolation algorithms in related technologies, it is unaffected by the interpolation direction and can calculate accurate pixel values ​​for missing pixels, resulting in a high-quality target image. On the other hand, using a fixed convolution kernel reduces the complexity of the image optimization model and improves image processing efficiency. This enables real-time image processing, facilitating deployment in scenarios such as real-time video processing. Attached Figure Description

[0011] Figure 1 A schematic diagram illustrating the operating environment system architecture of this exemplary embodiment is shown;

[0012] Figure 2 A flowchart illustrating an image processing method in this exemplary embodiment is shown;

[0013] Figure 3 A schematic diagram illustrating image super-resolution reconstruction in this exemplary embodiment is shown;

[0014] Figure 4 A schematic diagram illustrating a Bayer image in this exemplary embodiment;

[0015] Figure 5 This illustrates a flowchart of acquiring an image to be processed in this exemplary embodiment;

[0016] Figure 6 A schematic diagram illustrating the calculation of the difference between filled pixels in this exemplary embodiment is shown;

[0017] Figure 7 This example illustrates a flowchart for determining the compensation value in this exemplary embodiment;

[0018] Figure 8 This diagram illustrates the determination of grayscale values ​​in this exemplary embodiment.

[0019] Figure 9 A schematic diagram of a fixed convolution kernel is shown in this exemplary embodiment;

[0020] Figure 10This diagram illustrates the global model and image processing in this exemplary embodiment;

[0021] Figure 11 A schematic flowchart illustrating the image processing method in this exemplary embodiment is shown;

[0022] Figure 12 This diagram illustrates the structure of an image processing apparatus according to this exemplary embodiment.

[0023] Figure 13 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0024] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0025] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0026] In related technologies, interpolation is performed based on existing pixel values ​​in an image to calculate missing pixel values. However, if the interpolation direction is not chosen appropriately, the calculated pixel values ​​will be inaccurate, leading to problems such as zipper artifacts and jagged edges in the interpolated image, thus affecting image quality.

[0027] In view of the above problems, an exemplary embodiment of this disclosure provides an image processing method. A fixed convolution kernel is pre-configured and an image optimization model is trained. Convolution processing and optimization processing are then performed on the image to be processed, which contains pixels to be filled, to obtain a target image. In the target image, the pixels to be filled in the image to be processed are filled, resulting in a higher quality target image.

[0028] The following is combined Figure 1 The operating environment system architecture of this exemplary embodiment will be described.

[0029] refer to Figure 1As shown, the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be an electronic device such as a mobile phone, tablet computer, or smart wearable device. The server 120 generally refers to the backend system that provides image processing-related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to exchange data.

[0030] In one implementation, the image processing method of this exemplary embodiment can be executed by terminal 110. For example, server 120 can configure a fixed convolutional kernel and train an image optimization model, then deploy the fixed convolutional kernel and image optimization model to terminal 110; alternatively, terminal 110 can configure the fixed convolutional kernel and train the image optimization model locally; or, terminal 110 can configure the fixed convolutional kernel locally, server 120 can train the image optimization model, and then deploy it to terminal 110. Terminal 110 acquires the image to be processed, for example, a Bayer image before demosaicing, and then uses the fixed convolutional kernel and image optimization model to process the image to achieve an effect equivalent to demosaicing, obtaining the target image.

[0031] In one implementation, the image processing method of this exemplary embodiment can be executed by server 120. For example, server 120 can configure a fixed convolutional kernel and train an image optimization model. Terminal 110 uploads the image to be processed to server 120, which then executes the image processing method to obtain the target image. The target image can then be returned to terminal 110.

[0032] As can be seen from the above, in this exemplary embodiment, the entity executing the image processing method can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0033] The following is combined Figure 2 The process of image processing methods is explained.

[0034] refer to Figure 2 As shown, the image processing method may include the following steps S210 to S230:

[0035] Step S210: Obtain the image to be processed; the image to be processed includes pixels to be filled.

[0036] Step S220: Convolve the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged according to a preset direction; the intermediate image does not include pixels to be filled.

[0037] Step S230: Optimize the intermediate image using a pre-trained image optimization model to obtain the target image.

[0038] Based on the above method, on the one hand, this scheme uses a fixed convolution kernel and an image optimization model to perform convolution and optimization processing on the image in sequence to fill in the pixels to be filled in the image and achieve global image optimization. This combines the advantages of both non-machine learning algorithms and machine learning algorithms. Compared to interpolation algorithms in related technologies, it is not affected by the interpolation direction and can calculate accurate pixel values ​​for missing pixels, resulting in a high-quality target image. On the other hand, using a fixed convolution kernel for convolution processing helps reduce the complexity of the image optimization model and improve image processing efficiency. This enables this scheme to achieve real-time image processing, allowing it to be deployed in scenarios such as real-time video processing.

[0039] The following is about Figure 2 Each step in the process will be explained in detail.

[0040] refer to Figure 2 In step S210, the image to be processed is obtained; the image to be processed includes pixels to be filled.

[0041] A pixel to be filled refers to a pixel whose pixel value is missing. The initial pixel value of a pixel to be filled can be empty or a preset fill value representing the missing value, such as 0. In the image to be processed, all pixels except the pixel to be filled are filled pixels, which are pixels with actual pixel values.

[0042] The image to be processed has missing pixels and can be considered an incomplete image. This disclosure does not limit the type or source of the image to be processed.

[0043] In one implementation, the image processing method can be used for super-resolution reconstruction of an image, where the image to be processed can be a high-resolution image with pixel values ​​to be filled. For example, refer to... Figure 3 As shown, a low-resolution image can be acquired. According to the resolution requirements of super-resolution reconstruction, the resolution of the low-resolution image is doubled; for example, the width and height resolution can be doubled to obtain an initial high-resolution image. This image includes pixels from the low-resolution image (which already have pixel values ​​and are therefore filled pixels) and unassigned pixels (which do not yet have pixel values ​​and are therefore pixels to be filled). This image can be used as the image to be processed. By executing the image processing method in this exemplary embodiment, a complete high-resolution image can be obtained.

[0044] In one implementation, the image processing method can be used for de-mosaicing of an image, where the image to be processed can be an R-channel image, G-channel image, or B-channel image from a Bayer image. (See reference) Figure 4As shown, a Bayer image typically refers to a RAW format image, which is an image acquired using a Bayer filter and an image sensor. Each pixel has only one of the R, G, and B pixel values. Therefore, a Bayer image can be represented as a superposition of an R-channel image, a G-channel image, and a B-channel image, where a subset of pixels in each channel have pixel values ​​( Figure 3 The pixels marked with "R", "G" or "B" are the ones that are not in the range, while the other set of pixels are empty. Figure 3 (Pixels not labeled "R", "G", or "B"). For example, in a G channel image, pixels not labeled "G" are empty values, i.e., pixels to be filled; approximately half of the pixels in the G channel image are pixels to be filled. By performing the image processing method in this exemplary embodiment, a complete R channel image, G channel image, or B channel image can be obtained.

[0045] In one implementation, the above-mentioned acquisition of the image to be processed may include the following steps:

[0046] Obtain the G channel image from the Bayer image; the G channel image includes filled pixels and unfilled pixels, the G pixel values ​​of filled pixels in the G channel image are not null, and the G pixel values ​​of unfilled pixels in the G channel image are null.

[0047] The G channel image is subjected to green balance processing to obtain the image to be processed.

[0048] Generally, G pixels in a Bayer image can include both Gr pixels and Gb pixels. Gr pixels and R pixels are in the same row, and Gb pixels and B pixels are in the same row. Due to potential hardware differences (such as subtle differences in semiconductor structure caused by manufacturing processes), circuit fluctuations, and differences in incident light angles between adjacent rows of an image sensor, uneven illumination response occurs, manifesting as differences between Gr pixels and Gb pixels. Furthermore, pixel differences may also exist between Gr pixels or Gb pixels at different locations due to uneven illumination response. Therefore, green balance processing can be performed on the G channel image. Green balance processing can mitigate or eliminate G pixel differences caused by uneven illumination response between different G pixels. The green-balanced G channel image is used as the image to be processed. This improves the adverse effects of hardware differences, circuit fluctuations, and differences in incident light angles on the image to be processed, enhancing the quality of the image to be processed and thus improving the quality of the processed target image.

[0049] Refer to the above Figure 4As shown, the filled pixels in the G channel image can be pixels marked with "G" whose G pixel values ​​are not null. The filled pixels in the G channel image are the G pixels in the Bayer image. The pixels to be filled in the G channel image can be pixels not marked with "G" whose G pixel values ​​are null. In this exemplary embodiment, green balance processing can be performed on the filled pixels of the G channel image, that is, the existing G pixel values ​​can be balanced.

[0050] In one implementation, the filled pixels in the G channel image can be filtered to achieve a smoothing effect, thereby reducing the difference between adjacent filled pixels and achieving green balance.

[0051] In one implementation, reference Figure 5 As shown, the above-described green balance processing of the G channel image to obtain the image to be processed may include the following steps S510 to S530:

[0052] Step S510: Obtain the difference between each filled pixel in the G channel image and its neighboring filled pixels.

[0053] In this paper, a filled pixel *i* represents any filled pixel in the G-channel image. A local area of ​​3×3 or 5×5, centered on filled pixel *i*, is selected. The specific size of this local area is not limited; it can be determined based on the size of the G-channel image. The larger the size of the G-channel image, the larger the local area. Other filled pixels within the local area are considered as adjacent filled pixels to filled pixel *i*, and the difference between filled pixel *i* and each adjacent filled pixel is calculated. Since a filled pixel is a pixel with a G-pixel value, the difference can be the difference between G-pixel values. For example, the difference can be obtained by subtracting the G-pixel value of filled pixel *i* from the G-pixel value of an adjacent filled pixel, or by subtracting the G-pixel value of filled pixel *i* from the G-pixel value of an adjacent filled pixel and taking the absolute value.

[0054] In one implementation, if a filled pixel i has multiple adjacent filled pixels, the difference in step S510 can be the sum of the differences between the filled pixel i and each of its adjacent filled pixels, or the average of the differences between the filled pixel i and each of its adjacent filled pixels.

[0055] Step S520: Determine the compensation value for each filled pixel based on the difference between each filled pixel and its neighboring filled pixels.

[0056] The compensation value is used to compensate for the imbalance between G pixels. If the difference between a filled pixel and its neighboring filled pixels is large, it indicates that the imbalance between G pixels in the neighborhood of that filled pixel is more severe, and a larger compensation value can be used. Therefore, the compensation value can be positively correlated with the difference. This disclosure does not limit the specific relationship between the two. For example, the difference and the compensation value can be a linear function relationship, and after obtaining the difference, the compensation value can be calculated using a linear function; or, the difference and the compensation value can be a piecewise function relationship, and after obtaining the difference, the corresponding compensation value can be determined according to the interval in which the difference lies.

[0057] In one implementation, each filled pixel in the G-channel image may have four adjacent filled pixels. (See reference) Figure 6 As shown, in an 8×8 G-channel image (the solid line portion in the image), a 3×3 local area can be selected centered on each filled pixel. This local area includes four adjacent filled pixels, as indicated by the arrows in the image. Specifically, considering filled pixels located at the image corners, when determining their adjacent filled pixels, padding can be used to add rows and columns at the image edges. For example, a mirror mapping method can be used: copy the second row at the top edge of the G-channel image and the seventh row at the bottom edge, adding one row at each edge; then copy the second column at the left edge and the seventh column at the right edge, adding one column at each edge. Figure 6 The dashed lines represent the added rows and columns. Of course, preset pixel values ​​can also be filled into these added rows and columns. Therefore, a filled pixel at a corner can also have four adjacent filled pixels. Figure 6 It can be seen that each filled pixel is in a different row from its four neighboring filled pixels. This may result in the aforementioned uneven lighting response between each filled pixel and its four neighboring filled pixels.

[0058] To further accurately calculate the compensation value, in one implementation, reference is made to... Figure 7 As shown, determining the compensation value for each filled pixel based on the difference between each filled pixel and its neighboring filled pixels may include the following steps S710 to S730:

[0059] Step S710: For any filled pixel i in the G channel image, compare the four differences between the filled pixel i and its four adjacent filled pixels with the green balance threshold of the filled pixel i, and take the difference greater than the green balance threshold as the valid difference of the filled pixel i.

[0060] The green balance threshold is used to measure whether the difference between adjacent filled pixels is too large. If the difference is too large, it indicates that there may be an uneven illumination response; if the difference is too small, it may be a normal pixel difference. The green balance threshold can be determined based on experience or specific circumstances, or it can be iteratively optimized based on the image processing results. In one embodiment, the above-mentioned green balance processing of the G channel image to obtain the image to be processed may further include the following steps:

[0061] Determine the grayscale value of each filled pixel in the G-channel image;

[0062] Based on the grayscale value of each filled pixel, a green balance threshold is determined for each filled pixel. The green balance threshold is positively correlated with the grayscale value.

[0063] The grayscale value is used to characterize the brightness level of the filled pixel. A relationship can be established between the G pixel value and the grayscale value, such as a linear transformation relationship, to determine the grayscale value of each filled pixel based on its G pixel value. In one embodiment, the grayscale value of the G pixel can also be determined based on the G pixel in the Bayer image and its adjacent R and B pixels. For example, the R and B pixel values ​​of the G pixel can be obtained by interpolating the R and B pixels adjacent to the G pixel, and then the grayscale value of the G pixel can be calculated using the formula gray = 0.299R + 0.587G + 0.114B. Alternatively, refer to... Figure 7 As shown, a specific filter kernel can be used to filter the G pixels in the Bayer image. Specifically, Figure 8 The image shows the row and column coordinates of each pixel in the Bayer image. For example, G11 represents pixel G located in row 1 and column 1, and R12 represents pixel R located in row 1 and column 2. The filter kernel size is 3×3, which can filter a 3×3 region centered on pixel G in the Bayer image. For example, filtering the 3×3 region centered on G22 in the upper left corner yields the following grayscale value for G22:

[0064] gray(G22)=(G11+2R12+G13+2B21+4G22+2B23+G31+2R32+G33) / (1+2+1+2+4+2+1+2+1) (1)

[0065] Similarly, other G pixels can be filtered to obtain grayscale values. For G pixels located at corners, the padding method described above can be used to add rows and columns to the Bayer image, and then the G pixels at the corners can be filtered to obtain grayscale values. Thus, the grayscale value of each G pixel in the Bayer image can be obtained. The G pixels in the Bayer image are the filled pixels in the G channel image, therefore the grayscale value of each filled pixel in the G channel image is obtained.

[0066] Generally, the higher the grayscale value of a filled pixel, the higher its brightness level, and the greater the potential for normal pixel differences between that filled pixel and its neighboring filled pixels. Therefore, a higher green balance threshold is possible. Illumination response imbalance is considered to exist only when the difference between a filled pixel and its neighboring filled pixels exceeds the green balance threshold. The green balance threshold is positively correlated with the grayscale value; that is, the higher the grayscale value, the stricter the condition for measuring the existence of illumination response imbalance. This disclosure does not limit the specific relationship between the green balance threshold and the grayscale value. For example, the relationship between the green balance threshold and the grayscale value can be a linear function or a piecewise function. After obtaining the grayscale value of each filled pixel, the green balance threshold of each filled pixel is calculated based on this linear or piecewise function.

[0067] Let T be the green balance threshold of the filled pixel i. i The four differences between the filled pixel i and its four adjacent filled pixels are denoted as diff. i1 diff i2 diff i3 diff i4 The four differences are compared with T respectively. i Comparison, greater than T i The difference is the valid difference.

[0068] Step S720: Determine the equivalent difference of the filled pixel i based on the four differences of the filled pixel i.

[0069] The equivalent difference is the difference obtained by combining the four differences, and it can characterize the difference level between the filled pixel i and all its neighboring filled pixels. For example, the average of the four differences of the filled pixel i can be used as its equivalent difference.

[0070] In one implementation, determining the equivalent difference of filled pixel i based on the four differences of filled pixel i may include the following steps:

[0071] The product of the average of the four differences of the filled pixel i and the first compensation coefficient is used as the equivalent difference of the filled pixel i.

[0072] The first compensation coefficient is used to adjust the magnitude of the effective difference, preventing it from becoming too large. The first compensation coefficient is a value within the range [0,1], and can be determined based on experience or specific circumstances, or iteratively optimized based on the image processing results. For example, the first compensation coefficient could be 0.5 / count. i count i This represents the number of effective differences for filled pixel i. The equivalent difference *diff* for filled pixel i can be calculated using the following formula: i_eff :

[0073]

[0074] This approach takes into account the number of effective differences when calculating the equivalent difference, allowing the equivalent difference to more accurately reflect the uneven lighting response at the filled pixel i, which helps improve the accuracy of green balance processing.

[0075] Step S730: Determine the compensation value of the filled pixel i based on the number of effective differences of the filled pixel i and the equivalent difference of the filled pixel i.

[0076] Generally, the more effective differences there are in filled pixel i, the more prevalent the uneven lighting response at filled pixel i is, and the higher the compensation value is usually. When determining the specific compensation value, it is also necessary to consider the equivalent differences of filled pixel i so that the compensation value is compatible with the equivalent differences.

[0077] In one implementation, determining the compensation value of the filled pixel i based on the number of effective differences of the filled pixel i and the equivalent differences of the filled pixel i may include the following steps:

[0078] The second compensation coefficient is determined based on the number of valid differences of filled pixel i, and the second compensation coefficient is in the range of [0,1].

[0079] The product of the equivalent difference of the filled pixel i and the second compensation coefficient is used as the compensation value of the filled pixel i.

[0080] The second compensation coefficient is used to adjust the magnitude of the compensation value. It can be positively correlated with the number of effective differences, and the relationship between the two can be a linear function or a piecewise function. For example, the number of effective differences can be normalized to the range [0,1] to obtain the second compensation coefficient.

[0081] In one implementation, if the number of valid differences in filled pixel i is less than a first quantity threshold, the second compensation coefficient is determined to be 0. The first quantity threshold is equivalent to the lower limit of the number of valid differences. If the number of valid differences is less than the first quantity threshold, it indicates that the probability of uneven illumination response at filled pixel i is low. Therefore, the second compensation coefficient is determined to be 0, making the compensation value 0, meaning no compensation is needed for filled pixel i. If the number of valid differences in filled pixel i is greater than the second quantity threshold, the second compensation coefficient is determined to be 1. The second quantity threshold is equivalent to the upper limit of the number of valid differences. If the number of valid differences is greater than the second quantity threshold, it indicates that the probability of uneven illumination response at filled pixel i is high. Therefore, the second compensation coefficient is determined to be 1, making the compensation value larger, so as to compensate filled pixel i to a higher degree. It should be noted that the first and second quantity thresholds can be determined based on experience or specific circumstances, or can be iteratively optimized based on image processing results. If the number of valid differences in filled pixel i is greater than the first quantity threshold and less than the second quantity threshold, the number of valid differences in filled pixel i is normalized based on the first and second quantity thresholds, and the second compensation coefficient is determined based on the normalization result. For example, the number of effective differences can be normalized to the maximum and minimum values ​​based on the first and second quantity thresholds, and the normalization result can be used as the second compensation coefficient. Alternatively, the normalization result can be numerically adjusted to obtain the second compensation coefficient, which is between 0 and 1, so that the compensation value is between 0 and the equivalent difference, so as to provide a moderate degree of compensation for the filled pixel i.

[0082] Given the second compensation coefficient, the equivalent difference of the filled pixel i is multiplied by the second compensation coefficient to obtain the compensation value of the filled pixel i.

[0083] For example, the compensation value diff for the filled pixel i can be calculated using the following formula. i_comp :

[0084]

[0085] Where thr1 represents the first quantity threshold, thr2 represents the second quantity threshold, and k is the adjustment coefficient, which can be thr1-thr2 or a fixed value such as 32.

[0086] based on Figure 7 The method can determine an appropriate compensation value for each filled pixel, thereby accurately compensating each filled pixel.

[0087] Step S530: In the G channel image, each filled pixel is compensated using the compensation value of each filled pixel to obtain the image to be processed.

[0088] The G-pixel value of each filled pixel in the G-channel image can be increased by adding the corresponding compensation value to obtain the compensated G-pixel value. This achieves green balance processing of the G-channel image, and the processed G-channel image is the image to be processed.

[0089] based on Figure 5 The method compensates each filled pixel based on the difference between each filled pixel and its neighboring filled pixels. The degree of compensation is adapted to the local situation of each filled pixel, which helps to improve the effect of green balance processing and effectively improves the pixel difference caused by the uneven illumination response between different G pixels, so as to obtain a high-quality image to be processed.

[0090] Having obtained the image to be processed, continue to refer to... Figure 2 In step S220, the image to be processed is convolved using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged in a preset direction; the intermediate image does not include pixels to be filled.

[0091] The parameters of a convolutional kernel can include weights, etc. A fixed convolutional kernel has fixed parameters, meaning the weights in the kernel are constant. A preset direction refers to one or more pre-configured directions, such as, but not limited to, the following: 0-degree direction (horizontal), 45-degree direction, 90-degree direction (vertical), 135-degree direction, 180-degree direction (the reverse of the 0-degree direction), 225-degree direction (the reverse of the 45-degree direction), 270-degree direction (the reverse of the 90-degree direction), 315-degree direction (the reverse of the 135-degree direction), etc. The parameters in a fixed convolutional kernel can be arranged according to the preset direction, so that the parameters exhibit directional characteristics matching the preset direction. For example, Figure 9 The diagram shows six 3×3 fixed convolutional kernels and four 5×5 fixed convolutional kernels. Some cells do not have parameter labels, indicating that their parameters are 0. It can be seen that the parameter arrangement in each fixed convolutional kernel has a certain directionality. For example, in fixed convolutional kernel 1, parameters "1" and "-1" are symmetrically arranged along the 0-degree direction; in fixed convolutional kernel 2, parameters "1" and "-1" are symmetrically arranged along the 90-degree direction; in fixed convolutional kernel 3, parameters "1" and "-1" are symmetrically arranged along the 45-degree direction; in fixed convolutional kernel 7, parameters "1" and "-1" are symmetrically arranged along the 225-degree direction; in fixed convolutional kernel 8, parameters "1" and "-1" are symmetrically arranged along the 315-degree direction, and so on.

[0092] By performing convolution processing on the image to be processed using a fixed convolution kernel, the pixel value distribution features along a preset direction within a local region of the image can be extracted. For example, using... Figure 9The fixed convolution kernel 1 can extract the pixel value distribution features along the 0-degree direction in a local area of ​​the image to be processed. In particular, if the pixel value distribution along the 0-degree direction is uneven, the corresponding feature values ​​can be extracted by the fixed convolution kernel 1. Generally, the higher the degree of unevenness in the pixel value distribution, the larger the extracted feature values.

[0093] It should be noted that this disclosure does not limit the number of fixed convolutional kernels. In one embodiment, the number of fixed convolutional kernels can be multiple, and different fixed convolutional kernels have different sizes or different parameter arrangement characteristics. Here, parameter arrangement characteristics refer to the preset direction of the parameter arrangement. For example, Figure 9 The system uses 10 fixed convolutional kernels, with any two kernels having different sizes or parameter arrangements. Therefore, by convolving the image with different fixed kernels, feature information of different sizes, directions, and locations can be extracted. Of course, this disclosure does not specifically limit the size or parameter arrangement of the fixed convolutional kernels. For example... Figure 9 The diagram shows two fixed convolution kernel sizes: 3×3 and 5×5. Other fixed convolution kernel sizes, such as 7×7, can also be set. Figure 9 It shows parameter arrangement features that match preset directions such as 0 degrees, 90 degrees, 45 degrees, and 225 degrees, and can also set fixed convolution kernels with parameters arranged in other preset directions.

[0094] When performing convolution processing on the image to be processed, the stride can be set to any value according to specific needs. For pixels located at the corners of the image to be processed, the padding method described above can be used to fill the image, and then convolution processing can be performed on the filled image. This can fully extract the feature information of the corner positions.

[0095] During convolution processing, pixel values ​​can be calculated for the pixels in the image to be filled. These pixel values ​​can be further optimized in subsequent processing, and therefore can be called intermediate pixel values. Convolution processing transforms the image into a complete image, which is then called an intermediate image. Each pixel in the intermediate image has a pixel value; that is, the intermediate image does not include the pixels to be filled.

[0096] In one embodiment, the above-described convolutional processing of the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image may include the following steps:

[0097] A fixed convolution kernel is used to perform convolution operations on the local regions centered on each pixel to be filled in the image to be processed, so as to obtain the intermediate pixel value corresponding to each pixel to be filled.

[0098] The intermediate image is obtained by filling the pixels in the image to be filled with the intermediate pixel value corresponding to each pixel to be filled.

[0099] Since the purpose of image processing is to assign values ​​to the pixels to be filled, the center of a fixed convolution kernel can be aligned with the pixel to be filled to perform convolution operations. This extracts the feature information of the local region around the pixel to be filled, without needing to extract feature information of the local region around the already filled pixel. This reduces the amount of computation in convolution and improves the efficiency of image processing.

[0100] Convolution operations can be used to calculate the uneven distribution of pixels in the local area around each pixel to be filled. The calculated values ​​are then used as the intermediate pixel values ​​for each pixel to be filled, thus obtaining the intermediate pixel values ​​for each pixel to be filled in the intermediate image.

[0101] In one implementation, if there are multiple fixed convolution kernels, the above-mentioned convolution processing of the image to be processed using pre-configured fixed convolution kernels to obtain an intermediate image may include the following steps:

[0102] The image to be processed is convolved using each fixed convolution kernel to obtain multiple intermediate images.

[0103] Each fixed convolutional kernel corresponds to an intermediate image, and different intermediate images have different intermediate pixel values, representing feature information within different sizes, directions, and positions. By using multiple intermediate images, the pixel distribution features in the image to be processed can be comprehensively and fully represented, which is beneficial for calculating accurate pixel values ​​for the pixels to be filled, thus obtaining a high-quality target image.

[0104] Having obtained the intermediate image, continue to refer to... Figure 2 In step S230, the intermediate image is optimized using a pre-trained image optimization model to obtain the target image.

[0105] The image optimization model can be any type of machine learning model, such as a neural network. There may be inconsistencies in the numerical ranges of intermediate pixel values ​​compared to the original pixel values, or issues such as color parallax in the intermediate image. These problems can be corrected through image optimization processing to obtain a high-quality target image.

[0106] In one implementation, when multiple intermediate images are obtained, the above-mentioned optimization processing of the intermediate images using a pre-trained image optimization model to obtain the target image may include the following steps:

[0107] The target image is obtained by fusing and optimizing multiple intermediate images using an image optimization model.

[0108] It should be understood that multiple intermediate images can be merged into a single image first, and then optimized to obtain the target image; alternatively, multiple intermediate images can be optimized separately first, and then the optimized images can be merged into a single image to obtain the target image; alternatively, multiple intermediate images can be optimized first, image fusion can be performed at an intermediate stage of the optimization process, and then the remaining optimization processing can be performed on the merged image to obtain the target image. This allows the target image to incorporate feature information from different intermediate images, improving the accuracy of pixel values ​​in the target image.

[0109] In one embodiment, prior to step S220, the image processing method may further include the following steps:

[0110] Obtain a pre-trained global model, which includes fixed convolutional layers and an image optimization model. The fixed convolutional layers include fixed convolutional kernels.

[0111] Accordingly, the above-mentioned convolution processing of the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image may include the following steps:

[0112] The image to be processed is input into the global model, and the image is processed by convolution through a fixed convolutional layer to obtain an intermediate image.

[0113] In other words, the original convolutional processing can be achieved using fixed convolutional layers in the global model. In one implementation, a fixed convolutional layer can be added before the image optimization model to obtain the global model. The image to be processed is then input into the global model, which can complete both the convolutional and optimization steps, making the deployment of the solution simpler and more convenient.

[0114] Figure 10 The diagram illustrates the global model and image processing. Blank cells in the image to be processed represent pixels to be filled. The global model can include a fixed convolutional layer (which can be considered intermediate layer 1), intermediate layer 2, intermediate layer 3, ..., intermediate layer n. The image to be processed is input into the global model, and convolution is performed separately by multiple fixed convolutional kernels in the fixed convolutional layer to obtain multiple intermediate images. Intermediate layer 2 then performs weighted fusion of these intermediate images to obtain a fused intermediate image. Intermediate layers 3 through n further optimize the fused intermediate image to finally obtain the target image, where all pixels have pixel values.

[0115] It should be noted that during the optimization process, the original pixel values ​​of the filled pixels may be changed. In other words, the image processing method in this exemplary embodiment not only assigns values ​​to the pixels to be filled, but also optimizes the filled pixels to improve image quality.

[0116] In one implementation, the global model can be trained in the following way:

[0117] Obtain the first sample image and the corresponding label image. The first sample image includes pixels to be filled, while the label image does not include pixels to be filled.

[0118] The first sample image is input into the global model to be trained, and the second sample image is output.

[0119] With the parameters of the convolution kernel fixed, the parameters of the global model are updated based on the second sample image and the label image.

[0120] In this dataset, the first sample image and the image to be processed can be of the same type, such as R-channel, G-channel, or B-channel images with missing pixels. The label image is an image excluding missing pixels and serves as ground truth for model training. The first sample image and the label image can be combined to form a dataset. In one implementation, a label image can be obtained first, where all pixels are non-nullable. Then, some pixels in the label image are set to null values ​​to obtain the corresponding first sample image. The nullable pixels in the first sample image are the pixels to be filled. For example, a G-channel image can be extracted from a complete RGB image, where all pixels have G pixel values. This G-channel image can be used as the label image. Then, some pixel values ​​are removed from it to obtain the pixels to be filled, and this G-channel image can be used as the first sample image. By obtaining the first sample image from the label image, the difficulty of obtaining the dataset can be reduced, while ensuring the quality of the label image.

[0121] The first sample image is input into the global model. Since the global model is still being trained, the resulting second sample image is not ideal. A loss function can be constructed based on the second sample image and the label image, and the parameters of the global model are updated through this loss function. It should be noted that the parameters of the fixed convolutional kernel can remain unchanged during the parameter update process. This is because the fixed convolutional kernel is configured based on a preset direction, which is suitable for the needs of image feature extraction. Therefore, the fixed convolutional kernel can be considered to have a relatively reasonable configuration, and thus it does not need to participate in training. This allows the global model to converge faster, accelerates the training process, and improves efficiency.

[0122] With the global model fully trained, it can be deployed in an image processing environment, where the image to be processed can be input into the global model to complete the image processing process in this exemplary embodiment.

[0123] In one implementation, if the image to be processed is a G channel image in a Bayer image or a G channel image after green balance processing, the image processing method may further include the following steps after obtaining the target image, that is, obtaining the complete G channel image:

[0124] Obtain the R-channel and B-channel images from the Bayer image; the R-channel image includes filled pixels and unfilled pixels, the R-pixel values ​​of filled pixels in the R-channel image are not null, and the R-pixel values ​​of unfilled pixels in the R-channel image are null; the B-channel image includes filled pixels and unfilled pixels, the B-pixel values ​​of filled pixels in the B-channel image are not null, and the B-pixel values ​​of unfilled pixels in the B-channel image are null.

[0125] Based on the difference between the R pixel value of the filled pixel in the R channel image and the G pixel value of the pixel at the same position in the target image, fill the pixel to be filled in the R channel image with the R pixel value.

[0126] Based on the difference between the B pixel value of the filled pixel in the B channel image and the G pixel value of the pixel at the same position in the target image, fill the pixel to be filled in the B channel image with the B pixel value.

[0127] In other words, through Figure 2 After processing the G channel image, the method can also process the R channel image and B channel image to fill the pixels to be filled in the R channel image and B channel image, so as to obtain the complete R channel image and B channel image.

[0128] Missing R and B pixel values ​​can be calculated based on the principle of color constancy. Taking R pixel values ​​as an example, the following relationship exists:

[0129] R(i,j)-G(i,j)=R(i,j+1)-G(i,j+1) (4)

[0130] Where i and j represent the pixel coordinates in the Bayer image, i.e., the row and column where the pixel is located. G(i,j) is the G pixel value of pixel (i,j) in the target image, G(i,j+1) is the G pixel value of pixel (i,j+1) in the target image, R(i,j) is the R pixel value of pixel (i,j) in the R channel image, and R(i,j+1) is the R pixel value of pixel (i,j+1) in the R channel image. Formula (4) indicates that the difference between the R pixel value and the G pixel value of adjacent pixels is equal. Therefore, the R pixel value of the unfilled pixel next to the filled pixel in the R channel image can be calculated, thus obtaining the complete R channel image.

[0131] Similarly, based on the principle of color constancy, the difference between the B pixel value and the G pixel value of adjacent pixels is also equal. Therefore, the B pixel value of the unfilled pixel next to the filled pixel in the B channel image can be calculated, thus obtaining the complete B channel image.

[0132] Finally, an RGB image can be output based on the complete G-channel image, R-channel image, and B-channel image. This achieves complete demosaic processing.

[0133] Figure 11 An illustrative flow diagram of the image processing method in this exemplary embodiment is shown, including:

[0134] Step S1101: Obtain Bayer images, including R channel images, G channel images, and B channel images;

[0135] Step S1102: Calculate the gray value of pixel G in the Bayer image;

[0136] Step S1103: Determine the green balance threshold of pixel G based on the gray value of pixel G, and obtain the green balance threshold of each filled pixel in the G channel image.

[0137] Step S1104: Extract the G channel image;

[0138] Step S1105: Calculate the difference between adjacent filled pixels in the G channel image;

[0139] Step S1106: Calculate the equivalent difference of each filled pixel based on the difference between adjacent filled pixels and the green balance threshold, and then determine the compensation value.

[0140] Step S1107: Compensate for the filled pixels in the G channel image to obtain the image to be processed;

[0141] Step S1108: Input the image to be processed into the pre-trained global model;

[0142] Step S1109: The global model fills in pixels in the image to be processed and optimizes it globally to obtain the target image, i.e., the complete G-channel image.

[0143] Step S1110: Interpolate the R and B channel images based on color constancy to obtain complete R channel and B channel images;

[0144] Step S1111: Output an RGB image based on the complete R channel image, G channel image, and B channel image, thereby completing the image de-mosaic processing.

[0145] Exemplary embodiments of this disclosure also provide an image processing apparatus. (Reference) Figure 12 As shown, the image processing apparatus 1200 may include:

[0146] Image acquisition module 1210 is configured to acquire an image to be processed; the image to be processed includes pixels to be filled.

[0147] The convolution processing module 1220 is configured to perform convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged according to a preset direction; the intermediate image does not include pixels to be filled.

[0148] The optimization processing module 1230 is configured to optimize the intermediate image using a pre-trained image optimization model to obtain the target image.

[0149] In one embodiment, the above-mentioned convolution processing of the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes:

[0150] A fixed convolution kernel is used to perform convolution operations on the local regions centered on each pixel to be filled in the image to be processed, so as to obtain the intermediate pixel value corresponding to each pixel to be filled.

[0151] The intermediate image is obtained by filling the pixels in the image to be filled with the intermediate pixel value corresponding to each pixel to be filled.

[0152] In one implementation, the number of fixed convolutional kernels is multiple, and different fixed convolutional kernels have different sizes or different parameter arrangement characteristics.

[0153] In one embodiment, the above-mentioned convolution processing of the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes:

[0154] The image to be processed is convolved using each fixed convolution kernel to obtain multiple intermediate images;

[0155] The above-mentioned optimization of intermediate images using a pre-trained image optimization model to obtain the target image includes:

[0156] The target image is obtained by fusing and optimizing multiple intermediate images using an image optimization model.

[0157] In one embodiment, before performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel, the method further includes:

[0158] Obtain a pre-trained global model, which includes fixed convolutional layers and an image optimization model. The fixed convolutional layers include fixed convolutional kernels.

[0159] The above-mentioned convolution processing of the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes:

[0160] The image to be processed is input into the global model, and the image is processed by convolution through a fixed convolutional layer to obtain an intermediate image.

[0161] In one embodiment, the image processing apparatus 1200 may further include a model training module configured to:

[0162] Obtain the first sample image and the corresponding label image. The first sample image includes pixels to be filled, while the label image does not include pixels to be filled.

[0163] The first sample image is input into the global model to be trained, and the second sample image is output.

[0164] With the parameters of the convolution kernel fixed, the parameters of the global model are updated based on the second sample image and the label image.

[0165] In one embodiment, obtaining the first sample image and the label image corresponding to the first sample image includes:

[0166] Get the label image; all pixels in the label image are not empty.

[0167] Set some pixels in the label image to null values ​​to obtain the first sample image corresponding to the label image; the null pixels in the first sample image are the pixels to be filled.

[0168] In one embodiment, the above-mentioned acquisition of the image to be processed includes:

[0169] Obtain the G channel image from the Bayer image; the G channel image includes filled pixels and unfilled pixels, the G pixel values ​​of filled pixels in the G channel image are not null, and the G pixel values ​​of unfilled pixels in the G channel image are null.

[0170] The G channel image is subjected to green balance processing to obtain the image to be processed.

[0171] In one embodiment, the above-described green balance processing of the G channel image to obtain the image to be processed includes:

[0172] Obtain the difference between each filled pixel in the G-channel image and its neighboring filled pixels;

[0173] The compensation value for each filled pixel is determined based on the difference between each filled pixel and its neighboring filled pixels.

[0174] In the G-channel image, each filled pixel is compensated using the compensation value of each filled pixel to obtain the image to be processed.

[0175] In one implementation, each filled pixel in the G-channel image has four adjacent filled pixels; the determination of the compensation value for each filled pixel based on the difference between each filled pixel and its adjacent filled pixels includes:

[0176] For any filled pixel i in the G channel image, compare the four differences between the filled pixel i and its four adjacent filled pixels with the green balance threshold of the filled pixel i, and take the difference greater than the green balance threshold as the valid difference of the filled pixel i.

[0177] The equivalent difference of the filled pixel i is determined based on the four differences of the filled pixel i.

[0178] The compensation value of the filled pixel i is determined based on the number of effective differences of the filled pixel i and the equivalent difference of the filled pixel i.

[0179] In one implementation, determining the equivalent difference of filled pixel i based on the four differences of filled pixel i includes:

[0180] The product of the average of the four differences of the filled pixel i and the first compensation coefficient is used as the equivalent difference of the filled pixel i; the first compensation coefficient is in the range of [0,1].

[0181] In one implementation, determining the compensation value of the filled pixel i based on the number of effective differences of the filled pixel i and the equivalent differences of the filled pixel i includes:

[0182] The second compensation coefficient is determined based on the number of valid differences of filled pixel i, and the second compensation coefficient is in the range of [0,1].

[0183] The product of the equivalent difference of the filled pixel i and the second compensation coefficient is used as the compensation value of the filled pixel i.

[0184] In one embodiment, the above-described green balance processing of the G channel image to obtain the image to be processed further includes:

[0185] Determine the grayscale value of each filled pixel in the G-channel image;

[0186] Based on the grayscale value of each filled pixel, a green balance threshold is determined for each filled pixel. The green balance threshold is positively correlated with the grayscale value.

[0187] In one embodiment, the image processing apparatus 1200 may further include a pixel filling module, configured to:

[0188] Obtain the R-channel and B-channel images from the Bayer image; the R-channel image includes filled pixels and unfilled pixels, the R-pixel values ​​of filled pixels in the R-channel image are not null, and the R-pixel values ​​of unfilled pixels in the R-channel image are null; the B-channel image includes filled pixels and unfilled pixels, the B-pixel values ​​of filled pixels in the B-channel image are not null, and the B-pixel values ​​of unfilled pixels in the B-channel image are null.

[0189] Based on the difference between the R pixel value of the filled pixel in the R channel image and the G pixel value of the pixel at the same position in the target image, fill the pixel to be filled in the R channel image with the R pixel value.

[0190] Based on the difference between the B pixel value of the filled pixel in the B channel image and the G pixel value of the pixel at the same position in the target image, fill the pixel to be filled in the B channel image with the B pixel value.

[0191] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0192] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0193] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0194] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0195] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0196] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] Exemplary embodiments of this disclosure also provide an electronic device, such as the terminal 110 or server 120 described above. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes these executable instructions to perform the image processing method of this exemplary embodiment.

[0198] The following is based on Figure 13 Taking the mobile terminal 1300 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 13 The structure can also be applied to fixed types of equipment.

[0199] like Figure 13 As shown, the mobile terminal 1300 may specifically include: a processor 1301, a memory 1302, a bus 1303, a mobile communication module 1304, an antenna 1, a wireless communication module 1305, an antenna 2, a display screen 1306, a camera module 1307, an audio module 1308, a power module 1309, and a sensor module 1310.

[0200] Processor 1301 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The image processing method in this exemplary embodiment can be executed by one or more of the AP, GPU, ISP, DSP, and NPU.

[0201] The processor 1301 can be connected to the memory 1302 or other components via the bus 1303.

[0202] The memory 1302 can be used to store computer executable program code, which includes instructions. The processor 1301 executes various functional applications and data processing of the mobile terminal 1300 by running the instructions stored in the memory 1302. The memory 1302 can also store application data, such as images, videos, and other files.

[0203] The communication function of mobile terminal 1300 can be implemented through mobile communication module 1304, antenna 1, wireless communication module 1305, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1304 can provide 3G, 4G, and 5G mobile communication solutions for mobile terminal 1300. Wireless communication module 1305 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1300.

[0204] The display screen 1306 is used to implement display functions, such as displaying user interfaces, images, videos, etc.

[0205] The camera module 1307 is used to implement the shooting function. In one embodiment, the ISP can be set in the camera module 1307, and the ISP can acquire Bayer images, including R channel images, G channel images, and B channel images.

[0206] The audio module 1308 is used to implement audio functions, such as playing audio and capturing voice.

[0207] The power module 1309 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status.

[0208] The sensor module 1310 may include one or more sensors for implementing corresponding sensing and detection functions.

[0209] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0210] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0211] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; the image to be processed includes pixels to be filled; The image to be processed is convolved by a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged in a preset direction; the intermediate image does not include pixels to be filled. The intermediate image is optimized using a pre-trained image optimization model to obtain the target image; The process of acquiring the image to be processed includes: Obtain the G channel image from the Bayer image; the G channel image includes filled pixels and unfilled pixels, the G pixel values ​​of the filled pixels in the G channel image are not null, and the G pixel values ​​of the unfilled pixels in the G channel image are null; perform green balance processing on the G channel image to obtain the image to be processed; The step of performing green balance processing on the G channel image to obtain the image to be processed includes: Obtain the difference between each filled pixel and its neighboring filled pixels in the G-channel image; determine the compensation value of each filled pixel based on the difference between each filled pixel and its neighboring filled pixels; compensate each filled pixel in the G-channel image using the compensation value of each filled pixel to obtain the image to be processed; The step of performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes: using the fixed convolution kernel to perform convolution operations on local regions centered on each pixel to be filled in the image to be processed, to obtain an intermediate pixel value corresponding to each pixel to be filled; and filling the pixels to be filled in the image to be processed with the intermediate pixel values ​​corresponding to each pixel to be filled, to obtain the intermediate image.

2. The method according to claim 1, characterized in that, The number of fixed convolutional kernels is multiple, and different fixed convolutional kernels have different sizes or different parameter arrangement characteristics.

3. The method according to claim 2, characterized in that, The step of performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes: The image to be processed is convolved using each of the fixed convolution kernels to obtain multiple intermediate images; The step of optimizing the intermediate image using a pre-trained image optimization model to obtain the target image includes: The target image is obtained by fusing and optimizing the multiple intermediate images using the image optimization model.

4. The method according to claim 1, characterized in that, Before performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel, the method further includes: Obtain a pre-trained global model, the global model including a fixed convolutional layer and the image optimization model, the fixed convolutional layer including the fixed convolutional kernel; The step of performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes: The image to be processed is input into the global model, and the image to be processed is convolved through the fixed convolutional layer to obtain the intermediate image.

5. The method according to claim 4, characterized in that, The global model is trained in the following way: Obtain a first sample image and a label image corresponding to the first sample image, wherein the first sample image includes pixels to be filled, and the label image does not include pixels to be filled; The first sample image is input into the global model to be trained, and the second sample image is output. With the parameters of the fixed convolution kernel fixed, the parameters of the global model are updated based on the second sample image and the label image.

6. The method according to claim 5, characterized in that, The step of obtaining the first sample image and the label image corresponding to the first sample image includes: The label image is obtained; all pixels in the label image are non-empty. By setting some pixels in the label image to null values, the first sample image corresponding to the label image is obtained; the null pixels in the first sample image are pixels to be filled.

7. The method according to claim 1, characterized in that, Each filled pixel in the G-channel image has four adjacent filled pixels; The step of determining the compensation value for each filled pixel based on the difference between each filled pixel and its neighboring filled pixels includes: For any filled pixel in the G channel image, the four differences between the filled pixel and its four adjacent filled pixels are compared with the green balance threshold of the filled pixel. The difference greater than the green balance threshold is taken as the valid difference of the filled pixel. The equivalent difference of any filled pixel is determined based on the four differences of any filled pixel; The compensation value of any filled pixel is determined based on the number of effective differences of any filled pixel and the equivalent difference of any filled pixel.

8. The method according to claim 7, characterized in that, Determining the equivalent difference value of any filled pixel based on the four differences of any filled pixel includes: The product of the average of the four differences of any filled pixel and the first compensation coefficient is taken as the equivalent difference of any filled pixel; the first compensation coefficient is in the range of [0,1].

9. The method according to claim 7, characterized in that, The step of determining the compensation value of any filled pixel based on the number of effective differences of any filled pixel and the equivalent difference of any filled pixel includes: A second compensation coefficient is determined based on the number of valid differences for any of the filled pixels, and the second compensation coefficient is in the range of [0,1]. The product of the equivalent difference of any filled pixel and the second compensation coefficient is used as the compensation value of any filled pixel.

10. The method according to claim 7, characterized in that, The step of performing green balance processing on the G channel image to obtain the image to be processed further includes: Determine the grayscale value of each filled pixel in the G-channel image; Based on the grayscale value of each filled pixel, a green balance threshold is determined for each filled pixel, and the green balance threshold is positively correlated with the grayscale value.

11. The method according to claim 1, characterized in that, After obtaining the target image, the method further includes: Obtain the R-channel image and B-channel image from the Bayer image; the R-channel image includes filled pixels and unfilled pixels, the R-pixel values ​​of the filled pixels in the R-channel image are not null, and the R-pixel values ​​of the unfilled pixels in the R-channel image are null; the B-channel image includes filled pixels and unfilled pixels, the B-pixel values ​​of the filled pixels in the B-channel image are not null, and the B-pixel values ​​of the unfilled pixels in the B-channel image are null. The pixels to be filled in the R channel image are filled with R pixel values ​​based on the difference between the R pixel values ​​of the filled pixels in the R channel image and the G pixel values ​​of the pixels at the same position in the target image. The pixels to be filled in the B channel image are filled with B pixel values ​​based on the difference between the B pixel value of the filled pixels in the B channel image and the G pixel value of the pixels at the same position in the target image.

12. An image processing apparatus, characterized in that, include: An image acquisition module is configured to acquire an image to be processed; the image to be processed includes pixels to be filled. The convolution processing module is configured to perform convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image; the fixed convolution kernel has fixed parameters, and the parameters in the fixed convolution kernel are arranged according to a preset direction; the intermediate image does not include pixels to be filled. The optimization processing module is configured to optimize the intermediate image using a pre-trained image optimization model to obtain the target image; The process of acquiring the image to be processed includes: Obtain the G channel image from the Bayer image; the G channel image includes filled pixels and unfilled pixels, the G pixel values ​​of the filled pixels in the G channel image are not null, and the G pixel values ​​of the unfilled pixels in the G channel image are null; perform green balance processing on the G channel image to obtain the image to be processed; The step of performing green balance processing on the G channel image to obtain the image to be processed includes: Obtain the difference between each filled pixel and its neighboring filled pixels in the G-channel image; determine the compensation value of each filled pixel based on the difference between each filled pixel and its neighboring filled pixels; compensate each filled pixel in the G-channel image using the compensation value of each filled pixel to obtain the image to be processed; The step of performing convolution processing on the image to be processed using a pre-configured fixed convolution kernel to obtain an intermediate image includes: using the fixed convolution kernel to perform convolution operations on local regions centered on each pixel to be filled in the image to be processed, to obtain an intermediate pixel value corresponding to each pixel to be filled; and filling the pixels to be filled in the image to be processed with the intermediate pixel values ​​corresponding to each pixel to be filled, to obtain the intermediate image.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 11.

14. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 11 by executing the executable instructions.

Images