Image processing method and device, electronic equipment, storage medium and product

Abstract

The present disclosure relates to an image processing method, device, electronic equipment, storage medium and product. The image processing method comprises: acquiring an original image output by a color filter array, N×N same-color pixels in the color filter array being arranged adjacently; separating each color channel in three primary color channels of the original image, and dividing each color channel obtained by separation into a plurality of image blocks in units of N×N; respectively extracting semantic features of each image block for each color channel, and aligning each color channel in the three primary color channels; for each color channel obtained by alignment, respectively completing semantic features of image blocks in other color channels based on semantic features of image blocks in the respective color channel, to obtain a target image. Through the present disclosure, a target image with rich details and enhanced image quality can be obtained.

Description

Technical Field

[0001] This disclosure relates to imaging, and more particularly to image processing methods, apparatus, electronic devices, storage media, and products. Background Technology

[0002] In recent years, imaging equipment has developed rapidly and is widely used in many fields such as electronic devices, photography, and security. Imaging systems for images typically obtain the three primary colors (color, light, and dark) through color filtering using a Bayer array. Since each pixel in a Bayer array can only capture one of the three primary colors, de-mosaic processing is required to obtain an image with all three primary color channels. However, this process can introduce errors, leading to inaccurate color reproduction and affecting image quality. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and product.

[0004] According to a first aspect of some embodiments of this disclosure, an image processing method is provided, comprising: acquiring an original image output by a color filter array, wherein N×N pixels of the same color are arranged adjacently in the color filter array; separating each color channel in the three primary color channels of the original image, and dividing each separated color channel into multiple image blocks in units of N×N; extracting semantic features of each image block for each color channel, and aligning each color channel in the three primary color channels; and, for each aligned color channel, supplementing the semantic features of image blocks in other color channels based on the semantic features of the image blocks in their respective color channels, thereby obtaining a target image.

[0005] In one embodiment, extracting the semantic features of each image patch includes: performing a linear mapping on each image patch to obtain a mapping vector; adding a positional code to the mapping vector, wherein the positional code represents the pixel position of the image patch in the original image, and each image patch corresponds to a positional code; performing mask coding on the mapping vector based on the positional code, wherein the mapping vector with added positional code is the vector region to be encoded, and the mapping vector without added positional code is the vector region to be masked; and extracting the semantic features of the mask-coded mapping vector to obtain the semantic features of each image patch.

[0006] In one embodiment, aligning each color channel in the three primary color channels includes: for each color channel, adding a predicted image block at a position different from the existing image block, so as to align each color channel in the three primary color channels; wherein, the positions of the existing image block and the predicted image block in each color channel of the aligned three primary color channels are complementary.

[0007] In one embodiment, extracting the semantic features of each image patch includes: performing a linear mapping on each image patch to obtain a mapping vector; adding a positional code to the mapping vector, wherein the positional code represents the pixel position of the image patch in the original image, and each image patch corresponds to a positional code; performing mask coding on the mapping vector based on the positional code, wherein the mapping vector with added positional code is the vector region to be encoded, and the mapping vector without added positional code is the vector region to be masked; and extracting the semantic features of the mask-coded mapping vector to obtain the semantic features of each image patch.

[0008] In one implementation, the value of N is 64.

[0009] According to a second aspect of some embodiments of this disclosure, an image processing apparatus is provided, comprising: an acquisition unit for acquiring an original image output by a color filter array, wherein N×N pixels of the same color are arranged adjacently in the color filter array; a processing unit for separating each color channel in the three primary color channels of the original image, and dividing each separated color channel into multiple image blocks in units of N×N; extracting semantic features of each image block for each color channel, and aligning each color channel in the three primary color channels; and a determination unit for, for each aligned color channel, supplementing the semantic features of image blocks in other color channels based on the semantic features of the image blocks in their respective color channels, to obtain a target image.

[0010] In one embodiment, the processing unit extracts semantic features of each image block in the following manner: linearly mapping each image block to obtain a mapping vector; adding positional codes to the mapping vectors, wherein the positional codes represent the pixel positions of the image blocks in the original image, and each image block corresponds to a positional code; performing mask coding on the mapping vectors based on the positional codes, wherein the mapping vectors with added positional codes are the vector regions that need to be encoded, and the mapping vectors without added positional codes are the vector regions that need to be masked; and extracting the semantic features of the mask-coded mapping vectors to obtain the semantic features of each image block.

[0011] In one embodiment, the processing unit aligns each color channel in the three primary color channels in the following manner: for each color channel, an image block to be predicted is added at a position different from the existing image block, so as to align each color channel in the three primary color channels; wherein, the positions of the existing image block and the image block to be predicted in each color channel of the aligned three primary color channels are complementary.

[0012] In one embodiment, the determining unit completes the semantic features of image blocks in other color channels based on the semantic features of image blocks in their respective color channels to obtain a target image: For each target color channel, based on the semantic features of target image blocks in other color channels different from the target color channel, the semantic features of the image block to be predicted are predicted, and the predicted semantic features are completed for the image block to be predicted. The target image block is an existing image block in other color channels that is complementary in position to the image block to be predicted, and the target color channel is each of the three primary color channels. The existing image blocks that are complementary in position in the three primary color channels and the image block to be predicted are fused to obtain the target image.

[0013] In one implementation, the value of N is 64.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the image processing method described in the first aspect or any embodiment of the first aspect.

[0015] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the image processing method described in the first aspect or any embodiment of the first aspect.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, implements the image processing method described in the first aspect or any embodiment of the first aspect.

[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: Obtaining the original image output by a color filter array, wherein the color filter array has N×N adjacent pixels of the same color, which ensures both a simpler filter manufacturing process and enhanced semantic features. Based on the obtained original image, each color channel in the three primary color channels of the original image is separated, and each separated color channel is divided into multiple image blocks in N×N units. This allows for the extraction of semantic features from each image block for each color channel, and the alignment of each color channel in the three primary color channels. For each aligned color channel, based on the semantic features of the image blocks in its own color channel, the semantic features of the image blocks in other color channels are supplemented. Based on mutual learning of semantic features, a target image with enhanced image quality is obtained.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure.

[0021] Figure 2 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure.

[0022] Figure 3 This is a schematic diagram illustrating an N×N pixel arrangement of the same color according to some embodiments of the present disclosure.

[0023] Figure 4 This is a flowchart illustrating an extraction method according to some embodiments of the present disclosure.

[0024] Figure 5 This is a flowchart illustrating a method for determining a target image according to some embodiments of the present disclosure.

[0025] Figure 6 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure.

[0026] Figure 7 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure.

[0027] Figure 8 This is a block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0028] Figure 9 This is a block diagram illustrating an apparatus for image processing according to some embodiments of the present disclosure.

[0029] Figure 10 This is a block diagram illustrating an apparatus for image processing according to some embodiments of the present disclosure. Detailed Implementation

[0030] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0031] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] The image processing method provided in this disclosure is applied to image processing scenarios in camera imaging systems of various natural image imaging devices. Image processing in camera imaging systems is typically based on hardware designs using complementary metal-oxide-semiconductor (CMOS) sensor color filters.

[0033] In related technologies, image sensors can only detect brightness information. Therefore, to restore color information, various color filter arrays are designed for color filtering. Examples include Bayer arrays, Qued Bayer arrays, and RGBW arrays. The RGBW array is a display technology that includes four color channels: red (R), green (G), blue (B), and white (W).

[0034] The Bayer array consists of filter pixels for RGB colors, arranged in a repeating RGGB pattern. Since the human eye is more sensitive to green light than the other two colors, green information is crucial for perceiving image details and brightness; therefore, the number of green pixels in the Bayer array is twice that of red and blue pixels. However, when light passes through the Bayer array, each pixel can only capture one color of information; that is, each color filter pixel only allows light of that color to pass through and blocks light of other colors. Therefore, after passing through the Bayer array to output the raw image, it is usually necessary to perform operations such as demosaicing, white balance correction, and image quality enhancement to obtain an RGB full-channel image.

[0035] Quad Bayer is an extension and optimization of the Bayer array. Compared to the Bayer array, Quad Bayer achieves higher image quality and better performance by placing four sub-pixels at the position of each Bayer array pixel. Users can choose to output high-resolution images or high signal-to-noise ratio images.

[0036] The RGBW arrangement adds a white pixel to the traditional RGB pixels to improve image quality and reduce noise in low-light conditions. In addition, the white pixel can collect more light, improving image quality in low-light conditions. By increasing light sensitivity, it can increase image brightness without increasing noise and helps balance colors and reduce color deviation.

[0037] The following combination Figure 1 A brief explanation of the arrangement of the three types of filter arrays and the image processing flow is provided above.

[0038] Figure 1 This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure. For example... Figure 1 As shown, the first row represents the processing flow of Qued Bayer, based on... Figure 1 As we know, in a Qued Bayer array, four pixels of the same color are adjacent. For the raw image output by the Qued Bayer array, since every four pixels are distributed together, it needs to be reconstructed into a Bayer array using a remosaic algorithm, and then interpolated using demosaic to obtain a full-channel RGB image. As seen in the second row of the Bayer array, each pixel contains one color, and adjacent pixels have different colors. Therefore, the raw image needs to be interpolated using demosaic to obtain an image with complete RGB color channels. For the RGBW array in the third row, methods based on interpolation, least squares optimization, or deep learning can obtain a full-channel RGB image. However, while adding white pixels to capture full-band information increases sampling detail, it lacks color information, and the interpolation process is more difficult.

[0039] In related technologies, a color filter array (CFA) composed of randomly distributed pixels has been proposed, where the R, G, and B channels each account for 33% of the entire CFA pattern. This is combined with a variational autoencoder (VAE) algorithm to obtain a full-color RGB image. However, this method incorporates random Gaussian noise to output a monochrome holed image to reconstruct the full-color RGB image. The key is to separate the dependency between the input array (pattern) and the trained model, thus enabling de-mosaicing even with random pattern inputs. The random pattern design makes the task more focused on low-level visual recovery rather than semantic feature extraction. Furthermore, the random distribution design results in irregular RGB filter distribution, leading to manufacturing difficulties. The VAE iteratively adds random noise during training, resulting in high computational cost and making end-to-end applications difficult. Moreover, since the goal is low-level visual recovery, detailed scene sampling information is easily lost, leading to low image quality.

[0040] In view of this, this disclosure proposes an image processing method that changes the arrangement of the filter array and utilizes a masked autoencoder (MAE) algorithm to reduce computational load and memory consumption by operating on partial image blocks, thereby simplifying the complex image processing workflow. Based on the semantic features of image blocks, it captures image details and color information, achieving the conversion from the original image to an RGB full-channel image and improving image quality.

[0041] Figure 2 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure. Figure 2 As shown, it includes the following steps.

[0042] In step S11, the original image output by the color filter array is obtained.

[0043] In this embodiment, the color filter array consists of N×N adjacent pixels of the same color. The color filter array can include RGB colors and is arranged in RGGB mode, such as... Figure 3 As shown, Figure 3 This is a schematic diagram illustrating an N×N pixel arrangement of the same color according to some embodiments of this disclosure. Figure 3 The top left corner can be composed of N×N adjacent green pixels. Figure 3 The top right corner can be composed of N×N adjacent red pixels arranged in a row.

[0044] In this embodiment of the disclosure, the original image can be understood as the image output by a color filtering array. Each pixel of the original image contains light intensity information of a specific single color. Since the color filtering array includes three colors: red, green, and blue, the specific single color in the original image is a monochrome pixel such as red, green, or blue.

[0045] In step S12, each color channel in the three primary color channels of the original image is separated, and each separated color channel is divided into multiple image blocks in units of N×N.

[0046] In this embodiment of the disclosure, each color channel in the three primary color channels of the original image is separated to obtain a red channel, a green channel, and a blue channel. It is understood that, since the color filter array is arranged in RGGB mode, the number of green channels in the separated three primary color channels is twice the number of red channels, and the number of green channels is also twice the number of blue channels.

[0047] In this embodiment of the disclosure, each color channel is divided into N×N units of pixels to obtain multiple image blocks under each color channel. It can be understood that since N×N pixels of the same color are arranged adjacently in the color filter array, the color in each of the multiple image blocks obtained by dividing the color into N×N units is the same color.

[0048] In step S13, semantic features of each image block are extracted for each color channel, and each color channel in the three primary color channels is aligned.

[0049] In this embodiment of the disclosure, each color channel corresponds to multiple image blocks, and semantic features of each image block in each color channel are extracted. Semantic features refer to the meaning and information expressed by the image block, and can be associated with the object, scene, attribute, or structure represented by the image block.

[0050] In this embodiment of the disclosure, image blocks of other color channels that are different from the current color can be predicted based on the semantic features of each image block of each color channel. Therefore, each color channel in the three primary color channels is first aligned in order to predict image blocks of other color channels.

[0051] In some embodiments, since the color filter array is arranged in an RGGB pattern, R accounts for 25% of the color filter array, G accounts for 50%, and B accounts for 25%. Semantic features of each image patch are extracted for the red channel. Since the red channel does not include the green and blue channels, it currently has 75% empty image patches, meaning it lacks image patches representing the green and blue channels. Therefore, after obtaining the semantic features of each image patch in the red channel, the red channel image patches are aligned with the image patches of the blue and green channels to obtain multiple image patches including the three primary color channels. It is understood that the above processing is performed on each color channel.

[0052] In some embodiments, semantic features of each image block are extracted for the green channel. Since the green channel does not include the red and blue channels, the green channel currently has 50% of empty image blocks, that is, the green channel is missing image blocks of the red and blue channels. Therefore, after obtaining the semantic features of each image block of the green channel, the image blocks of the green channel are aligned with the image blocks of the blue and red channels to obtain multiple image blocks including the three primary color channels.

[0053] In some embodiments, semantic features of each image block are extracted for the blue channel. Since the blue channel does not include the green and red channels, the blue channel currently has 75% of empty image blocks, that is, the blue channel is missing image blocks of the green and red channels. Therefore, after obtaining the semantic features of each image block of the blue channel, the image blocks of the blue channel are aligned with the image blocks of the red and green channels to obtain multiple image blocks including the three primary color channels.

[0054] In step S14, for each color channel obtained by alignment, the semantic features of the image blocks in other color channels are completed based on the semantic features of the image blocks in their respective color channels to obtain the target image.

[0055] In this embodiment, other color channels in the three primary colors that are different from the current color are supplemented, so that each aligned color channel includes the three primary colors, i.e., each aligned color channel. Based on the semantic features of the image patches in each color channel, the semantic features of the image patches in the other color channels in the three primary colors that are different from the current color are supplemented, thereby obtaining the target image.

[0056] In some embodiments of this disclosure, the red channel is used as an example for illustration. The red channel includes multiple image blocks. The semantic features of each image block in the red channel are extracted and then supplemented into the blue and green channels, resulting in a color channel that includes multiple image blocks of red, green, and blue channels. In this channel, since the semantic features of each image block in the red channel are known, i.e., the semantic features of the image blocks in the red channel are used to supplement the semantic features of the image blocks in the green and blue channels.

[0057] In some embodiments of this disclosure, for the green channel, semantic features of each image block in multiple image blocks of the green channel are extracted, and the red and blue channels are supplemented to obtain color channels for multiple image blocks including red, blue, and green channels. For the blue channel, semantic features of each image block in multiple image blocks of the blue channel are extracted, and the red and green channels are supplemented to obtain color channels for multiple image blocks including red, blue, and green channels. Based on the above three channels and the semantic features of each image block in each color channel, the target image is obtained. It can be understood that the target image is an image based on the semantic features of each image block.

[0058] In this embodiment, the N×N color filter array, due to its large stride, saves on manufacturing processes. Furthermore, the dense distribution of the N×N color filter array improves the accuracy of semantic features. Dividing the image into N×N units yields multiple regular, non-overlapping image blocks. By extracting the semantic features of each image block for each color channel, predictions can be made for other color channels within each color channel. Based on the semantic features of the image blocks in each color channel, the semantic features of image blocks in other color channels are supplemented, resulting in a target image with clear details and rich colors.

[0059] In this embodiment of the disclosure, when extracting semantic features of each image patch for each color channel, the MAE algorithm can be used for extraction. Based on the MAE algorithm and an N×N color filter array, rich semantic features in the image patch can be extracted. The MAE algorithm is described below.

[0060] In this embodiment, MAE is a self-supervised learning method, including random masking, an encoder, feature extraction, a decoder, and a reconstruction part. In the random masking part, MAE randomly selects a portion of the input image, typically 75% of the pixels, and sets the masked portion to 0 or a specific value, while the remaining unmasked pixels retain their original values. The size of the masked pixels is not limited. The masked pixel portion can be understood as a vector region removed by the mask. In the encoder part, the unmasked image is input into the encoder; the unmasked image portion can be understood as the vector region to be encoded. The encoder is typically a transformer model, composed of multiple self-attention layers and a feedforward network. In the feature extraction part, the encoder outputs feature representations of the unmasked pixels, which can be used for image reconstruction. In the decoder part, the decoder receives the feature representations output by the encoder and the masked portion, reconstructing the masked portion of the original image. The decoder is typically a lightweight neural network. In the reconstruction part, the encoder reconstructs the masked pixels based on the provided features, and uses reconstruction loss, such as mean squared error (MSE), to measure the difference between the reconstructed pixels and the original pixels, thereby obtaining the reconstructed full-color RGB image.

[0061] In the MAE algorithm, 75% of the pixels are typically masked randomly. Since the three primary color channels in the original image are separated and processed separately in the above embodiments of this disclosure, for the red channel, the color channels other than the red channel are the masked portions. For the green channel, the color channels other than the green channel are the masked portions, and for the blue channel, the color channels other than the blue channel are the masked portions.

[0062] Figure 4 This is a flowchart illustrating an extraction method according to some embodiments of this disclosure. For example... Figure 4 As shown, it includes the following steps.

[0063] In step S21, each image block is linearly mapped to obtain a mapping vector.

[0064] In this embodiment of the disclosure, each image block in multiple image blocks of each color channel is linearly mapped to obtain a mapping vector for each image block. Based on the mapping vector, multiple image blocks can be flattened into a one-dimensional vector. The mapping vector contains feature information of the image blocks, such as semantic features. Based on the semantic features, it can be used to reconstruct the image or perform other tasks during the encoding and decoding process.

[0065] In step S22, positional encoding is added to the mapping vector.

[0066] In this embodiment of the disclosure, the positional encoding represents the pixel position of an image patch in the original image, and each image patch corresponds to a positional encoding. Positional encoding is added to the mapping vector so that each image patch corresponds to specific positional information. Positional encoding helps in understanding the relative positional relationships between different pixels in the image during image processing.

[0067] In this embodiment of the disclosure, the sin-cos function can be used to generate position codes to capture the relative relationships between different positions. This disclosure does not limit the method of generating position codes.

[0068] In step S23, the mapping vector is masked based on position encoding.

[0069] In this embodiment of the disclosure, masking the mapping vector can be understood as determining the masked portions of other color channels for each color channel and obtaining the semantic features of each image patch in each color channel. The masked portions are the vector regions removed by the mask, and the masked portions are the vector regions that need to be encoded.

[0070] In this embodiment of the disclosure, for each color channel, the mapping vector with added positional encoding is the vector region to be encoded, and the mapping vector without added positional encoding is the vector region to be masked. It should be noted that each image block corresponds to one positional encoding, but for multiple image blocks in each color channel, multiple image blocks in other color channels can be understood as having no added positional encoding. For example, in the red, green, and blue channels, each image block in each color channel corresponds to one positional encoding. However, for the green channel, multiple image blocks in the green channel have their own positional encodings. Since the red and blue channels are missing in the green channel, for the green channel, multiple image blocks in the red and blue channels are image blocks without added positional encoding, i.e., vector regions to be masked, while the image blocks in the green channel are vector regions to be encoded.

[0071] In this embodiment of the disclosure, for the red channel, multiple image blocks in the red channel have their own position codes. However, the green and blue channels are missing in the red channel. Therefore, for the red channel, multiple image blocks in the green and blue channels are image blocks without position codes, i.e., vector regions with mask removal, while each image block in the red channel is a vector region that needs to be encoded.

[0072] In this embodiment of the disclosure, for the blue channel, multiple image blocks in the blue channel have their own position codes. However, the green and red channels are missing in the blue channel. Therefore, for the blue channel, multiple image blocks in the green and red channels are image blocks without position codes, i.e., vector regions with mask removal, while each image block in the blue channel is a vector region that needs to be encoded.

[0073] In step S24, the semantic features of the mask-encoded mapping vector are extracted to obtain the semantic features of each image block.

[0074] In this embodiment of the disclosure, for each color channel, a vector region that needs to be encoded is determined, and based on the vector region that needs to be encoded, the semantic features of the mask-encoded mapping vector are extracted to obtain the semantic features of each image block in each color channel.

[0075] In this embodiment of the disclosure, semantic features of each image block in each color channel are extracted to obtain semantic features such as scene or structure that can characterize each image block in the color channel. This facilitates the prediction of semantic features of other color channels based on the semantic features of each image block in the color channel, resulting in a semantically complete and detailed target image.

[0076] In this embodiment of the disclosure, the semantic features of each image block in each color channel are obtained, and the semantic features of each image block in other color channels are predicted based on the semantic features of the color channel. However, before the prediction, it is necessary to align the other missing color channels in each color channel.

[0077] In this embodiment of the disclosure, for each color channel, a to-be-predicted image block is added at a position different from the existing image blocks. The to-be-predicted image block can be understood as an image block in other color channels excluding the current color channel. For example, for the red channel, the semantic features of each image block in the red channel are known, while multiple image blocks in the green and blue channels are to-be-predicted image blocks. Similarly, for the green channel, the semantic features of each image block in the green channel are known, while multiple image blocks in the red and blue channels are to-be-predicted image blocks. Likewise, for the blue channel, the semantic features of each image block in the blue channel are known, while multiple image blocks in the red and green channels are to-be-predicted image blocks.

[0078] In this embodiment of the disclosure, since a positional code is added to each image block during linear mapping, the image blocks to be predicted for other color channels in each color channel can be completed based on the positional code. For example, for the red channel, the green and blue channels need to be completed. Based on the positional code of each image block in each color channel, the missing green and blue channels in the red channel are completed, forming a one-dimensional vector. The above processing is performed on both the green and blue channels to align each color channel in the three primary color channels, thereby obtaining three color channels including the three primary colors.

[0079] Understandably, the three color channels, each containing the three primary colors, include semantic features of image patches in the red channel, and image patches to be predicted in the green and blue channels. Furthermore, the aligned color channels already contain complementary positions between image patches and image patches to be predicted, allowing for the reconstruction of each pixel position in the original image.

[0080] In this embodiment of the disclosure, aligning each color channel ensures that the positions of image blocks in each color channel are complementary, which can improve image quality and enhance the accuracy and reliability of image processing.

[0081] In this embodiment of the disclosure, the semantic features of image blocks in other color channels are supplemented based on the semantic features of image blocks in their respective color channels to obtain the target image. Figure 5 This is a flowchart illustrating a method for determining a target image according to some embodiments of this disclosure. Figure 5 As shown, it includes the following steps.

[0082] In step S31, for each target color channel, based on the semantic features of the target image patch in other color channels different from the target color channel, the semantic features of the image patch to be predicted are predicted, and the predicted semantic features are completed for the image patch to be predicted.

[0083] In this embodiment, the target color channel is each of the three primary color channels, and the target image block is an existing image block in other color channels that is complementary in position to the image block to be predicted. The existing image block is an image block with consistent semantic features. For each of the aligned primary color channels, one color channel is determined as the target color channel. Based on the semantic features of the target image blocks in other color channels different from the target color channel, the image block to be predicted in the target color channel is predicted to obtain its semantic features. Since each of the three aligned color channels contains image blocks with known semantic features, as well as the image block to be predicted, the target color channel can be predicted based on the image blocks with known semantic features in the target color channel and the semantic features of the target image blocks in other color channels.

[0084] In this embodiment of the disclosure, for ease of description, the channel comprising image blocks of the red channel, the blue channel, and / or the green channel to be predicted is referred to as the first color channel, wherein each image block of the red channel corresponds to a semantic feature. The channel comprising image blocks of the green channel, the blue channel, and / or the red channel to be predicted is referred to as the second color channel, wherein each image block of the green channel corresponds to a semantic feature. The channel comprising image blocks of the blue channel, the red channel, and / or the green channel to be predicted is referred to as the third color channel, wherein each image block of the blue channel corresponds to a semantic feature.

[0085] In this embodiment of the disclosure, taking the red channel as the target color channel as an example, the process of obtaining semantic features through completion prediction is explained. If the semantic features of the image blocks to be predicted in the blue and green channels of the first color channel need to be completed, completion prediction needs to be performed based on the semantic features of each image block in the first color channel, as well as the semantic features of the existing image blocks in the second and third color channels. The existing image blocks in the second and third color channels are the target image blocks.

[0086] In this embodiment of the disclosure, taking the green channel as the target color channel as an example, the process of obtaining semantic features through completion prediction is explained. If the semantic features of the image blocks to be predicted in the blue and red channels of the second color channel are to be completed, completion prediction needs to be performed based on the semantic features of each image block in the second color channel, as well as the semantic features already existing in the first and third color channels. The image blocks already existing in the first and third color channels are the target image blocks.

[0087] In this embodiment of the disclosure, the process of obtaining semantic features through completion prediction is explained using the blue channel as an example. If the semantic features of the image blocks to be predicted in the red and green channels of the third color channel need to be completed, completion prediction needs to be performed based on the semantic features of each image block in the third color channel, as well as the semantic features already existing in the second and first color channels. The image blocks already existing in the second and first color channels are the target image blocks.

[0088] In the embodiments disclosed above, each of the three primary color channels needs to complete the semantic features of the image block to be predicted in its respective color channel.

[0089] In step S32, existing image blocks with complementary positions in the three primary color channels and the image block to be predicted are fused to obtain the target image.

[0090] In this embodiment of the disclosure, since each color channel in the three primary color channels is complementary in position to both the image block and the image block to be predicted, the existing image blocks and the image block to be predicted that are complementary in position to the three primary color channels are fused to obtain the fused color channels. The fused color channels can be understood as existing image blocks including each color channel. They then learn from each other to determine the pixel prediction value of each image so that the pixel prediction value can be filled into the image block. The image block is then reconstructed based on the position encoding of the image block to obtain the target image.

[0091] In this embodiment of the disclosure, the details and color richness of the target image are improved by fusing semantic features with the detail reconstruction in the underlying vision, and the computational load is reduced by using an image processing method based on complementary information.

[0092] In this embodiment, N×N pixels of the same color are arranged adjacently in the color filter array, where N can be 64, thus obtaining a densely distributed monochrome filter array, which is helpful for the design and production of the filter. The image blocks obtained by dividing the image into 64×64 step sizes reduce the amount of computation.

[0093] In the image processing of this disclosure, the MAE algorithm can be used to train a training model to obtain a target model, which is then applied to an electronic device. The target model may include an encoder and a decoder. During model training, MSE can be used as a reconstruction loss optimization model to obtain the target model, enabling the output of an RGB full-channel image, i.e., the target image. Furthermore, in this disclosure, the model's input is the original image processed by an N×N color filter array of aggregated colors, and the output is the target image including all RGB channels.

[0094] like Figure 6 As shown, Figure 6This is a schematic diagram illustrating an image processing method according to some embodiments of the present disclosure. N can be 64, and based on the raw data processing method of MAE, it tightly integrates hardware design with advanced algorithms, proposing an N×N color filter array arrangement. By leveraging the advantages of the MAE algorithm, through hardware and software complementarity and deep fusion, it can effectively simplify the traditional complex image processing process, realizing the conversion from the raw image to an RGB full-channel image, resulting in a new imaging link. Furthermore, by utilizing the asymmetric design, it fully utilizes the scene and color information sampled by the hardware, outputting a detailed and color-rich RGB full-channel image, improving image quality based on feature complementarity and fusion. In addition, the MAE asymmetric encoder can correspond to the sampling positions of missing color filters in each color channel, operating only on the unmasked parts, thereby reducing the consumption of computing resources, lowering the requirements for the computing power of the edge device, facilitating the edge deployment of the model, and forming an end-to-end output RGB full-channel image.

[0095] In this embodiment of the disclosure, combined with Figure 7 The image processing methods are explained. Figure 7 This is a flowchart illustrating an image processing method according to some embodiments of the present disclosure. It should be noted that pixels with horizontal lines can be considered red pixels, pixels with vertical lines can be considered green pixels, and pixels with diagonal lines can be considered blue pixels. For example... Figure 7 As shown, Figure 7 The image on the left is the original image output by a color filter array with N×N adjacent pixels of the same color. The pixels with diagonal lines can be considered blue pixels. The RGB color paths in the original image are separated to obtain red, green, and blue channels. Each of the red, green, and blue channels is then divided according to a preset step size, resulting in multiple image blocks for each color channel. For example, dividing multiple pixels in the red channel according to a preset step size yields multiple image blocks corresponding to the red channel. The preset step size can be 64×64.

[0096] like Figure 7 As shown, Figure 7 The input to the encoder is multiple image blocks corresponding to each color channel. It can be understood that the encoder input has three columns: the left column can be multiple image blocks corresponding to the red channel, the middle column can be multiple image blocks corresponding to the green channel, and the right column can be multiple image blocks corresponding to the blue channel.

[0097] In this embodiment, the encoder can employ a Vision Transformer (ViT) structure to extract deep semantic features from image patches through ViT encoding, providing guidance for the extension of blank areas in the input feature image. It is understood that since the blue and green channels are missing in the red channel, the red channel is a visible feature, while the green and blue channels are masked parts that need to be predicted. For the green channel, the red and blue channels are masked parts that need to be predicted, and vice versa. Image redundancy can be eliminated through a high proportion of masked areas. Multiple image patches for each color channel are input into the encoder, and the encoder outputs the semantic features of multiple image patches for each color channel. It should be noted that the encoder operates on a small subset of the complete set without masking, and the masked image patches are removed to enable training a larger encoder with a smaller amount of computation and memory. Figure 7 As shown, the three columns output by the encoder are: the left column contains the semantic features corresponding to each image block in the red channel, the middle column contains the semantic features corresponding to each image block in the green channel, and the right column contains the semantic features corresponding to each image block in the blue channel.

[0098] In this embodiment of the disclosure, the semantic features of each color output by the decoder are incomplete; for example, the green and blue channels are missing from the red channel. Therefore, for each color channel, they can be aligned and rearranged, such as... Figure 7 As shown, the input portion of the fusion decoder has three columns. The left column contains the data rearranged in red alignment. The horizontally lined squares represent the semantic features of each image patch in the red channel output by the encoder, while the remaining gray portion represents the image patches to be predicted in the green and blue channels. The position of the predicted image patches can be indicated by filling in random vectors. The middle column contains the green channel portion. The vertically lined squares represent the semantic features of each image patch in the green channel, while the remaining gray portion represents the image patches to be predicted that lack the red and blue channels. The position of the predicted image patches can be indicated by filling in random vectors. The right column contains the blue channel portion. The diagonally lined squares represent the semantic features of each image patch in the blue channel, while the remaining gray portion represents the image patches to be predicted that lack the red and green channels. The position of the predicted image patches can be indicated by filling in random vectors. The left column can be the first color channel in the above embodiment, the middle column can be the second color channel in the above embodiment, and the right column can be the third color channel in the above embodiment.

[0099] based on Figure 7As can be seen, the fusion decoder currently has 3 color channels. The feature information and colors corresponding to the 3 color channels are complementary. Therefore, the 3 color channels can be fused to obtain the fused color channel, and the fused color channel is input into the fusion decoder for prediction output.

[0100] In this embodiment, to save on model computation and edge deployment, a lightweight decoder structure can be used in the fusion decoder, such as a 6-layer Transformer structure. A full-color RGB image is output by reconstructing the input features. It should be noted that a convolutional layer can be added to the fusion decoder to fuse the three reconstructed color channels, making them compatible with the decoder's input.

[0101] In this embodiment of the disclosure, the above steps can be applied to real-time image processing or to model training. If the above process is applied to model training, the MSE can be used as the reconstruction loss to optimize the model.

[0102] In this embodiment, based on the MAE algorithm principle, the color filter array is redesigned and deeply integrated with the MAE algorithm for information complementarity, forming an image processing flow that integrates N×N and the MAE algorithm. N can be 64; a 64×64 clustered color filter array design with adjacent colors improves the yield rate in manufacturing processes. The MAE-based processing flow employs an asymmetric encoding and decoding method, optimizing computation and facilitating edge deployment. Furthermore, by fusing semantic features with detail reconstruction from the underlying vision, it helps obtain target images rich in detail and color.

[0103] Based on the same concept, embodiments of this disclosure also provide an image processing apparatus.

[0104] It is understood that the image processing apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0105] Figure 8 This is a block diagram of an image processing apparatus 100 according to some embodiments of the present disclosure. (Refer to...) Figure 8 The device includes an acquisition unit 101, a processing unit 102, and a determination unit 103.

[0106] The acquisition unit 101 is used to acquire the original image output by the color filter array, in which N×N pixels of the same color are arranged adjacently.

[0107] The processing unit 102 is used to separate each color channel in the three primary color channels of the original image, and divide each separated color channel into multiple image blocks in N×N units; extract the semantic features of each image block for each color channel, and align each color channel in the three primary color channels.

[0108] The determining unit 103 is used to, for each color channel obtained by alignment, supplement the semantic features of the image blocks in other color channels based on the semantic features of the image blocks in their respective color channels, so as to obtain the target image.

[0109] In one embodiment, the processing unit 102 extracts the semantic features of each image block in the following manner: performing linear mapping on each image block to obtain a mapping vector; adding positional codes to the mapping vectors, where the positional codes represent the pixel positions of the image blocks in the original image, and each image block corresponds to a positional code; performing mask coding on the mapping vectors based on the positional codes, wherein the mapping vectors with added positional codes are the vector regions that need to be encoded, and the mapping vectors without added positional codes are the vector regions that are removed by the mask; and extracting the semantic features of the mask-coded mapping vectors to obtain the semantic features of each image block.

[0110] In one embodiment, the processing unit 102 aligns each color channel in the three primary color channels in the following manner: for each color channel, an image block to be predicted is added at a position different from the existing image block, so as to align each color channel in the three primary color channels; wherein, the positions of the existing image block and the image block to be predicted in each color channel of the aligned three primary color channels are complementary.

[0111] In one embodiment, the determining unit 103 completes the semantic features of image blocks in other color channels based on the semantic features of image blocks in their respective color channels to obtain the target image: For each target color channel, based on the semantic features of target image blocks in other color channels different from the target color channel, the semantic features of the image block to be predicted are predicted, and the predicted semantic features are completed for the image block to be predicted. The target image block is an existing image block in other color channels that is complementary in position to the image block to be predicted, and the target color channel is each of the three primary color channels; the existing image blocks that are complementary in position in the three primary color channels and the image block to be predicted are fused to obtain the target image.

[0112] In one implementation, N is 64.

[0113] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0114] Figure 9 This is a block diagram illustrating an apparatus 200 for image processing according to some embodiments of the present disclosure. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0115] Reference Figure 9 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0116] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0117] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0118] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0119] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0120] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0121] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0122] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0123] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 216 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0124] In some embodiments of this disclosure, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0125] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by the processor 220 of the device 200 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0126] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0127] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the methods described above.

[0128] Figure 10 This is a block diagram illustrating an apparatus 300 for image processing according to some embodiments disclosed in a publication. For example, apparatus 300 may be provided as a server. See also... Figure 10 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the methods described above.

[0129] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0130] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0131] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0132] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the methods described above.

[0133] In some embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the image processing methods involved in any of the above embodiments.

[0134] In some embodiments of this disclosure, the processor that executes the computer program may be deployed in an electronic device, for example.

[0135] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0136] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0137] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0138] In this description, "multiple" means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more."

[0139] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0140] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0141] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0142] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if it is not structurally equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0144] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the original image output by the color filter array, wherein N×N pixels of the same color are arranged adjacently in the color filter array; Separate each color channel from the three primary color channels of the original image, and divide each separated color channel into multiple image blocks in units of N×N; Semantic features of each image patch are extracted for each color channel, and each color channel in the three primary color channels is aligned. For each color channel obtained from alignment, the semantic features of image patches in other color channels are supplemented based on the semantic features of image patches in their respective color channels to obtain the target image.

2. The method according to claim 1, characterized in that, The extraction of semantic features from each image patch includes: Linear mapping is performed on each image block to obtain a mapping vector; A positional code is added to the mapping vector, the positional code representing the pixel position of the image block in the original image, and each image block corresponds to a positional code; Based on the position encoding, the mapping vector is mask encoded, wherein the mapping vector with added position encoding is the vector region that needs to be encoded, and the mapping vector without added position encoding is the vector region that is removed by mask. The semantic features of the mask-encoded mapping vector are extracted to obtain the semantic features of each image block.

3. The method according to claim 1 or 2, characterized in that, Aligning each of the three primary color channels includes: For each color channel, a new image block to be predicted is added at a position different from the existing image block to align each color channel in the three primary color channels. In this context, the positions of the image blocks and the image blocks to be predicted already exist in each color channel of the aligned three primary color channels.

4. The method according to claim 3, characterized in that, The step of completing the semantic features of image patches in other color channels based on the semantic features of image patches in their respective color channels to obtain the target image includes: For each target color channel, based on the semantic features of the target image block in other color channels that are different from the target color channel, the semantic features of the image block to be predicted are predicted, and the predicted semantic features are completed for the image block to be predicted. The target image block is an existing image block in other color channels that is complementary to the image block to be predicted in position, and the target color channel is each of the three primary color channels. The target image is obtained by fusing the existing image blocks and the image blocks to be predicted that are complementary in position in the three primary color channels.

5. The method according to claim 1, characterized in that, The value of N is 64.

6. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the original image output by the color filter array, wherein N×N pixels of the same color are arranged adjacently in the color filter array; The processing unit is used to separate each color channel in the three primary color channels of the original image, and divide each separated color channel into multiple image blocks in units of N×N; extract the semantic features of each image block for each color channel, and align each color channel in the three primary color channels; The determining unit is used to, for each aligned color channel, supplement the semantic features of image blocks in other color channels based on the semantic features of image blocks in their respective color channels, so as to obtain the target image.

7. The apparatus according to claim 6, characterized in that, The processing unit extracts the semantic features of each image patch in the following manner: Linear mapping is performed on each image block to obtain a mapping vector; A positional code is added to the mapping vector, the positional code representing the pixel position of the image block in the original image, and each image block corresponds to a positional code; Based on the position encoding, the mapping vector is mask encoded, wherein the mapping vector with added position encoding is the vector region that needs to be encoded, and the mapping vector without added position encoding is the vector region that is removed by mask. The semantic features of the mask-encoded mapping vector are extracted to obtain the semantic features of each image block.

8. The apparatus according to claim 6 or 7, characterized in that, The processing unit aligns each color channel in the three primary color channels in the following manner: For each color channel, a new image block to be predicted is added at a position different from the existing image block to align each color channel in the three primary color channels. In this context, the positions of the image blocks and the image blocks to be predicted already exist in each color channel of the aligned three primary color channels.

9. The apparatus according to claim 8, characterized in that, The determining unit uses the semantic features of image patches in its respective color channels to complete the semantic features of image patches in other color channels, thereby obtaining the target image: For each target color channel, based on the semantic features of the target image block in other color channels that are different from the target color channel, the semantic features of the image block to be predicted are predicted, and the predicted semantic features are completed for the image block to be predicted. The target image block is an existing image block in other color channels that is complementary to the image block to be predicted in position, and the target color channel is each of the three primary color channels. The target image is obtained by fusing the existing image blocks and the image blocks to be predicted that are complementary in position in the three primary color channels.

10. The apparatus according to claim 6, characterized in that, The value of N is 64.

11. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the image processing method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a processor of an electronic device, enable the processor to perform the image processing method according to any one of claims 1 to 5.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1 to 5.

Images