Image processing method, device and computer equipment

Abstract

This disclosure provides an image processing method, apparatus, and computer device, relating to the field of artificial intelligence technology, and particularly to the field of image processing. The implementation scheme is as follows: acquiring a Bayer image; generating a location encoding vector corresponding to the spatial location of the Bayer image; fusing the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data; performing spatial context analysis on the fused feature data to generate context feature data; using a pre-trained neural network to determine multiple routing weights corresponding to the Bayer image's optimization strategy for multiple image features; using multiple preset lookup tables to map the Bayer image to generate multiple candidate images, wherein the resolution of the multiple candidate images is higher than that of the Bayer image; and performing weighted fusion of the multiple candidate images based on the multiple routing weights to generate a target image.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of image processing technology, and especially to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In modern digital imaging systems, most image sensors use color filter arrays (CFAs) to capture image information, with the Bayer array mode being the most common. In Bayer mode, each pixel captures only one of the three primary colors: red (R), green (G), and blue (B). To obtain a full-color image that is visible to the human eye, a series of complex image signal processing steps are usually required, including demosaicing, noise reduction, and super-resolution.

[0003] With the widespread adoption of mobile devices and embedded vision systems, users have increasingly higher demands for image quality, especially in low-light conditions and for improved image resolution. Existing image processing solutions typically utilize lookup tables or deep learning networks to enhance images in order to achieve better image quality with limited computing resources. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of this disclosure, an image processing method is provided, comprising: acquiring a Bayer image; generating a location encoding vector corresponding to the spatial location of the Bayer image; fusing the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data; performing spatial context analysis on the fused feature data to generate context feature data; using a pre-trained neural network to determine multiple routing weights corresponding to multiple image feature optimization strategies for the Bayer image based on the context feature data; using each of a preset multiple lookup tables to perform mapping processing on the Bayer image to generate multiple candidate images, wherein the multiple lookup tables correspond to the multiple image feature optimization strategies respectively, and wherein the resolution of the multiple candidate images is higher than that of the Bayer image; and performing weighted fusion on the multiple candidate images based on the multiple routing weights to generate a target image.

[0006] In some embodiments, generating a position encoding vector corresponding to the spatial location of the Bayer image includes: determining the phase information of each pixel in the Bayer image based on the color filter array arrangement pattern of the Bayer image; performing one-hot encoding on the phase information of each pixel to generate a position encoding vector corresponding to the spatial location of the Bayer image, wherein the dimension of the position encoding vector is equal to the number of pixel phases contained in the smallest repeating unit of the color filter array arrangement pattern of the Bayer image.

[0007] In some embodiments, fusing the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data includes: concatenating the location encoding vector with the corresponding pixel values ​​of the Bayer image along the channel dimension to generate the multi-channel fused feature data.

[0008] In some embodiments, the pixel values ​​of the Bayer image are single-channel, and the dimensions of the fused feature data are H x W x C, where H and W represent the height and width of the Bayer image, respectively, and C represents the total number of channels including pixel value channels and position encoding vector channels.

[0009] In some embodiments, performing spatial context analysis on the fused feature data to generate context feature data includes: extracting the corresponding channel weights of different channels in the fused feature data using a channel attention module, wherein the dimension of the channel weights is 1 x 1 x C; extracting the corresponding spatial weights of different spatial locations in the fused feature data using a spatial attention module, wherein the dimension of the spatial weights is H x W x 1; performing channel multiplication between the channel weights and the fused feature data to generate channel feature data, wherein the dimension of the channel feature data is H x W x C; performing pointwise multiplication between the spatial weights and the fused feature data to generate spatial feature data, wherein the dimension of the spatial feature data is H x W x C; and concatenating the channel feature data and the spatial feature data to obtain the context feature data, wherein the dimension of the context feature data is H x W x 2C.

[0010] In some embodiments, determining the multiple routing weights corresponding to the Bayer image optimization strategies for multiple image features based on the context feature data using a pre-trained neural network includes: inputting the context feature data into the pre-trained neural network; obtaining an intermediate feature tensor of dimension H x W x (N x R²) output by the neural network, where N represents the number of the preset multiple lookup tables, and R represents the resolution magnification factor of the multiple candidate images relative to the Bayer image; performing a pixel reorganization operation on the intermediate feature tensor to rearrange the intermediate feature tensor into a routing weight map of dimension RH x RW x N, wherein each pixel position in the routing weight map contains N weight values ​​corresponding to N lookup tables.

[0011] In some embodiments, the plurality of image feature optimization strategies include at least one of the following: an edge optimization strategy that sharpens and enhances contrast for image edge regions; a texture optimization strategy that preserves high-frequency details for image texture regions; a smooth region optimization strategy that improves color consistency for smooth regions of the image; and a noise optimization strategy that suppresses noise for high-noise regions.

[0012] In some embodiments, the preset multiple lookup tables are pre-generated through the following operations: constructing multiple independent lightweight convolutional neural networks, wherein each lightweight convolutional neural network is configured for one of the multiple image feature optimization strategies; training the multiple lightweight convolutional neural networks using loss functions optimized for different strategies; and establishing corresponding multiple lookup tables based on the input-output mapping relationship of the trained multiple lightweight convolutional neural networks.

[0013] In some embodiments, the pre-trained neural network, the channel attention module, the spatial attention module, and the multiple lightweight convolutional neural networks are trained end-to-end as a whole.

[0014] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: an acquisition module configured to acquire a Bayer image; a generation module configured to generate a location encoding vector corresponding to a spatial location of the Bayer image; a first fusion module configured to fuse the location encoding vector with pixel values ​​of the Bayer image to generate fused feature data; an analysis module configured to perform spatial context analysis on the fused feature data to generate context feature data; a determination module configured to use a pre-trained neural network to determine multiple routing weights corresponding to multiple image feature optimization strategies for the Bayer image based on the context feature data; a mapping module configured to use each of a preset multiple lookup tables to perform mapping processing on the Bayer image to generate multiple candidate images, wherein the multiple lookup tables correspond to the multiple image feature optimization strategies, and wherein the resolution of the multiple candidate images is higher than that of the Bayer image; and a second fusion module configured to perform weighted fusion of the multiple candidate images based on the multiple routing weights to generate a target image.

[0015] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0016] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0018] According to one or more embodiments of this disclosure, an image processing method is provided that effectively solves the spatial irregularity problem in the Bayer domain by fusing location encoding with Bayer images, ensuring the accuracy of feature extraction. Simultaneously, by dynamically weighting and fusing the output of a multi-strategy lookup table using routing weights generated based on spatial context analysis, adaptive optimization for different image regions is achieved. This avoids the accumulation of errors in traditional step-by-step processing while balancing the quality of high-resolution reconstruction with computational efficiency through the lookup table mechanism.

[0019] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0020] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0021] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating a portion of the processing flow of an image processing method according to an exemplary embodiment.

[0023] Figure 3 This is a schematic diagram illustrating a portion of the processing flow of an image processing method according to an exemplary embodiment.

[0024] Figure 4 This is a schematic diagram illustrating the fusion mechanism of multiple lookup tables according to an exemplary embodiment.

[0025] Figure 5 This is a schematic block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0026] Figure 6 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0029] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0030] In related technologies, denoising, depigmentation, and super-resolution are typically performed sequentially as independent modules. This cascaded processing approach suffers from significant error accumulation. For example, if texture details are lost in the early stages of denoising, subsequent depigmentation and super-resolution steps will be unable to recover these details, and may even amplify artifacts. Furthermore, each module is usually optimized for only a single objective, resulting in a final output that is often a suboptimal solution.

[0031] In recent years, although methods have emerged that process directly in the Bayer domain, they face challenges related to spectral sparsity (each pixel has only one color channel) and spatial irregularity (color channels are arranged in a specific phase). Conventional convolution operations struggle to accurately model the spatial dependencies of Bayer patterns and are prone to producing mesh artifacts.

[0032] Furthermore, while LUT-based solutions are computationally efficient, existing LUT methods typically lack content awareness, and a single LUT cannot simultaneously address the noise reduction requirements of smooth regions and the detail preservation requirements of texture regions. On the other hand, simple multi-LUT parallel solutions, without intelligent fusion mechanisms, are also difficult to achieve pixel-level precision optimization.

[0033] To address this issue, embodiments of this disclosure propose an image processing method that effectively solves the spatial irregularity problem in the Bayer domain by fusing location encoding with Bayer images, ensuring the accuracy of feature extraction. Simultaneously, by dynamically weighting and fusing the output of a multi-strategy lookup table using routing weights generated based on spatial context analysis, adaptive optimization for different image regions is achieved. This avoids the accumulation of errors in traditional step-by-step processing while balancing the quality of high-resolution reconstruction with computational efficiency through the lookup table mechanism.

[0034] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0036] like Figure 1 As shown, the image processing method includes steps S101 to S107.

[0037] Step S101: Obtain the Bayer image.

[0038] The image processing method in this embodiment begins with step S101, acquiring the raw Bayer image as input data. This step typically involves directly reading a single-channel two-dimensional array from the image sensor. The data format depends on the sensor's color filter array (CFA) arrangement, such as the common RGGB mode. Acquiring raw data without traditional ISP processing preserves the most complete original optical signal information, laying the foundation for subsequent end-to-end lossless restoration and avoiding artifacts or information loss that may result from traditional steps such as de-mosaicing.

[0039] Step S102: Generate a location encoding vector corresponding to the spatial location of the Bayer image.

[0040] In step S102, the phase information of each pixel is determined based on the CFA arrangement pattern (such as 2×2 repeating unit) and encoded to generate a position encoding vector that can explicitly identify the pixel color attribute.

[0041] Step S103: Fuse the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data.

[0042] In step S103, the position encoding vector and the original Bayer pixel values ​​are concatenated and fused along the channel dimension to generate fused feature data. This process endows the data with position awareness, enabling subsequent processing to accurately distinguish physically adjacent pixels with drastically different color attributes, effectively preventing the grid effect or color shift produced by traditional convolution on Bayer images.

[0043] After obtaining location-aware feature data, step S104 is executed.

[0044] Step S104: Perform spatial context analysis on the fused feature data to generate context feature data.

[0045] Specifically, step S104 aims to extract contextual information reflecting the local environment or overall structure of the image from the fused data containing location information, providing sufficient feature basis for subsequent strategy selection and weight allocation. Utilizing spatial context analysis, the generated contextual feature data not only contains the original pixel information but also is rich in filtered and enhanced global and local environmental information. This is equivalent to providing high-quality judgment criteria for subsequent decisions, significantly enhancing the system's ability to understand image content.

[0046] Step S105: Using a pre-trained neural network, determine the multiple routing weights corresponding to the Bayer image optimization strategies for multiple image features based on the context feature data.

[0047] These weights are determined for Bayer images and correspond to multiple different image feature optimization strategies. This means that the neural network can intelligently determine how to allocate the weight ratios of different optimization strategies at the current processing position of the image based on the analysis of contextual information, thereby achieving adaptive perception of image content.

[0048] Step S106: The Bayer image is mapped using each of the preset lookup tables to generate multiple candidate images. The multiple lookup tables correspond to the multiple image feature optimization strategies, and the resolution of the multiple candidate images is higher than that of the Bayer image.

[0049] This mapping process generates multiple candidate images, all of which have a higher resolution than the original Bayer image. Step S106 utilizes a lookup table mechanism to simultaneously perform image feature transformation and resolution enhancement.

[0050] Step S107: Based on the multiple routing weights, the multiple candidate images are weighted and fused to generate the target image.

[0051] Step S107 performs weighted fusion on the multiple candidate images generated in step S106 based on the multiple routing weights determined in step S105. Through this weighted fusion operation, high-resolution candidate images corresponding to different optimization strategies can be synthesized into the final target image according to the weight ratios calculated by the neural network.

[0052] Therefore, the image processing method innovatively combines location-aware technology, the contextual analysis capabilities of deep learning, and the efficiency of lookup tables through the aforementioned steps. This method breaks through the limitations of traditional ISP's step-by-step processing, simultaneously completing denoising, de-mosaicing, and super-resolution within a single process, effectively avoiding error accumulation. Utilizing a lightweight routing network in conjunction with efficient LUT lookups significantly reduces computational requirements, making it adaptable to resource-constrained devices. Simultaneously, through pixel-level adaptive fusion, it significantly improves image restoration quality in complex scenes, achieving a perfect balance between high performance and low power consumption.

[0053] The key technical details of the image processing method will be further elaborated below with reference to specific implementation methods. Especially when processing Bayer images, due to the spatial irregularity of their pixel arrangement (i.e., adjacent pixels represent different color channels), accurately conveying the identity information of each pixel to the neural network is a prerequisite for achieving high-quality reconstruction. Therefore, the position encoding vector generated in step S102 plays a crucial role as a spatial anchor.

[0054] According to some embodiments, step S102 includes: determining the phase information of each pixel in the Bayer image based on the color filter array arrangement pattern of the Bayer image; performing one-hot encoding on the phase information of each pixel to generate a position encoding vector corresponding to the spatial position of the Bayer image, wherein the dimension of the position encoding vector is equal to the number of pixel phases contained in the smallest repeating unit of the color filter array arrangement pattern of the Bayer image.

[0055] Specifically, taking the most common RGGB Bayer arrangement in the industry as an example, its color filter array is arranged periodically with 2×2 pixel blocks as the smallest repeating unit. In this repeating unit, there are four different pixel phases, corresponding to the red (R), green (Gr), green (Gb), and blue (B) color channels, respectively.

[0056] During execution, the phase assignment of a pixel within a repeating unit is first determined based on its coordinate position (x, y). For example, the phase can be defined as:

[0057] Phase 0 (corresponding to pixel R): coordinates (0,0);

[0058] Phase 1 (corresponding to Gr pixel): coordinates (0,1);

[0059] Phase 2 (corresponding to Gb pixels): coordinates (1,0);

[0060] Phase 3 (corresponding to pixel B): coordinates (1,1).

[0061] Subsequently, one-hot encoding is used to convert the discrete integer phase identifiers into high-dimensional sparse vectors. Since the minimum repeating unit contains four phases, the generated position encoding vector is set to 4 dimensions. The specific encoding mapping relationship is as follows:

[0062] Phase 0 is mapped to the vector [1, 0, 0, 0];

[0063] Phase 1 is mapped to the vector [0, 1, 0, 0];

[0064] Phase 2 is mapped to the vector [0, 0, 1, 0];

[0065] Phase 3 is mapped to the vector [0, 0, 0, 1].

[0066] In this way, the spatial location and color attributes inherent in Bayer images are explicitly transformed into vector forms that are easily processed by computers. This process also applies to non-RGGB patterns (such as BGGR or other special CFA permutations), requiring only adjustments to the vector dimension and the corresponding phase encoding based on the actual smallest repeating unit size. This design ensures that the position-encoded vectors accurately reflect the physical characteristics of the Bayer image's microstructure.

[0067] After successfully generating the location-encoded vector that explicitly identifies the pixel color attribute through the above steps, the image processing method not only grasps the original light intensity information of the image but also the exact phase assignment of each pixel in the Bayer array. To enable the subsequent neural network model to utilize both types of information simultaneously—how bright a pixel is and what color it is—they need to be physically bound together in the data structure. This process is the feature fusion operation in step S103, which tightly combines discrete spatial prior knowledge with continuous image intensity signals to construct semantically rich input features.

[0068] According to some embodiments, step S103 includes: concatenating the location encoding vector with the pixel values ​​corresponding to the Bayer image in the channel dimension to generate the multi-channel fused feature data.

[0069] Specifically, the fusion operation is not a spatial patching of images, but rather an increase in the depth or thickness of the data. For any pixel in a Bayer image, the image processing method treats its original pixel grayscale value as data from channel 0, and its corresponding generated 4D positional encoding vector as data from channels 1 to 4. By concatenating along the channel dimensions, each pixel is no longer just a single numerical value, but is expanded into a vector containing rich information. For example, if a location is an R pixel with an intensity of 128, and its positional encoding is [1, 0, 0, 0], then the concatenated feature vector is [128, 1, 0, 0, 0]. This processing method ensures a one-to-one correspondence between pixel values ​​and their positional attributes, allowing the neural network's convolutional kernel to simultaneously capture the light intensity and phase type at that location during sliding window computation, thereby achieving position-aware feature extraction.

[0070] According to some embodiments, the pixel value of the Bayer image is a single channel, and the dimension of the fused feature data is H x W x C, where H and W represent the height and width of the Bayer image, respectively, and C represents the total number of channels including pixel value channels and position encoding vector channels.

[0071] In this embodiment, the input data is typically the original single-channel RAW data, with a tensor dimension of H x W x 1. Since the positional encoding vector is usually set to 4 dimensions based on the minimum repeating unit of the Bayer array (such as the 2×2 unit in the previous embodiment), the total number of channels C is equal to the sum of the original image channel number and the positional encoding dimension, i.e., C = 1 + 4 = 5. The final generated fused feature data is a three-dimensional tensor with dimensions H x W x 5. This 5-channel feature map serves as the input to the subsequent spatial context analysis module, where the first channel carries the image's content information, such as texture and edges, while the latter four channels constitute the image's structural skeleton (phase arrangement). Together, they support the subsequent complex end-to-end inference process.

[0072] After obtaining fused feature data containing rich location information and original pixel intensity through the aforementioned steps, the image processing workflow enters the core cognitive stage. While the feature data alone contains all the necessary information, it is still necessary to further identify which feature channels are more critical for the current restoration task, and which spatial regions in the image contain high-frequency details (such as edges or textures) that require focused processing. To achieve this refined feature perception, step S104 employs a dual-path parallel analysis strategy, introducing an attention mechanism to weight and filter features from both the channel and spatial dimensions. The specific process is as follows: Figure 2 As shown.

[0073] Figure 2 This is a flowchart illustrating a portion of the processing flow of an image processing method according to an exemplary embodiment.

[0074] like Figure 2 As shown, step S104 specifically includes steps S201 to S205.

[0075] Step S201: Use the channel attention module to extract the corresponding channel weights of different channels in the fused feature data, wherein the dimension of the channel weights is 1 x 1 x C.

[0076] In this step, the channel attention module focuses on analyzing the interdependencies of the input features along the channel dimension. Since the input fused feature data contains pixel value channels and multiple location-encoded channels, this module aims to learn the global importance of each channel for image reconstruction. After computation, the module outputs a 1 x 1 x C channel weight vector (denoted as Mc), where each value represents the response weight of the corresponding channel.

[0077] Step S202: Use the spatial attention module to extract the spatial weights corresponding to different spatial locations in the fused feature data, wherein the dimension of the spatial weights is H x W x 1.

[0078] In parallel or sequentially with step S201, the spatial attention module focuses on analyzing the distribution characteristics of the input features in the spatial dimension. This module aims to distinguish between information-rich regions (such as edges and corners) and flat regions in an image. After calculation, the module outputs a spatial weight matrix (denoted as Ms) of dimension H x W x 1, where each value represents the global importance of that pixel location.

[0079] Step S203: Perform channel multiplication on the channel weights and the fused feature data to generate channel feature data, wherein the dimension of the channel feature data is H x W x C.

[0080] In this step, the original fused feature data (denoted as F) is weighted using the channel weight vector Mc generated in step S201. Specifically, a channel multiplication operation is performed, which involves broadcasting the k-th scalar value in the weight vector and multiplying it by all pixels in the k-th channel of the input feature. This operation enhances specific feature channels that are beneficial to the recovery task (e.g., specific phase marker channels) while suppressing irrelevant or redundant channels, generating channel feature data (Mc x F) filtered by channel dimension.

[0081] Step S204: Perform point-by-point multiplication between the spatial weights and the fused feature data to generate spatial feature data, wherein the dimension of the spatial feature data is H x W x C.

[0082] In this step, the original fused feature data F is weighted using the spatial weight matrix Ms generated in step S202. Specifically, a pointwise multiplication operation is performed, that is, the weight value at coordinate (x, y) in the weight matrix is ​​applied to all channel data of the input feature at that coordinate. This operation can highlight high-frequency detail regions or key structural regions in the image, making the generated spatial feature data (Ms x F) more discriminative in spatial distribution.

[0083] Step S205: Concatenate the channel feature data and the spatial feature data to obtain the context feature data, wherein the dimension of the context feature data is H x W x 2C.

[0084] Finally, to comprehensively utilize the results of the two analyses, the channel feature data generated in step S203 and the spatial feature data generated in step S204 are concatenated along the channel dimension. The concatenated result is the context feature data, with its total number of channels expanded to 2C and its dimension becoming H x W x 2C.

[0085] Through steps S201 to S205 described above, this embodiment achieves deep feature reconstruction of the input data. Parallel processing via a dual-path attention mechanism simultaneously captures key information from both channel importance and spatial importance dimensions. The final concatenated contextual feature data not only preserves the integrity of the original signal but also significantly enhances the expressive power and semantic richness of the features. This provides a high-quality, highly discriminative input foundation for subsequent neural networks, enabling them to make more accurate routing decisions and achieve superior adaptive processing results when faced with complex and varied image content.

[0086] After obtaining contextual feature data rich in global and local semantic information, the image processing workflow enters the crucial decision-making stage. The core challenge here is how to accurately map the feature information in the low-resolution space (H x W) to the high-resolution space (RH x RW), and assign the most suitable lookup table mixing strategy to each final output pixel. To address this pixel-level prediction problem across resolutions, step S105 introduces a specially designed pre-trained neural network, namely the route weight generator. This network uses contextual feature data as input and, through a specific tensor transformation mechanism, achieves end-to-end mapping from low-dimensional features to a high-dimensional weight map.

[0087] According to some embodiments, step S105 includes: inputting the context feature data into the pre-trained neural network; obtaining an intermediate feature tensor of dimension H x W x (N x R²) output by the neural network, where N represents the number of the preset multiple lookup tables, and R represents the resolution magnification factor of the multiple candidate images relative to the Bayer image; performing a pixel reorganization operation on the intermediate feature tensor to rearrange the intermediate feature tensor into a routing weight graph of dimension RH xRW x N, wherein each pixel position in the routing weight graph contains N weight values ​​corresponding to N lookup tables.

[0088] Specifically, the pre-trained neural network in this step is typically configured as a lightweight convolutional neural network (CNN). This network receives contextual feature data of dimension H x W x 2C as input. Since this embodiment involves not only Bayer-to-RGB conversion but also super-resolution reconstruction (magnification factor R, e.g., R=2), the network needs to predict the weight distribution on a high-resolution grid. To achieve this on low-resolution feature maps, the network outputs a special intermediate feature tensor at the last convolutional layer, with dimensions set to H x W x (N x R²). Here, the channel dimension N x R² has a clear physical meaning: for each pixel location in the input image (H x W in total), the network predicts the weight information of its corresponding R² sub-pixels after magnification. Since each sub-pixel needs to be assigned mixed weights to N lookup tables, a total of N x R² values ​​need to be predicted for each input location. Subsequently, to restore these weights stacked in the channel dimension to their correct spatial locations, step S105 performs a pixel recombination operation. This operation periodically splits and rearranges the channel dimensions of the intermediate feature tensor, flattening the (N x R²) channel data onto the spatial dimension. After this transformation, the spatial resolution of the tensor expands from H x W to RH x RW, while the number of channels is correspondingly reduced to N. The final generated routing weight map (with dimensions RH x RW x N) has the same resolution as the target image, ensuring that subsequent weighted fusion operations can be performed precisely pixel-to-pixel, thereby achieving fine-grained control over high-frequency details such as edges and textures.

[0089] Through the pixel recombination operation described above, a routing weight map with the same resolution as the target image (RH x RW) and containing N channels has been obtained. However, the values ​​directly output at this point are usually the raw response values ​​calculated by the neural network. These values ​​may contain negative numbers, and their sum across the channel dimensions is not necessarily equal to 1. To ensure the correct physical meaning of the subsequent multi-lookup table fusion process, that is, to ensure that the sum of the contribution ratios of each lookup table is 100% to maintain the brightness and color balance of the output image, the routing weight map must be normalized. This embodiment provides two specific normalization implementation methods to meet different hardware deployment requirements and computational process designs.

[0090] According to some embodiments, normalization is performed as an internal layer of the pre-trained neural network. Specifically, the pre-trained neural network, i.e., the route weight generator, includes a Softmax activation layer after performing pixel recombination. In this embodiment, the neural network directly outputs the normalized probability distribution during the inference phase. Softmax computation is performed on the rearranged route weight map along the channel dimension (i.e., the dimension corresponding to N lookup tables). For each pixel position (x, y) in the weight map, the Softmax function maps N original input values ​​to N output values, such that these N output values ​​are all between 0 and 1, and their sum is strictly equal to 1. The advantage of this approach is that the final output of the neural network is a mixing ratio that can be directly used for weighted calculation, simplifying the subsequent signal processing flow and making it suitable for deployment on neural network processors that support Softmax operator acceleration.

[0091] In some embodiments, normalization is not included in the output layer of the neural network, but is performed as pre-calculation logic for subsequent weighted fusion steps. In this implementation, the pre-trained neural network is only responsible for outputting the raw feature response values, where the values ​​in the output routing weight graph represent the relative confidence scores of each lookup table, rather than absolute probabilities. Before performing weighted fusion in step S107, the image processing pipeline reads these raw scores and performs normalization in real time using the processor's mathematical computation unit by calculating Softmax or a simple division by sum operation. The advantage of this approach is that it provides greater flexibility, allowing algorithm engineers to fine-tune the final image fusion style by adjusting post-processing parameters without retraining the network.

[0092] To more intuitively illustrate the complete data flow from feature input to the final generation of normalized route weights, and especially to clearly demonstrate the structural relationship between the dual-path attention mechanism and the route weight generator, Figure 3 A detailed modular processing flow diagram is provided. This diagram is a system-level summary of the spatial context analysis in step S104 and the routing weight generation in step S105. It specifically shows the dimensional changes of data tensors at each layer of the network and the key computational logic.

[0093] Figure 3 This is a schematic diagram illustrating spatial context analysis and route weight generation in an image processing method according to an exemplary embodiment.

[0094] like Figure 3As shown, the processing flow begins with the fusion input Features at the top. This input is the fusion feature data generated in the previous steps, with its tensor dimensions labeled H x W x C, where H and W are the spatial dimensions of the input Bayer image, and C is the total number of channels containing pixel intensity and location encoding. The data flow branches here, entering two independent attention analysis branches in parallel:

[0095] like Figure 3 As shown, the left branch is the channel attention path. The fused input first enters the channel attention module. This module compresses the spatial dimension through operations such as global average pooling, outputting a channel weight Mc with a dimension of 1 x 1 C. This weight vector represents the global response intensity of different feature channels. Subsequently, in the element-wise multiplication module, a broadcast multiplication operation is performed along the channel dimension, applying the weight Mc to the original Features to generate channel-enhanced feature data (Mc × Features).

[0096] like Figure 3 As shown, the right branch is the spatial attention path. The fused input simultaneously enters the spatial attention module. This module outputs a spatial weight Ms with dimensions H x W x 1 through operations such as channel compression. This weight matrix represents the information importance of different pixel positions in the image. Subsequently, in the element-wise multiplication module, pointwise multiplication operations in the spatial dimension are performed, applying the weight Ms to all channels of the original Features, generating spatially enhanced feature data (Ms × Features). The outputs of the above two branches converge in the feature concatenation module. The two sets of feature data are stacked in the channel dimension to form a context feature tensor with dimensions expanded to H x W x 2C. The concatenated data is then input into the routing weight generator, i.e., the aforementioned pre-trained neural network. Figure 3 As shown, the generator integrates convolutional layers and Softmax logic. It first increases the feature space resolution from H x W to 2H x 2W (where R is 2 in this example) through convolutional operations and an implicit pixel recombination layer, and adjusts the number of channels to the number of lookup tables N. Finally, a Softmax layer normalizes the channel dimensions. The final output, pixel-level routing weights at the bottom, is explicitly labeled as 2H x 2W x N, representing the normalized multi-lookup table hybrid weights for each pixel in the target high-resolution image.

[0097] After generating pixel-level, normalized routing weights using the aforementioned neural network, the image processing method has essentially determined whether each subtle region in the image belongs to a contour edge, fine texture, flat background, or high-noise region, and quantified the degree to which that region requires different restoration strategies. The next step involves invoking a pre-defined set of lookup tables to perform targeted restoration and reconstruction of the original Bayer image. These lookup tables are not general mappings, but rather each focuses on solving a specific type of image quality problem.

[0098] According to some embodiments, the plurality of image feature optimization strategies include at least one of the following: an edge optimization strategy that sharpens and enhances contrast for image edge regions; a texture optimization strategy that preserves high-frequency details for image texture regions; a smooth region optimization strategy that improves color consistency for smooth regions of the image; and a noise optimization strategy that suppresses noise for high-noise regions.

[0099] Specifically, in order to address the complex challenges in Bayer domain image restoration, this embodiment designs four optimization strategies with different dimensions and solidifies them into independent lookup tables.

[0100] Edge optimization strategies focus on reconstructing structural information in images. Due to the sparsity of spatial sampling in Bayer images, edge regions are prone to jagged edges or blurring. During training, edge optimization strategies emphasize identifying edge features in horizontal, vertical, and 45-degree diagonal directions. By enhancing local contrast and sharpening edge transitions, they ensure that object outlines remain clear and sharp after super-resolution magnification, eliminating common interpolation blurring phenomena.

[0101] Texture optimization strategies focus on restoring high-frequency information. For areas with complex textures such as hair, grass, and fabric, conventional noise reduction often results in a loss of detail (i.e., a "smearing" effect). Texture optimization strategies aim to suppress noise while preserving the high-frequency variation trends between pixels to the greatest extent possible, restoring rich texture details and making the image look more realistic and delicate.

[0102] The smoothing region optimization strategy targets areas with gentle color changes, such as the sky, walls, and skin tones. In these areas, the human eye is highly sensitive to color banding and artifacts. This strategy focuses on improving color consistency and smoothness, eliminating false colors or mesh effects that may result from improper demosaicing algorithms, and ensuring visual purity in large areas of solid color.

[0103] The noise optimization strategy is specifically designed to address high noise issues in low-light or high-sensitivity scenes. Strong noise reduction directly in the Bayer domain is more effective than in the RGB domain. This strategy employs more robust mapping rules, aiming to significantly suppress random and fixed-pattern noise while minimizing damage to the main image structure, providing a clean background.

[0104] By running the above strategies in parallel and combining them with the aforementioned routing weights for dynamic fusion, it can be ensured that each part of the image receives the most suitable processing: sharp edges, fine textures, and clean backgrounds, thereby achieving the optimal solution for global image quality.

[0105] It should be noted that the four image feature optimization strategies (edge, texture, smoothing, and noise) listed above are merely typical examples of this disclosure in specific implementation scenarios, and not an exhaustive or limiting representation of the technical solutions of this invention. In practical applications, those skilled in the art can flexibly adjust, reduce, or expand the type and number of image feature optimization strategies according to the specific characteristics of the imaging device, the requirements of the application scenario, or the limitations of hardware computing power. For example, in addition to the dimensions mentioned above, lookup tables for specific color styles or tone mapping lookup tables for wide dynamic range scenes can be expanded. Furthermore, the number of lookup tables N can be adjusted to more or less according to actual needs. Any scheme that uses parallel mapping of multiple lookup tables and combines dynamic weights for fusion should be understood as falling within the protection scope of this invention.

[0106] To more intuitively demonstrate how the aforementioned multi-strategy lookup tables work together in real-world data flows, particularly how they map input data in parallel and combine it with routing weights to generate the final result, Figure 4 A detailed schematic diagram of the processing mechanism is provided. The diagram clearly shows the data flow from the input pixel block to the final adaptive output, as well as the parallel relationship of the lookup tables for each specific function in the system architecture.

[0107] Figure 4 This is a schematic diagram illustrating the fusion mechanism of multiple lookup tables according to an exemplary embodiment.

[0108] like Figure 4 As shown, the processing flow of this fusion mechanism begins with the input pixel block at the top. This input pixel block is typically local data of a Bayer image carrying location-encoded information. The data then enters the lookup table set module, where it is branched into four parallel processing branches.

[0109] These four branches correspond to the four optimization strategies mentioned above, specifically including:

[0110] LUT1: Edge optimization, used to process image contours and structure;

[0111] LUT2: Texture optimization used to restore high-frequency details;

[0112] LUT3: Smooth area optimization, used to maintain color consistency;

[0113] LUT4: High-noise region optimization, used to suppress noise.

[0114] Each branch independently maps the input data to generate corresponding intermediate candidate results. Meanwhile, the routing weights of the left input (generated by the preceding neural network) are introduced into the dynamic fusion module.

[0115] In the dynamic fusion module, routing weights are used as coefficients to weight and combine candidate results from the four LUT branches. This process achieves pixel-level policy mixing. Ultimately, the fused data forms the bottom adaptive output, which is a high-quality target image that has undergone targeted enhancement and reconstruction.

[0116] The reason why the above-mentioned multi-lookup table fusion mechanism can operate efficiently and with high quality is that each underlying lookup table (LUT) is not a simple linear interpolation or general mapping, but rather carries a specific image restoration capability.

[0117] According to some embodiments, the preset multiple lookup tables are pre-generated through the following operations: constructing multiple independent lightweight convolutional neural networks, wherein each lightweight convolutional neural network is configured for one of the multiple image feature optimization strategies; training the multiple lightweight convolutional neural networks using loss functions optimized for different strategies; and establishing corresponding multiple lookup tables based on the input-output mapping relationship of the trained multiple lightweight convolutional neural networks.

[0118] For example, during the training phase, instead of using a general loss function, these networks are trained separately using differentiated loss functions optimized for different strategies. For instance, a network optimized for edges might focus more on gradient fitting, while a network optimized for textures might focus more on preserving perceptual features. Once these networks have converged, the complex floating-point convolution operations are solidified into multiple corresponding lookup tables based on their input-output mapping, thereby reducing the computational complexity during inference from cumbersome convolution operations to constant-time table lookup operations.

[0119] To ensure that the entire image processing workflow, from feature extraction to routing decisions and finally expert fusion, can work together to achieve optimal performance, this embodiment adopts a sophisticated end-to-end joint training architecture.

[0120] According to some embodiments, the pre-trained neural network, the channel attention module, the spatial attention module, and multiple lightweight convolutional neural networks are jointly trained end-to-end as a whole.

[0121] Specifically, while each lightweight CNN can be initialized independently, they need to be chained together for joint optimization to ensure accurate scheduling of the lightweight CNNs by the attention module and routing network when building the entire image processing pipeline. During training, the lookup table is temporarily replaced by a differentiable CNN model. The training data, after positional encoding fusion, flows through the attention module and routing weight generator, and also flows in parallel through all expert networks. The routing weights are used as coefficients to perform a weighted summation of the expert network outputs. The calculated total error gradient is not only used to fine-tune the expert networks, but more importantly, it is fed back to the routing weight generator and attention module, teaching them how to allocate the optimal weight combination when facing different image features.

[0122] It should be noted that the neural network model training involved in this embodiment can be performed based on a general deep learning framework and high-performance computing hardware. The following uses an exemplary training process as an example to explain in detail the construction of training data, the phased training strategy, and the model deployment and conversion process.

[0123] The training process typically begins with a rigorous data preparation phase. First, a high-quality RGB image dataset containing rich scenes (such as natural landscapes, urban architecture, portraits, and night scenes) needs to be constructed or selected as the ground truth, such as DIV2K, Flickr2K, or a proprietary high-resolution image library. Then, to prevent overfitting and improve the model's generalization ability, the ground truth images are usually augmented, including random rotation, flipping, scaling, or cropping. Next, corresponding training input data is generated by simulating a real imaging process: high-resolution RGB images are downsampled to match the target low-resolution input, and resampled according to a specific color filter array pattern (such as RGGB) to generate single-channel Bayer images. Furthermore, to improve the model's robustness in real low-light or high-sensitivity scenes, Gaussian or Poisson noise of varying intensities is superimposed on the Bayer images to simulate the performance of real sensors under different signal-to-noise ratio environments.

[0124] The training process is mainly divided into two stages: expert network pre-training and system end-to-end joint training.

[0125] In the first stage, the aim is to train N lightweight convolutional neural networks with independent structures or partially shared parameters, targeting N different image feature optimization strategies (such as edges, textures, smoothing, and noise). During training, differentiated objective functions are designed for different task characteristics: for example, for edge optimization networks, gradient loss or L1 loss can be combined to force the network to focus on the accurate reconstruction of high-frequency contours; for texture optimization networks, perceptual loss or adversarial loss can be introduced to preserve subtle texture details and avoid over-smoothing; for smoothing or denoising networks, total variation loss can be used to suppress color artifacts and random noise. Through independent training in this stage, it is ensured that each network achieves optimal performance in its specific domain of expertise.

[0126] In the second stage, the pre-trained expert networks are integrated into the complete image processing framework, forming a network with the channel attention module, spatial attention module, and routing weight generator for end-to-end joint fine-tuning. During this stage, to ensure effective gradient propagation, the expert networks are temporarily maintained as differentiable convolutional neural networks, rather than static lookup tables. Input data flows through the attention module and routing network to generate pixel-level weights. These weights are used as coefficients to weight and fuse the outputs of each expert network to obtain the final predicted image. The mixing loss between the predicted and ground truth images is calculated, and the parameters of all modules are updated synchronously through backpropagation. The key to this process is forcing the routing network to learn the intrinsic correspondence between local image features and the processing capabilities of the expert networks, enabling it to intelligently determine which expert should be called and the proportion of that expert to be called under what image features.

[0127] After training, the model conversion and deployment phase begins. To achieve extreme inference speed, a traversal algorithm is used to scan the receptive field space of each lightweight expert network after training convergence, solidifying the convolutional mapping relationship between its input and output and converting it into a static lookup table (LUT) file. In practical application deployment, the original cumbersome convolutional computation is replaced by efficient table lookup operations, retaining only the lightweight routing and attention structures for real-time calculation of index weights, thus achieving a perfect balance between high-performance image quality and low computational consumption on mobile or embedded devices.

[0128] In summary, the image processing method proposed in this disclosure innovatively integrates position-aware technology, deep context analysis, and a multi-strategy lookup table mechanism to construct an efficient and robust end-to-end Bayer image restoration framework. First, it explicitly solves the spatial irregularity problem of Bayer domain data using position encoding, ensuring the accuracy of feature extraction. Second, it breaks down the barriers between denoising, de-mosaicing, and super-resolution processing in traditional ISP pipelines, effectively avoiding the accumulation of cascading errors and signal loss. More importantly, by combining pixel-level routing weights generated by lightweight neural networks with efficient parallel queries of multiple lookup tables, the image processing method significantly reduces computational resource consumption and inference latency while achieving adaptive and refined restoration of image edges, textures, smoothing, and high-noise regions. This achieves a perfect balance between high performance and low power consumption on resource-constrained devices such as mobile terminals.

[0129] Embodiments of this disclosure also provide an image processing apparatus. Figure 5 This is a schematic block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0130] like Figure 5 As shown, the image processing apparatus 500 includes: an acquisition module 501 configured to acquire a Bayer image; a generation module 502 configured to generate a location encoding vector corresponding to the spatial location of the Bayer image; a first fusion module 503 configured to fuse the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data; an analysis module 504 configured to perform spatial context analysis on the fused feature data to generate context feature data; a determination module 505 configured to use a pre-trained neural network to determine multiple routing weights corresponding to multiple image feature optimization strategies for the Bayer image based on the context feature data; a mapping module 506 configured to use each of a preset multiple lookup tables to perform mapping processing on the Bayer image to generate multiple candidate images, wherein the multiple lookup tables correspond to the multiple image feature optimization strategies, and wherein the resolution of the multiple candidate images is higher than that of the Bayer image; and a second fusion module 507 configured to perform weighted fusion on the multiple candidate images based on the multiple routing weights to generate a target image.

[0131] The operation of modules 501-507 of the image processing apparatus 500 is similar to the operation of steps S101-S107 described above, and will not be repeated here.

[0132] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0133] It should also be understood that the above regarding Figure 5 The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0134] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0135] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0136] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0137] In the following text, combined with Figure 6 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0138] Figure 6 An example configuration of a computer device 600 that can be used to implement the methods described herein is shown.

[0139] Computer device 600 can be a variety of different types of devices. Examples of computer device 600 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0140] Computer device 600 may include at least one processor 602, memory 604, multiple communication interfaces 606, display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612 capable of communicating with each other, such as via system bus 614 or other suitable connections.

[0141] Processor 602 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 602 may be configured to acquire and execute computer-readable instructions stored in memory 604, mass storage device 612, or other computer-readable media, such as program code of operating system 616, program code of application program 618, program code of other program 620, etc.

[0142] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to perform the various functions described above. For example, memory 604 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 612 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to perform the operations and functions described in the examples herein.

[0143] Multiple programs may be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they may be loaded into memory 604 for execution. Examples of such application programs or program modules may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0144] Although Figure 6 The modules 616, 618, 620, and 622, or portions thereof, are illustrated as being stored in memory 604 of computer device 600; however, modules 616, 618, 620, and 622 may be implemented using any form of computer-readable medium accessible by computer device 600. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0145] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0146] One or more communication interfaces 606 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 606 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 606 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0147] In some examples, a display device 608, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0148] The technologies described herein can be supported by these various configurations of computer device 600, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 600. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 600 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 600 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. An image processing method, characterized in that, The method includes: Get the Bayer image; Generating a location encoding vector corresponding to the spatial location of the Bayer image includes: Based on the color filter array arrangement pattern of the Bayer image, the phase information of each pixel in the Bayer image is determined; One-hot encoding is performed on the phase information of each pixel to generate a position encoding vector corresponding to the spatial position of the Bayer image, wherein the dimension of the position encoding vector is equal to the number of pixel phases contained in the smallest repeating unit of the color filter array arrangement pattern of the Bayer image. The location encoding vector is fused with the pixel values ​​of the Bayer image to generate fused feature data; Spatial context analysis is performed on the fused feature data to generate context feature data; Using a pre-trained neural network based on the contextual feature data, multiple routing weights corresponding to multiple image feature optimization strategies for the Bayer image are determined, wherein the multiple image feature optimization strategies include at least one of the following: An edge optimization strategy that sharpens and enhances contrast in image edge regions; Texture optimization strategies that preserve high-frequency details in image texture regions; Smooth region optimization strategies to improve color consistency in smooth areas of an image; and Noise optimization strategies for noise suppression in high-noise areas; The Bayer image is mapped using each of a plurality of pre-defined lookup tables to generate a plurality of candidate images. The plurality of lookup tables correspond to the plurality of image feature optimization strategies, and the resolution of the plurality of candidate images is higher than that of the Bayer image. Based on the multiple routing weights, the multiple candidate images are weighted and fused to generate the target image.

2. The method according to claim 1, characterized in that, The step of fusing the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data includes: The location encoding vector and the corresponding pixel values ​​of the Bayer image are concatenated along the channel dimension to generate the multi-channel fused feature data.

3. The method according to claim 2, characterized in that, The pixel value of the Bayer image is a single channel, and the dimension of the fused feature data is H x W x C, where H and W represent the height and width of the Bayer image, respectively, and C represents the total number of channels including pixel value channels and position encoding vector channels.

4. The method according to claim 3, characterized in that, The step of performing spatial context analysis on the fused feature data to generate context feature data includes: The channel weights of different channels in the fused feature data are extracted using a channel attention module, wherein the dimension of the channel weights is 1 x 1 x C; The spatial weights corresponding to different spatial locations in the fused feature data are extracted using a spatial attention module, wherein the dimension of the spatial weights is H x W x 1; The channel weights are multiplied with the fused feature data to generate channel feature data, wherein the dimension of the channel feature data is H x W x C; The spatial weights are multiplied point-by-point with the fused feature data to generate spatial feature data, wherein the spatial feature data has a dimension of H x W x C; The channel feature data and the spatial feature data are concatenated to obtain the context feature data, wherein the dimension of the context feature data is H x W x 2C.

5. The method according to claim 4, characterized in that, The step of using a pre-trained neural network to determine the multiple routing weights corresponding to the Bayer image optimization strategy for multiple image features based on the context feature data includes: The context feature data is input into the pre-trained neural network; Obtain an intermediate feature tensor of dimension H x W x (N x R²) from the output of the neural network, where N represents the number of the preset multiple lookup tables and R represents the resolution magnification factor of the multiple candidate images relative to the Bayer image; A pixel recombination operation is performed on the intermediate feature tensor to rearrange the intermediate feature tensor into a routing weight graph of dimension RH xRW x N, wherein each pixel position in the routing weight graph contains N weight values ​​corresponding to N lookup tables.

6. The method according to claim 4, characterized in that, The preset lookup tables are generated in advance through the following operations: Construct multiple independent lightweight convolutional neural networks, wherein each lightweight convolutional neural network is configured for one of the multiple image feature optimization strategies; Multiple lightweight convolutional neural networks are trained using loss functions optimized for different strategies. Based on the input-output mapping relationships of the trained lightweight convolutional neural networks, multiple corresponding lookup tables are established.

7. The method according to claim 6, characterized in that, The pre-trained neural network, the channel attention module, the spatial attention module, and the multiple lightweight convolutional neural networks are trained end-to-end as a whole.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire Bayer images; The generation module is configured to generate a location encoding vector corresponding to the spatial location of the Bayer image, wherein the generation module is further configured to: Based on the color filter array arrangement pattern of the Bayer image, the phase information of each pixel in the Bayer image is determined; One-hot encoding is performed on the phase information of each pixel to generate a position encoding vector corresponding to the spatial position of the Bayer image, wherein the dimension of the position encoding vector is equal to the number of pixel phases contained in the smallest repeating unit of the color filter array arrangement pattern of the Bayer image. The first fusion module is configured to fuse the location encoding vector with the pixel values ​​of the Bayer image to generate fused feature data; An analysis module is configured to perform spatial context analysis on the fused feature data to generate context feature data; The determination module is configured to use a pre-trained neural network to determine multiple routing weights corresponding to multiple image feature optimization strategies for the Bayer image based on the context feature data, wherein the multiple image feature optimization strategies include at least one of the following: An edge optimization strategy that sharpens and enhances contrast in image edge regions; Texture optimization strategies that preserve high-frequency details in image texture regions; Smooth region optimization strategies to improve color consistency in smooth areas of an image; and Noise optimization strategies for noise suppression in high-noise areas; The mapping module is configured to perform mapping processing on the Bayer image using each of a preset plurality of lookup tables to generate a plurality of candidate images, wherein the plurality of lookup tables correspond to the plurality of image feature optimization strategies, and wherein the resolution of the plurality of candidate images is higher than that of the Bayer image. The second fusion module is configured to perform weighted fusion of the multiple candidate images based on the multiple routing weights to generate a target image.

9. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-7.

Images