Array image demosaicing method based on dynamic convolution and adaptive coding

Abstract

The invention provides an array image demosaicing method based on dynamic convolution and adaptive coding, which relates to the technical field of image processing, and comprises the following steps: acquiring single-channel original image data and a corresponding color filtering array arrangement type identifier; converting the arrangement type identifier into a multi-dimensional physical feature vector, and inputting the multi-dimensional physical feature vector into a feature processor of a neural network model to generate a weighting coefficient vector; carrying out weighted combination on the plurality of special arrangement transformation matrixes through a weighting coefficient vector to obtain a transformation component, adding the transformation component and a basic convolution kernel parameter to obtain a dynamic convolution kernel parameter, and carrying out directional modulation on a specific spatial position of the dynamic convolution kernel parameter based on a direction weight component in a multi-dimensional physical feature vector; and performing convolution operation on the original image data by using the dynamic convolution kernel parameters, extracting multi-scale features, reconstructing image features, and outputting multi-channel color image data, so that the method can be adaptive to different color filter array types, and the demosaicing precision and generalization capability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an array image demosaicking method based on dynamic convolution and adaptive coding. BACKGROUND

[0002] In the field of digital image acquisition, due to cost and volume restrictions, most digital cameras and image sensors use a single sensor combined with a color filter array to capture color images. The color filter array captures only one of the three colors red, green, and blue at each pixel position, resulting in an incomplete single-channel mosaic image.

[0003] To obtain a complete three-channel color image, a demosaicking algorithm is needed to estimate the missing color components at each pixel position. Traditional demosaicking methods mainly include interpolation-based methods and edge-oriented methods, which rely on artificially designed rules and assumptions.

[0004] With the development of deep learning technology, neural network-based demosaicking methods have gradually become a research hotspot. Deep learning methods can automatically learn complex color reconstruction patterns through a large amount of training data, significantly improving image quality compared to traditional methods. However, existing technologies still have problems such as poor model universality, difficulty in adapting to multiple array types, difficulty in effectively capturing specific structural information, limited feature extraction ability for different array types, and failure to fully utilize directional information to guide feature extraction and image reconstruction, which limits the recovery of complex textures and edge details. SUMMARY

[0005] The present application provides a kind of, based on dynamic convolution and adaptive coding array image demosaicking method, can solve the problems in prior art.

[0006] The first aspect of the embodiment of the present application provides a kind of based on dynamic convolution and adaptive coding array image demosaicking method, comprising: obtain the single-channel original image data to be processed and its corresponding color filter array arrangement type identifier; convert the color filter array arrangement type identifier into a multi-dimensional physical feature vector, and input the multi-dimensional physical feature vector into a feature processor of a neural network model to generate a weighting coefficient vector, wherein the neural network model includes an encoder path and a decoder path, the encoder path and the decoder path are connected by a jump connection for feature fusion, the convolution layer of the neural network model adopts a dynamic convolution structure, and the dynamic convolution structure includes a basic convolution kernel parameter and a plurality of arrangement-specific transformation matrices; The multiple arrangement-specific transformation matrices are combined by the weighting coefficient vector to obtain a transformation component, the base convolution kernel parameter is added to the transformation component to obtain a dynamic convolution kernel parameter, and a specific spatial position of the dynamic convolution kernel parameter is directionally modulated based on a direction weight component in the multi-dimensional physical feature vector; The single-channel original image data is subjected to a convolution operation using the dynamic convolution kernel parameter, multi-scale feature representations are extracted through the encoder path, image features are reconstructed through the decoder path, and multi-channel color image data is output.

[0007] In an alternative embodiment, The direction weight component represents the interpolation priority of the horizontal direction, the vertical direction, the first diagonal direction, and the second diagonal direction, the color position component represents the spatial position index of different color channels in a preset grid unit, the neighborhood complexity component represents the complexity of the relationship between adjacent pixels, and the symmetry attribute component represents the geometric symmetry property of the color filter array arrangement.

[0008] In an alternative embodiment, The directional modulation of the specific spatial position of the dynamic convolution kernel parameter based on the direction weight component in the multi-dimensional physical feature vector includes: The multi-dimensional physical feature vector is input into a spatial feature processor, linearly transformed by a learnable weight matrix and a bias vector, and then subjected to a normalized exponential function to obtain a weighting coefficient vector, the number of components of the weighting coefficient vector being equal to the number of the multiple arrangement-specific transformation matrices; Each component of the weighting coefficient vector is scaled and summed with the corresponding arrangement-specific transformation matrix to obtain a transformation component; The base convolution kernel parameter is added to the transformation component to obtain a preliminary dynamic convolution kernel parameter; The direction weight component in the multi-dimensional physical feature vector is input into a direction feature processor, linearly transformed by a learnable direction weight matrix and a direction bias vector, and then subjected to a saturation activation function to obtain a direction modulation vector containing a horizontal and vertical modulation factor and a diagonal line modulation factor; The horizontal center position, the vertical center position, and the diagonal line position are determined according to the spatial index position of the preliminary dynamic convolution kernel parameter, the parameter values of the corresponding positions are multiplied by the scaling function values of the horizontal and vertical modulation factor and the diagonal line modulation factor, respectively, to obtain the dynamic convolution kernel parameter after directional modulation.

[0009] In an alternative embodiment, the method further includes: The gradient difference of the multi-channel color image data and reference image data in multiple directions is calculated, a loss weight of each direction is determined according to the multi-dimensional physical feature vector, a weighted directional gradient loss is combined with a pixel-level reconstruction loss to form a total loss function, and the parameters of the neural network model are updated based on the total loss function.

[0010] In an alternative embodiment, The updating of the parameters of the neural network model based on the total loss function comprises: An expert neural network model is constructed for each color filter array arrangement type, a multi-dimensional physical feature vector corresponding to the arrangement type is fixedly input into the expert neural network model, the parameters of the expert neural network model are optimized and trained using a first loss function, and an expert parameter set of multiple expert neural network models is obtained; A unified neural network model is constructed, the architecture of the unified neural network model is the same as that of the expert neural network model, and the unified neural network model receives a variable multi-dimensional physical feature vector as input; In the first training stage, a training sample is input into the unified neural network model to obtain a unified model output, the same training sample is input into the expert neural network model corresponding to the arrangement type to obtain an expert model output, a knowledge distillation loss between the unified model output and the expert model output is calculated, and all parameters of the unified neural network model are updated based on the knowledge distillation loss; In the second training stage, a norm difference between each parameter of the unified neural network model and a corresponding parameter in the expert parameter set is calculated to obtain an importance score, the parameters of the unified neural network model are sorted according to the importance score, and parameters with an importance score exceeding a preset proportion threshold are selected as an important parameter set; In the parameter updating process in the second training stage, only the parameters in the important parameter set are calculated for gradient and parameter updating, and the gradient values of the parameters not belonging to the important parameter set are set to zero.

[0011] In an alternative embodiment, The gradient difference of the multi-channel color image data and reference image data in multiple directions is calculated, a loss weight of each direction is determined according to the multi-dimensional physical feature vector, a weighted directional gradient loss is combined with a pixel-level reconstruction loss to form a total loss function, and the parameters of the neural network model are updated based on the total loss function. The multi-channel color image data and reference image data are respectively applied to multiple directional gradient operators for convolution operation to obtain a predicted directional gradient set and a real directional gradient set containing a horizontal directional gradient map, a vertical directional gradient map, a main diagonal directional gradient map and a secondary diagonal directional gradient map; For each direction, a difference between a corresponding direction gradient map in the set of predicted direction gradients and a corresponding direction gradient map in the set of real direction gradients is calculated to obtain a set of direction gradient losses including a horizontal direction gradient loss, a vertical direction gradient loss, a main diagonal direction gradient loss and a sub-diagonal direction gradient loss; The multi-dimensional physical feature vector is input into a learnable weight network, and the weight network outputs adaptive weight coefficients for each direction, and initial values of the adaptive weight coefficients are preset according to physical properties of the color filter array arrangement type corresponding to the multi-dimensional physical feature vector; Each direction gradient loss in the set of direction gradient losses is weighted and summed with the adaptive weight coefficient of the corresponding direction to obtain a direction perception loss. A pixel-level reconstruction loss between the multi-channel color image data and reference image data is calculated. The direction perception loss and the pixel-level reconstruction loss are weighted and combined to obtain the total loss function.

[0012] In a second aspect of the embodiment of the present application, an array image demosaicking system based on dynamic convolution and adaptive coding is provided, comprising: A first unit is configured to obtain single-channel original image data to be processed and a color filter array arrangement type identifier corresponding thereto. A second unit is configured to convert the color filter array arrangement type identifier into a multi-dimensional physical feature vector, and input the multi-dimensional physical feature vector into a feature processor of a neural network model to generate a weight coefficient vector, wherein the neural network model comprises an encoder path and a decoder path, the encoder path and the decoder path are connected by a skip connection for feature fusion, a convolution layer of the neural network model adopts a dynamic convolution structure, and the dynamic convolution structure comprises a basic convolution kernel parameter and a plurality of arrangement-specific transformation matrices. A third unit is configured to obtain a transformation component by weighted combination of the plurality of arrangement-specific transformation matrices through the weight coefficient vector, add the basic convolution kernel parameter and the transformation component to obtain a dynamic convolution kernel parameter, and directionally modulate a specific spatial position of the dynamic convolution kernel parameter based on a direction weight component in the multi-dimensional physical feature vector. A fourth unit is configured to perform convolution operation on the single-channel original image data using the dynamic convolution kernel parameter, extract multi-scale feature representation through the encoder path, reconstruct image features through the decoder path, and output multi-channel color image data.

[0013] In a third aspect of the embodiment of the present application, an electronic device is provided, comprising: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke the instructions stored by the memory to perform the aforementioned method.

[0014] In a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the aforementioned method.

[0015] In the present application, by converting the color filter array arrangement type identification into a multi-dimensional physical feature vector and generating a weighted coefficient vector using a feature processor to dynamically adjust the convolution kernel parameters, it can adaptively process color filter arrays of various different arrangement types, avoiding the need to train a model for each array type, significantly improving the versatility and practicality of the model, reducing the development and maintenance cost of the model, generating dynamic convolution kernel parameters through the weighted combination of the basic convolution kernel parameters and the arrangement-specific transformation matrix, and modulating the specific spatial position of the dynamic convolution kernel parameters based on the direction weight component in the multi-dimensional physical feature vector, so that the convolution operation can be optimized according to the physical characteristics and spatial arrangement rules of different color filter arrays, effectively improving the precision of the demosaicing process and the image reconstruction quality, using a neural network architecture containing an encoder path and a decoder path, and effectively fusing features of different levels through a skip connection, which can fully extract and utilize multi-scale feature representations, while preserving image detail information and realizing high-quality color image reconstruction, improving the robustness and adaptability of the demosaicing algorithm in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of the array image demosaicing method based on dynamic convolution and adaptive coding of the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0018] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0019] Figure 1A flowchart of an array image demosaicking method based on dynamic convolution and adaptive coding according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps. Figure 1 Obtaining single-channel raw image data to be processed and a corresponding color filter array arrangement type identifier. Converting the color filter array arrangement type identifier into a multi-dimensional physical feature vector, and inputting the multi-dimensional physical feature vector into a feature processor of a neural network model to generate a weighting coefficient vector, wherein the neural network model comprises an encoder path and a decoder path, the encoder path and the decoder path are connected by a skip connection for feature fusion, a convolution layer of the neural network model adopts a dynamic convolution structure, and the dynamic convolution structure comprises a basic convolution kernel parameter and a plurality of arrangement-specific transformation matrices. Weighting and combining the plurality of arrangement-specific transformation matrices to obtain a transformation component through the weighting coefficient vector, adding the basic convolution kernel parameter and the transformation component to obtain a dynamic convolution kernel parameter, and directionally modulating a specific spatial position of the dynamic convolution kernel parameter based on a direction weight component in the multi-dimensional physical feature vector. Performing convolution operation on the single-channel raw image data using the dynamic convolution kernel parameter, extracting multi-scale feature representation through the encoder path, reconstructing image features through the decoder path, and outputting multi-channel color image data.

[0020] In an optional implementation, The direction weight component represents interpolation priority of a horizontal direction, a vertical direction, a first diagonal direction, and a second diagonal direction, the color position component represents a spatial position index of different color channels in a preset grid unit, the neighborhood complexity component represents a complexity of a neighboring pixel relationship, and the symmetry attribute component represents a geometric symmetry property of the color filter array arrangement.

[0021] ​In the process of color image interpolation, four core components are constructed to represent the interpolation characteristics, which are direction weight component, color position component, neighborhood complexity component and symmetry attribute component. For the direction weight component, the gradient intensity values are calculated for the horizontal, vertical, first diagonal and second diagonal directions around the current pixel to be interpolated. Specifically, the gradient intensity of the horizontal direction is obtained by the difference accumulation of the left three pixel values and the right three pixel values of the current pixel, and the difference accumulation value reflects the change intensity of the horizontal direction. The gradient intensity of the vertical direction is calculated by the difference accumulation of the upper three pixel values and the lower three pixel values. The first diagonal direction refers to the diagonal direction from the top left to the bottom right, and the gradient intensity is determined by the difference accumulation of the top left pixel value sequence and the bottom right pixel value sequence. The second diagonal direction refers to the diagonal direction from the top right to the bottom left, and the gradient intensity is determined by the difference accumulation of the top right pixel value sequence and the bottom left pixel value sequence.

[0022] Exemplarily, assuming that the current pixel to be interpolated is located at the coordinate position, the luminance values of the left three pixels in the horizontal direction are 120, 118 and 115, and the luminance values of the right three pixels are 125, 128 and 130, then the gradient intensity value of the horizontal direction is the absolute value of the difference between the left mean value 118 and the right mean value 128, i.e. 10. Using the same calculation method, the gradient intensity value of the vertical direction is 8, the gradient intensity value of the first diagonal direction is 15, and the gradient intensity value of the second diagonal direction is 12. After taking the reciprocal of the four gradient intensity values and normalizing, the weight sum of the four directions is 1, so that the horizontal direction weight is 0.35, the vertical direction weight is 0.38, the first diagonal direction weight is 0.12, and the second diagonal direction weight is 0.15. The four weight values constitute the direction weight component, and the larger the weight value, the higher the interpolation priority of the direction.

[0023] The construction of the color position component needs to establish a spatial position index system of the color channel according to the arrangement mode of the color filter array. For the common Bayer array format, in a 2x2 basic repeating unit, the green filter is placed at the top left corner, the red filter is placed at the top right corner, the blue filter is placed at the bottom left corner, and the green filter is placed at the bottom right corner again. The system defines this 2x2 unit as a preset grid unit, and assigns index numbers to the four positions in the unit, with the top left corner being index 0, the top right corner being index 1, the bottom left corner being index 2, and the bottom right corner being index 3. In the actual image sensor array, any pixel point can determine its position index in the preset grid unit by taking the coordinate value modulo 2 operation.

[0024] Assuming that the row coordinate of a certain pixel point is 57 and the column coordinate is 83, the row index of the point is obtained by taking 57 modulo 2 to obtain 1, and the column index is obtained by taking 83 modulo 2 to obtain 1, so the pixel point corresponds to the index position 1 in the preset grid unit, that is, the upper right corner position, which means that the position should be placed with a red filter, and the pixel point records the original data of the red channel. For the green channel and the blue channel that need to be interpolated for the pixel point, according to the color position component, it is determined which positions in the adjacent pixels store the original data of the green and blue channels to guide the data selection in the interpolation calculation. The color position component is stored in the form of a two-dimensional array, each element of the array corresponds to a pixel position in the sensor array, and the element value is the index number of the position in the preset grid unit.

[0025] The neighborhood complexity component is used to quantify the complexity of the image content in the region around the current pixel point. A 5x5 neighborhood window is defined with the current pixel point to be interpolated as the center, and the known pixel values of the same color channel in the window are extracted. For these pixel values, the variance is calculated as a basic measurement index of complexity. The calculation of the variance is to first calculate the arithmetic mean of all pixel values in the window, then calculate the square of the difference between each pixel value and the mean, and finally average all square values. In a specific case, there are 12 known pixel values of the red channel in the 5x5 neighborhood window of a certain red channel pixel point, which are 110, 115, 108, 120, 118, 112, 125, 122, 116, 119, 113, and 121. The average of these 12 values is 116.58, and the square of the difference between each value and the average is calculated to obtain 298.92, which is divided by 12 to obtain the variance of 24.91. The variance value reflects the degree of variation in the local region, and the larger the variance, the higher the neighborhood complexity, and the image contains more details or edge structures. The variance value is mapped to the interval of 0 to 1 as the numerical representation of the neighborhood complexity component, and the region with high complexity needs to use a more conservative strategy to avoid artifacts in interpolation.

[0026] The symmetry attribute component represents the symmetry characteristics of the color filter array in the geometric structure. For the Bayer array, the 2x2 basic unit has rotational symmetry and mirror symmetry, and by analyzing the symmetry relationship of the preset grid unit relative to the center point, a symmetry attribute identifier is assigned to each position index. Specifically, the upper left corner position index 0 and the lower right corner position index 3 both correspond to the green channel, and have 180-degree rotational symmetry about the center point of the unit, so they are identified as a symmetric pair group. The right upper corner position index 1 corresponds to the red channel, and the left lower corner position index 2 corresponds to the blue channel, which also form a rotational symmetry relationship and are identified as another symmetric pair group. In actual interpolation calculation, when interpolating the missing color channel of a certain position, reference data is preferentially obtained from the corresponding symmetric position.

[0027] With an interpolation instance as an example, for a pixel point with row coordinate 48 and column coordinate 62, the position index thereof is 0, and the green channel thereof is known. When the red channel needs to be interpolated, the index 1 position storing the original red value is determined according to the symmetry attribute component, and the position is located at the positions of one pixel on the right side and one pixel on the lower side relative to the current point. At the same time, due to the symmetry, the index 3 position of one pixel on the left side and one pixel on the upper side is the green channel, but the index 1 diagonal position thereof also provides a red reference. The data of these symmetric positions are comprehensively utilized for weighted average, and the weights are determined by the direction weight component and the neighborhood complexity component, so that the interpolation calculation of the red channel of the pixel point is completed. In the whole process, the four core components work cooperatively to ensure that the interpolation result not only maintains the image details but also avoids color distortion.

[0028] In the embodiment, the gradient intensity of the horizontal direction, the vertical direction and the two diagonal directions is quantitatively analyzed by the direction weight component, so that the interpolation process can be adaptively estimated along the direction in which the texture changes the weakest, the false edges and the blur problems caused by the interpolation along the wrong direction can be effectively avoided, the edge fidelity and the detail maintaining ability can be significantly improved, the spatial structure of the Bayer array is indexed and described by the color position component, the color channel corresponding to each pixel position can be accurately identified, and only the correct same color or related color original data is selected to participate in the calculation during the interpolation, so that the color leakage and the color distortion problems caused by the misuse of pixels are eliminated, the color accuracy of the interpolation is improved, the degree of content change of the region is measured based on the local statistical variance by the neighborhood complexity component, so that the interpolation strategy can be dynamically adjusted according to the local complexity, a more conservative interpolation mode is used in the texture-rich or edge-dense region, the step false image, the mosaic false image and the ringing effect commonly seen in the high-frequency region are effectively suppressed, and the visual consistency and the naturalness of the interpolation result can be improved.

[0029] In an alternative embodiment, The directionality modulation of the specific spatial position of the dynamic convolution kernel parameter based on the direction weight component in the multi-dimensional physical feature vector includes: The multi-dimensional physical feature vector is input into a spatial feature processor, linearly transformed through a learnable weight matrix and a bias vector, and then normalized exponential function is applied to obtain a weighted coefficient vector, the number of components of the weighted coefficient vector is equal to the number of the plurality of arrangement-specific transformation matrices; Each component of the weighted coefficient vector is scaled and summed with the corresponding arrangement-specific transformation matrix to obtain a transformation component; The base convolution kernel parameter is added to the transformation component to obtain a preliminary dynamic convolution kernel parameter; The direction weight component in the multi-dimensional physical feature vector is input into a direction feature processor, and a linear transformation is performed through a learnable direction weight matrix and a direction bias vector, and then a saturation activation function is applied to obtain a direction modulation vector containing a horizontal and vertical modulation factor and a diagonal line modulation factor; A horizontal center position, a vertical center position, and a diagonal line position are determined according to the spatial index position of the preliminary dynamic convolution kernel parameter, and the parameter values of the corresponding positions are multiplied by the scaling function values of the horizontal and vertical modulation factor and the diagonal line modulation factor respectively to obtain the dynamic convolution kernel parameter after direction modulation.

[0030] The multi-dimensional physical feature vector is input into a spatial feature processor for processing. The multi-dimensional physical feature vector contains multiple numerical components, and each component represents a specific physical characteristic parameter. The spatial feature processor is internally provided with a learnable weight matrix and a bias vector. The weight matrix has a dimension of 64 rows by 8 columns, and the bias vector contains 8 elements. When the input multi-dimensional physical feature vector contains 64 components, a linear transformation is performed on the feature vector through the weight matrix. Each component of the feature vector is multiplied by the corresponding column elements of the corresponding row of the weight matrix and then summed to obtain 8 intermediate result values. The intermediate result values are added to the corresponding elements of the bias vector to complete the linear transformation process.

[0031] A normalization exponential function is applied to the 8 numerical values obtained by linear transformation. The function calculates the natural exponential value of each numerical value. For example, the first numerical value is 2.3, and its exponential value is 9.974. The second numerical value is 1.8, and its exponential value is 6.050. All 8 exponential values are summed to obtain a total value of 53.421. Each exponential value is divided by the total value to obtain 8 normalized weighting coefficients, which are 0.187, 0.113, 0.095, 0.142, 0.168, 0.091, 0.127, and 0.077, respectively. The 8 weighting coefficients form a weighting coefficient vector, and the number of components is exactly equal to the number of pre-set permutation special transformation matrices.

[0032] Eight permutation special transformation matrices are pre-stored. Each matrix has a dimension of 3 by 3 by 64 by 128, corresponding to the height, width, input channel number, and output channel number of the convolution kernel. The first component 0.187 of the weighting coefficient vector is multiplied by all the elements of the first permutation special transformation matrix one by one to obtain a scaled first matrix. The remaining 7 components and the corresponding transformation matrices are processed in the same way. The elements at the corresponding positions of the 8 scaled matrices are element-wise summed to obtain a final transformation component matrix, which is also a four-dimensional tensor structure with a dimension of 3 by 3 by 64 by 128.

[0033] The base convolution kernel parameters are pre-initialized as a four-dimensional tensor with the same dimensions, where the element value of a specific position, for example, height index 1, width index 1, input channel index 10, and output channel index 20, is 0.045. The base convolution kernel parameters are added to the elements of the corresponding positions of the transformation component, for example, the value of the transformation component at this position is 0.023, and the result of the addition is 0.068. The same addition operation is performed for all positions to obtain the preliminary dynamic convolution kernel parameters, which maintain the original four-dimensional tensor structure.

[0034] A direction weight component is extracted from the multi-dimensional physical feature vector, which contains four numerical elements representing the weight information of different directions, with specific values of 0.8, 0.6, 0.7, and 0.5. The direction weight component is input into a direction feature processor, which is configured with a learnable direction weight matrix and a direction bias vector. The direction weight matrix has a dimension of 4 rows by 4 columns, and the direction bias vector contains four elements. The direction weight component is linearly transformed by the direction weight matrix, where each element of the direction weight component is multiplied by the elements of the corresponding column of the matrix and accumulated to obtain four intermediate transformation values of 1.2, -0.8, 0.9, and -0.6. These intermediate values are added to the corresponding elements of the direction bias vector to obtain four linear transformation result values.

[0035] A saturation activation function is applied to the linear transformation result values to map the input values to the range of 0 to 1. When the input value is 1.5, the function calculates 0.905, and when the input value is -0.5, the function calculates 0.378. The function is applied to the four linear transformation result values respectively to obtain four processed values of 0.768, 0.310, 0.711, and 0.354. These four values constitute a direction modulation vector, where the first two components 0.768 and 0.310 are used as horizontal and vertical modulation factors, and the last two components 0.711 and 0.354 are used as diagonal modulation factors.

[0036] The spatial index of a specific position is determined according to the spatial structure of the preliminary dynamic convolution kernel parameters. The spatial dimension of the convolution kernel is 3 by 3, and all column positions with center row index 1 are defined as horizontal center positions, including index positions (1, 0), (1, 1), and (1, 2). All row positions with center column index 1 are defined as vertical center positions, including index positions (0, 1), (1, 1), and (2, 1). The diagonal positions are defined as the position set of index (0, 0), (1, 1), (2, 2), and index (0, 2), (1, 1), (2, 0).

[0037] The parameter values at the horizontal center position are modulated. The first component 0.768 of the horizontal and vertical modulation factor is extracted, and the scaling function value is calculated, which is the factor value minus 0.5, multiplied by 2, and then plus 1, to obtain the scaling value 1.536. The parameter values of all input and output channel combinations at the horizontal center position (1, 0) are multiplied by the scaling value, for example, the original parameter value of a certain channel combination position is 0.052, and after multiplication, 0.080 is obtained. The second component 0.310 of the horizontal and vertical modulation factor is used for the same processing at the vertical center position, and the scaling value is 0.620, which is multiplied by the corresponding position parameter value.

[0038] The parameter values at the diagonal position are modulated. The first component 0.711 of the diagonal modulation factor is extracted, and the scaling value 1.422 is calculated by the same scaling function. The parameter values at the main diagonal positions (0, 0), (1, 1), and (2, 2) are multiplied by the scaling value. The second component 0.354 of the diagonal modulation factor is used for the parameter values at the positions (0, 2) and (2, 0). The position (1, 1) belongs to both the horizontal and vertical center and the diagonal, and the product effect of all related modulation factors is applied to this position in turn. After all directional modulation is completed, the dynamic convolution kernel parameter is obtained, which exhibits targeted directional sensitivity characteristics at different spatial positions.

[0039] In this embodiment, the transformation mode based on physical feature driving breaks through the limitation that it cannot respond flexibly according to scene changes, so that the convolution kernel has content adaptive ability, and the adaptability and generality under different physical conditions are improved. The preliminary dynamic convolution kernel parameter is obtained by adding the weighted transformation component and the basic convolution kernel parameter, which realizes controllable adjustment of the convolution kernel under the premise of maintaining the stability of the basic structure, avoids the destruction of the convolution kernel structure caused by dynamic updating, can enhance the expression ability of the convolution kernel for local features while retaining the advantages of the original convolution kernel, and improves the fineness and robustness of feature extraction. The directional weight component is extracted from the multi-dimensional physical feature vector by the directional feature processor, and the directional modulation vector containing the horizontal and vertical modulation factor and the diagonal modulation factor is generated, so that the response ability of the convolution kernel in a certain direction can be enhanced or inhibited according to the local directional feature, effectively overcoming the problem that the traditional convolution kernel lacks directional recognition ability when processing physical images with clear directionality, significantly improving the capture ability of directional sensitive information and the preservation of directional structure. By applying the directional modulation factor to the horizontal, vertical, and diagonal spatial positions of the convolution kernel, the dynamic convolution kernel realizes differential modulation in spatial distribution, and further enhances the adaptive ability to different directional physical features.

[0040] In an alternative embodiment, the method further comprises: The gradient difference of the multi-channel color image data and the reference image data in multiple directions is calculated, a loss weight of each direction is determined according to the multi-dimensional physical feature vector, a weighted directional gradient loss is combined with a pixel-level reconstruction loss to form a total loss function, and parameters of the neural network model are updated based on the total loss function.

[0041] In the training process of the neural network model, a total loss function containing directional gradient perception is constructed to guide the optimization and update of the model parameters. Starting from the gradient difference calculation of the multi-channel color image data and the reference image data, the gradient loss of different directions is weighted through the multi-dimensional physical feature vector, and finally combined with the pixel-level reconstruction loss to form a complete training target.

[0042] For the multi-channel color image data and the reference image data, gradient information in the horizontal direction, the vertical direction, the left diagonal direction and the right diagonal direction needs to be calculated respectively. The calculation of the horizontal direction gradient is obtained by the numerical difference between the current pixel position and the adjacent pixel position on the right side, and the specific operation is to shift the image data by one pixel unit to the right in the width dimension, and subtract the data after shifting from the original data before shifting. For example, when the pixel value of a certain position is 125 and the pixel value on the right side is 130, the horizontal gradient value of the position is -5. The calculation of the vertical direction gradient adopts a similar way, which is obtained by the numerical difference between the current pixel position and the adjacent pixel position below, and the image data is shifted by one pixel unit downward in the height dimension, and the data after shifting is subtracted from the data before shifting. The calculation of the left diagonal direction gradient needs to consider the horizontal and vertical movements at the same time, and the image data is shifted to the right and down, that is, it is shifted by one unit in the width dimension and by one unit in the height dimension, and then the difference is calculated. The right diagonal direction gradient is calculated by shifting the image data to the left and down.

[0043] After completing the gradient calculation, the gradient difference between the multi-channel color image data and the reference image data is calculated for each direction. Taking the horizontal direction as an example, the horizontal gradient of the multi-channel color image data is subtracted from the horizontal gradient of the reference image data pixel by pixel to obtain the gradient error distribution in the horizontal direction. The absolute value of each value in the gradient error distribution is taken, the absolute value error of all pixel positions is summed and divided by the total number of pixels to obtain the average gradient difference in the horizontal direction. The same calculation method is used to obtain the average gradient difference in the vertical direction, the left diagonal direction and the right diagonal direction. For example, in an image with a resolution of 256*256, the absolute value sum of the horizontal direction gradient error of a certain channel is 3.2*10 6 , and the average gradient difference in this direction is 3.2*10 6 / 65536, about 48.83.

[0044] The multi-dimensional physical feature vector contains a variety of physical attribute information of the current image acquisition scene, and the physical attribute information is used to determine the importance weight of the gradient loss in different directions. When the component value representing the environmental illumination in the multi-dimensional physical feature vector is high, it indicates that the imaging scene is well-lit, and the contrast and clarity of the image are good, and the edge information in each direction is reliable, so relatively balanced weight values can be assigned to the four directions, for example, the horizontal direction weight is 0.26, the vertical direction weight is 0.25, the left diagonal direction weight is 0.24, and the right diagonal direction weight is 0.25. When the component value representing the motion speed in the multi-dimensional physical feature vector is high, it indicates that there is a large relative motion in the image acquisition process, and the gradient constraint weight in the main motion direction needs to be enhanced. Assuming that the motion mainly occurs in the horizontal direction, the horizontal direction weight can be increased to 0.45, and the weights of the other three directions can be reduced to 0.2, 0.18 and 0.17 respectively.

[0045] When the component value representing the imaging distance in the multi-dimensional physical feature vector is low, it indicates that the target object is close to the sensor, and the structure information in the vertical and horizontal directions of the image is usually more significant, so the weights of these two directions can be appropriately increased to 0.32 and 0.31, and the weights of the two diagonal directions can be reduced to 0.19 and 0.18. When the component value representing the atmospheric transparency in the multi-dimensional physical feature vector is low, it indicates that there are factors such as fog and smoke affecting the image, and the high-frequency details of the image are mainly concentrated in the diagonal directions, so the weights of the left and right diagonal directions need to be increased to 0.3 and 0.29, and the weights of the horizontal and vertical directions need to be reduced to 0.21 and 0.2. By establishing a mapping relationship between the component values of the multi-dimensional physical feature vector and the direction weights, the adaptive adjustment of the weights can be realized, and the mapping relationship is realized by a multi-layer fully connected neural network, the input is the multi-dimensional physical feature vector, and the output is the normalized weight values of the four directions.

[0046] After obtaining the loss weights of each direction, the average gradient difference in the horizontal direction is multiplied by the corresponding weight to obtain the weighted horizontal direction gradient loss, and the weighted gradient losses of the other three directions are calculated, and the weighted gradient losses of the four directions are added to obtain the total direction gradient loss. For example, when the horizontal direction average gradient difference is 48.83 and the weight is 0.26, the weighted loss of this direction is 12.69. Assuming that the weighted losses of the four directions are 12.69, 11.5, 9.8 and 10.25 respectively, the total direction gradient loss is 44.24.

[0047] The pixel-level reconstruction loss is obtained by calculating the direct difference between the multi-channel color image data and the reference image data. The two sets of image data are subtracted pixel by pixel, the absolute value or square value of the difference value is taken, and the error values of all pixel positions and all channels are averaged. For example, the error sum of squares of three color channels of an image at all pixel positions is 1.5*10 7 , the total number of pixels is 65536, and the number of channels is 3, then the pixel-level reconstruction loss is 1.5*10 7 / 196608, about 76.29.

[0048] The direction gradient loss and the pixel-level reconstruction loss are combined to form a total loss function. The relative importance of the two losses is adjusted by setting a balance coefficient, and the balance coefficient is determined according to the training stage and the data characteristics. In the early stage of training, the balance coefficient of the pixel-level reconstruction loss can be set to 0.7, and the balance coefficient of the direction gradient loss can be set to 0.3, so as to focus on the overall color and brightness restoration first. In the later stage of training, the pixel-level reconstruction loss coefficient can be adjusted to 0.5, and the direction gradient loss coefficient can be adjusted to 0.5, so as to balance the color restoration and detail preservation. The numerical value of the total loss function is equal to the product of the pixel-level reconstruction loss and its balance coefficient plus the product of the direction gradient loss and its balance coefficient. For example, when the pixel-level reconstruction loss is 76.29, the direction gradient loss is 44.24, and the two balance coefficients are both 0.5, the numerical value of the total loss function is 60.265.

[0049] Based on the calculated numerical value of the total loss function, the parameters of the neural network are updated using the back propagation algorithm. The gradient information of each parameter is obtained by calculating the partial derivative of the total loss function with respect to each learnable parameter in the network. The optimizer adjusts the parameter values according to the gradient information. The optimizer can use an adaptive learning rate method, and the initial learning rate is set to 0.001, which gradually decays to 0.0001 as the number of training iterations increases. During the parameter update process, a batch processing method is used, each batch contains sixteen image samples, and the total loss of each batch is calculated and averaged to obtain the overall loss of the batch, based on which the parameters are updated. After multiple rounds of iterative training, the parameters of the neural network gradually converge to the state that minimizes the total loss function, thereby realizing the high-quality multi-spectral image reconstruction capability.

[0050] In this embodiment, by calculating the gradient difference of multi-channel color image data and reference image data in multiple directions, the training process can explicitly focus on the structural changes of the image in the horizontal, vertical and diagonal directions, thereby enhancing the learning ability of directionally sensitive features such as edges and textures, effectively overcoming the difficulty in constraining structural and gradient information, making the reconstructed image perform better in contour clarity and detail preservation. By determining the loss weight of each direction according to the multi-dimensional physical feature vector, the training target dynamically changes with the local physical properties, which can optimize different physical feature scenes such as different noise levels, texture complexity and directionally prominent areas, and significantly improve the adaptability and robustness of the training process. By updating the neural network parameters based on the total loss function, the network can more effectively converge to an optimal state with structure fidelity and color restoration ability in the iterative optimization process, thereby improving the accuracy and stability in complex color image reconstruction tasks.

[0051] In an alternative embodiment, Updating the parameters of the neural network model based on the total loss function comprises: Respectively constructing an expert neural network model for each color filter array arrangement type, inputting a multi-dimensional physical feature vector corresponding to the arrangement type into the expert neural network model, optimizing and training the parameters of the expert neural network model using a first loss function, and obtaining an expert parameter set of the multiple expert neural network models; Constructing a unified neural network model, the architecture of the unified neural network model being the same as that of the expert neural network model, and the unified neural network model receiving a variable multi-dimensional physical feature vector as input; In the first training stage, inputting a training sample into the unified neural network model to obtain a unified model output, inputting the same training sample into the expert neural network model corresponding to the arrangement type to obtain an expert model output, calculating a knowledge distillation loss between the unified model output and the expert model output, and updating all parameters of the unified neural network model based on the knowledge distillation loss; In the second training stage, calculating the norm difference between each parameter of the unified neural network model and the corresponding parameter in the expert parameter set to obtain an importance score, sorting the parameters of the unified neural network model according to the importance score and selecting parameters with an importance score exceeding a preset proportion threshold as an important parameter set; In the parameter updating process of the second training stage, only the parameters in the important parameter set are calculated for gradient and parameter updating, and the gradient values of the parameters not belonging to the important parameter set are set to zero.

[0052] For the common color filter array arrangement types in the market, including RGGB type, BGGR type, GRBG type and GBRG type, an expert neural network model is constructed for each arrangement type. Each expert neural network model adopts the same architecture, which is composed of an encoder-decoder structure. The encoder contains 5 convolutional layers, each using a 3x3 convolutional kernel, and the channel numbers are 64, 128, 256, 512, and 512, respectively. The decoder also contains 5 transposed convolutional layers, and the channel numbers are 512, 256, 128, 64, and 3, respectively. The input of the expert neural network model is a multi-dimensional physical feature vector corresponding to the arrangement type, which contains original light sensing values, local gradient information, color channel correlation, and pixel position encoding, etc. The dimension is height x width x 12. For the RGGB type arrangement, the feature vector contains R channel original value, G1 channel original value, G2 channel original value, B channel original value, and corresponding gradient features and position encoding. The output of the expert neural network model is a complete RGB image, with a dimension of height x width x 3.

[0053] The expert neural network model is optimized and trained using a first loss function, which is composed of a pixel-level loss and a perception loss. The pixel-level loss calculates the absolute difference between the predicted RGB value and the true RGB value, and the perception loss extracts features through a pre-trained image recognition network and calculates the difference between the predicted result and the true image in the feature space. The training data set contains 10,000 pairs of images of different scenes, each pair containing a color filter array image and a corresponding true RGB image. The training process uses a batch size of 32, an initial learning rate of 0.0001, and an optimizer for parameter update, and is trained for 300 cycles until convergence. An expert model is trained for each array arrangement type, and an expert parameter set is obtained. Taking the RGGB type expert model as an example, it contains about 1,500,000 parameters, including convolutional layer weights and bias terms.

[0054] A unified neural network model is constructed, which has the same architecture as the expert neural network model, but receives a variable multi-dimensional physical feature vector as input and can automatically adjust the processing strategy according to the array arrangement type information contained in the input feature vector. The training of the unified neural network model is divided into two stages.

[0055] In the first training stage, the knowledge of the expert models is transferred to the unified model by the knowledge distillation technique. Specifically, for each sample in the training dataset, its array arrangement type is determined, and then the sample is input into the unified neural network model to obtain an output result, while the same sample is input into the expert neural network model of the corresponding arrangement type to obtain an expert output result. The knowledge distillation loss between the unified model output and the expert model output is calculated, and the loss function uses divergence to measure the difference between the two output distributions. The knowledge distillation loss is combined with the first loss function as the overall loss function of the unified model. Based on the loss function, the parameters of the unified neural network model are updated by the back propagation algorithm. Taking a batch of samples as an example, if it contains 8 RGGB type, 8 BGGR type, 8 GRBG type and 8 GBRG type images, they are input into the unified model and the corresponding type of expert model respectively, and the output difference is calculated to form the knowledge distillation loss. The learning rate is 0.00005 in this stage, and the training is 150 cycles.

[0056] In the second training stage, a parameter importance evaluation mechanism is introduced to identify key parameters that have a greater impact on model performance. The norm difference between each parameter of the unified neural network model and the corresponding parameter in the expert parameter set is calculated as the importance score. For the parameters in the unified model and the corresponding parameters in the expert model, the norm difference value between them is calculated as the importance score of the parameter. For example, the first weight parameter value of the first layer convolution of the unified model is 0.0328, while the corresponding parameter value of the RGGB expert model is 0.0356, the corresponding parameter value of the BGGR expert model is 0.0341, the corresponding parameter value of the GRBG expert model is 0.0335, and the corresponding parameter value of the GBRG expert model is 0.0349. The square root of the sum of squares of the differences between the current parameter and the parameters of each expert model is 0.0042, which is the importance score of the current parameter. According to the importance score, all parameters of the unified neural network model are sorted in descending order, and the top 20% of parameters with the highest importance scores are selected as the important parameter set. For a total of 1,500,000 parameters, the top 300,000 parameters are selected to form the important parameter set.

[0057] In the parameter updating process of the second training stage, only the parameters in the important parameter set are calculated for gradient and parameter updating, and the gradient values of the parameters not belonging to the important parameter set are set to 0. For example, if the fifth filter of the third layer convolution of the unified model is not in the important parameter set, then in the back propagation process, the gradient of all weight parameters of the filter is set to 0, and the parameter value remains unchanged.

[0058] Exemplarily, for a 640x480 pixel RGGB type color filter array image, its physical feature vector with a dimension of 640x480x12 is extracted first, which contains original photosensitive values, gradient features and position encodings. The feature vector is input into the trained unified neural network model, and after the encoder and decoder processing, a complete RGB image with a dimension of 640x480x3 is obtained. The entire processing process takes about 0.05 seconds on a device configured with a graphics processing unit, and the generated RGB image has clear texture details and natural color transitions. The same processing procedure is applicable to different types of color filter array images such as BGGR, GRBG and GBRG.

[0059] In this embodiment, by constructing an expert neural network model for each color filter array arrangement type and independently optimizing using a fixed multi-dimensional physical feature vector, each expert model can fully learn the feature distribution and optimal mapping relationship of its corresponding arrangement type, overcoming the training difficulty and ability dispersion problem caused by having to learn multiple arrangement patterns simultaneously. By comparing the unified model output with the corresponding expert model output and introducing a knowledge distillation loss, the structural knowledge and feature extraction capability of multiple expert models can be directly inherited, which can significantly improve the performance consistency and stability of the model under different array arrangements. Gradient calculation and update are performed on the important parameter set, while the gradients of the remaining parameters are set to zero, realizing an efficient selective optimization strategy that can maintain the important structures shared between expert models while avoiding performance instability caused by repeated updates of non-critical parameters. The cross-arrangement generalization capability, parameter sparsity, computational overhead and overall robustness are all improved.

[0060] In an alternative embodiment, The gradient differences of the multi-channel color image data and the reference image data in multiple directions are calculated, the loss weights of each direction are determined according to the multi-dimensional physical feature vector, and the weighted directional gradient loss and the pixel-level reconstruction loss are combined to form a total loss function, which includes: A plurality of directional gradient operators are applied to the multi-channel color image data and the reference image data respectively for convolution operation to obtain a set of predicted directional gradients and a set of true directional gradients containing horizontal directional gradient maps, vertical directional gradient maps, main diagonal directional gradient maps and secondary diagonal directional gradient maps; For each direction, the difference between the corresponding directional gradient map in the set of predicted directional gradients and the corresponding directional gradient map in the set of true directional gradients is calculated to obtain a set of directional gradient losses containing horizontal directional gradient loss, vertical directional gradient loss, main diagonal directional gradient loss and secondary diagonal directional gradient loss; inputting the multi-dimensional physical feature vector into a learnable weight network, the weight network outputting adaptive weight coefficients for each direction, initial values of the adaptive weight coefficients being preset according to physical characteristics of the color filter array arrangement type corresponding to the multi-dimensional physical feature vector; weighting and summing each directional gradient loss in the directional gradient loss set with the adaptive weight coefficient of the corresponding direction to obtain a direction-aware loss; calculating a pixel-level reconstruction loss between the multi-channel color image data and the reference image data; weighting and combining the direction-aware loss and the pixel-level reconstruction loss to obtain the total loss function.

[0061] applying a plurality of directional gradient operators to the multi-channel color image data and the reference image data respectively to obtain a gradient map set containing directional gradient information of different directions. Specifically, four classical directional gradient operators are used, including a Sobel horizontal direction operator, a Sobel vertical direction operator, a Prewitt main diagonal direction operator and a Prewitt secondary diagonal direction operator. The convolution kernel of the Sobel horizontal direction operator is a 3x3 matrix, the first row elements are -1, -2, -1, the second row elements are 0, 0, 0, and the third row elements are 1, 2, 1. The convolution kernel of the Sobel vertical direction operator is a 3x3 matrix, the first column elements are -1, -2, -1, the second column elements are 0, 0, 0, and the third column elements are 1, 2, 1. The convolution kernel of the Prewitt main diagonal direction operator is a 3x3 matrix, the diagonal elements are -2, the adjacent elements on the main diagonal are -1, and the remaining elements are 1. The convolution kernel of the Prewitt secondary diagonal direction operator is a 3x3 matrix, the secondary diagonal elements are -2, the adjacent elements on the secondary diagonal are -1, and the remaining elements are 1.

[0062] In practical applications, for a multi-channel color image data with a resolution of 640x480 (i.e. the output result of the neural network), it is decomposed into R, G, B three channels, each channel is respectively convolved with the above four directional gradient operators to obtain four directional gradient maps, all with a size of 638x478. Similarly, the reference image data, i.e. the ground truth image, is also processed in the same way to obtain the corresponding four directional gradient maps. The predicted directional gradient set contains the horizontal directional gradient map, the vertical directional gradient map, the main diagonal directional gradient map and the secondary diagonal directional gradient map, and the true directional gradient set also contains the corresponding four directional gradient maps.

[0063] For each direction, the difference between the predicted direction gradient map and the real direction gradient map is calculated to obtain a set of direction gradient losses. Specifically, for the horizontal direction gradient map, the absolute value of the difference between the predicted horizontal direction gradient map and the real horizontal direction gradient map corresponding to the pixel points is calculated, the sum of all pixel points is summed and averaged to obtain the horizontal direction gradient loss, and the vertical direction gradient loss, the main diagonal direction gradient loss and the secondary diagonal direction gradient loss are calculated. In the foregoing example of a 640x480 image, assuming that the horizontal direction gradient loss is 0.052, the vertical direction gradient loss is 0.063, the main diagonal direction gradient loss is 0.048, and the secondary diagonal direction gradient loss is 0.055.

[0064] The multi-dimensional physical feature vector is input into a learnable weight network to output adaptive weight coefficients for each direction. The weight network adopts a three-layer fully connected network structure, the input is the feature of the multi-dimensional physical feature vector after global average pooling, the dimension is 12, the output dimension of the first layer is 32, the ReLU activation function is used, the output dimension of the second layer is 16, the ReLU activation function is used, the output dimension of the third layer is 4, and the Softmax activation function is used to ensure that the sum of the four direction weight coefficients is 1. The initial value of the weight coefficient is preset according to the physical characteristics of the color filter array arrangement type. For example, for the RGGB type arrangement, since the green pixels are more evenly distributed along the diagonal line, the weight coefficients of the main diagonal and secondary diagonal directions are initialized to be slightly higher, set to 0.28, and the horizontal and vertical direction weight coefficients are set to 0.22; for the GRBG type arrangement, since the green pixels are more evenly distributed along the horizontal and vertical directions, the weight coefficients of the horizontal and vertical directions are initialized to be slightly higher, set to 0.28, and the diagonal direction weight coefficients are set to 0.22.

[0065] In actual application, for an RGGB type color filter array image, the four direction adaptive weight coefficients output after the multi-dimensional physical feature vector is processed by the weight network are: horizontal direction 0.15, vertical direction 0.20, main diagonal direction 0.35, and secondary diagonal direction 0.30.

[0066] The direction gradient losses in the direction gradient loss set are respectively weighted and summed with the adaptive weight coefficients of the corresponding direction to obtain a direction perception loss. Specifically, the direction perception loss is equal to the horizontal direction gradient loss multiplied by the horizontal direction weight coefficient, plus the vertical direction gradient loss multiplied by the vertical direction weight coefficient, plus the main diagonal direction gradient loss multiplied by the main diagonal direction weight coefficient, plus the secondary diagonal direction gradient loss multiplied by the secondary diagonal direction weight coefficient. In the foregoing example, the direction perception loss is calculated as: 0.052x0.15+0.063x0.20+0.048x0.35+0.055x0.30=0.0537.

[0067] The pixel-level reconstruction loss between the multi-channel color image data and the reference image data is calculated, and the average absolute error is used as the pixel-level reconstruction loss, that is, the absolute value of the difference between the corresponding pixels of the multi-channel color image data and the reference image data is calculated, and the sum of all pixel points is averaged. In the foregoing example, the pixel-level reconstruction loss is 0.047.

[0068] The direction perception loss and the pixel-level reconstruction loss are combined to obtain a total loss function, and the total loss function is equal to the pixel-level reconstruction loss multiplied by a first weight coefficient plus the direction perception loss multiplied by a second weight coefficient. In actual application, the first weight coefficient is set to 0.7, and the second weight coefficient is set to 0.3, so as to balance the pixel-level reconstruction accuracy and the direction structure retention capability. In the foregoing example, the total loss function is calculated as: 0.047*0.7+0.0537*0.3=0.0491.

[0069] In the embodiment, by applying a plurality of direction gradient operators to the multi-channel color image data and the reference image data respectively, the predicted gradient and the real gradient of four directions (horizontal, vertical, main diagonal line and secondary diagonal line) are obtained, so that the training process of the model is no longer limited to the overall brightness error, but can explicitly focus on the structural differences in different directions, overcoming the problem that the conventional pixel loss is difficult to constrain the edge directionality feature, and can better retain the texture, contour and direction structure of the image. By independently calculating the gradient loss for each direction, the loss measurement can reflect the error size of each direction with finer granularity, thereby avoiding the limitation that the direction information is weakened after the direction error is mixed in the traditional gradient loss, which helps to reduce the problems of direction blur, jaggy artifacts and detail collapse. By inputting the multi-dimensional physical feature vector into the weight network, the model can automatically generate adaptive weight coefficients for each direction according to the arrangement type of the color filter array and its physical characteristics, realize differentiated attention to gradient loss in different directions, improve the flexibility when processing different color filter array structures, make the training target more consistent with the actual physical imaging law, and improve the cross-arrangement generalization ability and training stability.

[0070] In a second aspect of the embodiment of the application, an array image demosaicking system based on dynamic convolution and adaptive coding is provided, comprising: A first unit is configured to obtain single-channel original image data to be processed and a corresponding color filter array arrangement type identifier. The second unit is configured to convert the color filter array arrangement type identifier into a multi-dimensional physical feature vector, and input the multi-dimensional physical feature vector into a feature processor of a neural network model to generate a weighted coefficient vector, wherein the neural network model comprises an encoder path and a decoder path, the encoder path and the decoder path are connected by a skip connection to fuse features, a convolution layer of the neural network model adopts a dynamic convolution structure, and the dynamic convolution structure comprises a basic convolution kernel parameter and a plurality of arrangement-specific transformation matrices. The third unit is configured to weight and combine the plurality of arrangement-specific transformation matrices to obtain a transformation component through the weighted coefficient vector, add the basic convolution kernel parameter and the transformation component to obtain a dynamic convolution kernel parameter, and directionally modulate a specific spatial position of the dynamic convolution kernel parameter based on a direction weight component in the multi-dimensional physical feature vector. The fourth unit is configured to perform convolution operation on the single-channel original image data using the dynamic convolution kernel parameter, extract a multi-scale feature representation through the encoder path, reconstruct an image feature through the decoder path, and output multi-channel color image data.

[0071] In a third aspect, an electronic device is provided, comprising: a processor; a memory configured to store processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0072] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0073] The present application can be a method, apparatus, system, and / or computer program product. The computer program product can include a computer-readable storage medium having computer-readable program instructions loaded thereon, which are used to perform various aspects of the present application.

[0074] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for demosaicing array images based on dynamic convolution and adaptive coding, characterized in that, include: Obtain the single-channel raw image data to be processed and its corresponding color filter array arrangement type identifier; The color filter array arrangement type identifier is converted into a multi-dimensional physical feature vector, and the multi-dimensional physical feature vector is input into the feature processor of the neural network model to generate a weighted coefficient vector. The neural network model includes an encoder path and a decoder path. The encoder path and the decoder path are fused through skip connections. The convolutional layer of the neural network model adopts a dynamic convolutional structure, which includes basic convolutional kernel parameters and multiple arrangement-specific transformation matrices. The transformation components are obtained by weighting and combining multiple permutation-specific transformation matrices using the weighted coefficient vector. The basic convolution kernel parameters are added to the transformation components to obtain dynamic convolution kernel parameters. The dynamic convolution kernel parameters are then directionally modulated at a specific spatial position based on the directional weight components in the multidimensional physical feature vector. The single-channel raw image data is subjected to convolution operation using the dynamic convolution kernel parameters. Multi-scale feature representations are extracted through the encoder path, and image features are reconstructed through the decoder path to output multi-channel color image data.

2. The method according to claim 1, characterized in that, The directional weight component represents the interpolation priority of the horizontal direction, vertical direction, first diagonal direction and second diagonal direction; the color position component represents the spatial position index of different color channels in the preset grid cell; the neighborhood complexity component represents the complexity of the relationship between adjacent pixels; and the symmetry attribute component represents the geometric symmetry property of the color filter array arrangement.

3. The method according to claim 1, characterized in that, Directional modulation of the dynamic convolution kernel parameters at specific spatial locations based on the directional weight components in the multidimensional physical feature vector includes: The multidimensional physical feature vector is input into the spatial feature processor. After linear transformation through the learnable weight matrix and bias vector, a normalized exponential function is applied to obtain the weighted coefficient vector. The number of components of the weighted coefficient vector is equal to the number of the multiple permutation-specific transformation matrices. The transformation components are obtained by scaling each component of the weighted coefficient vector with the corresponding permutation-specific transformation matrix and summing the results. The basic convolution kernel parameters are added to the transform components to obtain the preliminary dynamic convolution kernel parameters; The directional weight components in the multidimensional physical feature vector are input into the directional feature processor. After linear transformation of the learnable directional weight matrix and directional bias vector, a saturation activation function is applied to obtain a directional modulation vector containing horizontal and vertical modulation factors and diagonal modulation factors. The horizontal center position, vertical center position, and diagonal position are determined based on the spatial index position of the preliminary dynamic convolution kernel parameters. The parameter values ​​at the corresponding positions are multiplied by the scaling function values ​​of the horizontal and vertical modulation factors and the diagonal modulation factors, respectively, to obtain the dynamic convolution kernel parameters that complete directional modulation.

4. The method according to claim 1, characterized in that, The method further includes: The gradient differences between the multi-channel color image data and the reference image data in multiple directions are calculated. The loss weights for each direction are determined based on the multi-dimensional physical feature vector. The weighted directional gradient loss is combined with the pixel-level reconstruction loss to form a total loss function. The parameters of the neural network model are updated based on the total loss function.

5. The method according to claim 4, characterized in that, The parameters of the neural network model updated based on the total loss function include: For each type of color filter array arrangement, an expert neural network model is constructed. The multidimensional physical feature vector of the corresponding arrangement type is fixedly input into the expert neural network model. The parameters of the expert neural network model are optimized and trained using a first loss function to obtain a set of expert parameters for multiple expert neural network models. A unified neural network model is constructed, the architecture of which is the same as that of the expert neural network model, and the unified neural network model receives a variable multidimensional physical feature vector as input; In the first training phase, training samples are input into the unified neural network model to obtain the unified model output, and the same training samples are input into the expert neural network model of the corresponding permutation type to obtain the expert model output. The knowledge distillation loss between the unified model output and the expert model output is calculated, and all parameters of the unified neural network model are updated based on the knowledge distillation loss. In the second training phase, the norm difference between each parameter of the unified neural network model and the corresponding parameter in the expert parameter set is calculated to obtain an importance score. The parameters of the unified neural network model are sorted according to the importance scores, and parameters with importance scores exceeding a preset proportion threshold are selected as the important parameter set. During the parameter update process in the second training phase, gradients are calculated and parameter updates are performed only for parameters in the set of important parameters, and gradient values ​​for parameters that do not belong to the set of important parameters are set to zero.

6. The method according to claim 4, characterized in that, The gradient differences between the multi-channel color image data and the reference image data in multiple directions are calculated. The loss weights for each direction are determined based on the multi-dimensional physical feature vector. The weighted directional gradient loss is combined with the pixel-level reconstruction loss to form the total loss function, which includes: Multiple directional gradient operators are applied to the multi-channel color image data and the reference image data to perform convolution operations, resulting in a predicted directional gradient set and a true directional gradient set containing horizontal directional gradient maps, vertical directional gradient maps, main diagonal directional gradient maps, and secondary diagonal directional gradient maps; For each direction, the difference between the corresponding direction gradient map in the predicted direction gradient set and the corresponding direction gradient map in the true direction gradient set is calculated to obtain a direction gradient loss set including horizontal direction gradient loss, vertical direction gradient loss, main diagonal direction gradient loss and secondary diagonal direction gradient loss. The multidimensional physical feature vector is input into a learnable weight network, and the weight network outputs adaptive weight coefficients for each direction. The initial values ​​of the adaptive weight coefficients are preset according to the physical characteristics of the color filter array arrangement type corresponding to the multidimensional physical feature vector. The orientation gradient loss in the set of orientation gradient losses is weighted and summed with the adaptive weight coefficients of the corresponding directions to obtain the orientation-aware loss. Calculate the pixel-level reconstruction loss between the multi-channel color image data and the reference image data; The total loss function is obtained by weighting and combining the orientation perception loss and the pixel-level reconstruction loss.

7. An array image desacrifice system based on dynamic convolution and adaptive coding, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire the single-channel raw image data to be processed and its corresponding color filter array arrangement type identifier; The second unit is used to convert the color filter array arrangement type identifier into a multi-dimensional physical feature vector, and input the multi-dimensional physical feature vector into the feature processor of the neural network model to generate a weighted coefficient vector. The neural network model includes an encoder path and a decoder path. The encoder path and the decoder path are fused through skip connections. The convolutional layer of the neural network model adopts a dynamic convolutional structure. The dynamic convolutional structure includes basic convolutional kernel parameters and multiple arrangement-specific transformation matrices. The third unit is used to obtain transformation components by weighting multiple permutation-specific transformation matrices through the weighted coefficient vector, add the basic convolution kernel parameters to the transformation components to obtain dynamic convolution kernel parameters, and perform directional modulation on the specific spatial position of the dynamic convolution kernel parameters based on the directional weight components in the multidimensional physical feature vector. The fourth unit is used to perform convolution operations on the single-channel raw image data using the dynamic convolution kernel parameters, extract multi-scale feature representations through the encoder path, reconstruct image features through the decoder path, and output multi-channel color image data.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images