Method for calculating color constancy of sequential images based on convolutional neural network

Abstract

The application relates to a convolutional neural network-based sequence image color constancy calculation method, which comprises the following steps: reading an original image sequence by using a preset composite model, performing image correction and cutting on the original image sequence to obtain a to-be-processed image sequence meeting preset image standard conditions; performing preset transformation processing on key frame images of the to-be-processed image sequence to obtain a simulation image sequence; extracting a first image feature sequence from the original image sequence and a second image feature sequence from the simulation image sequence by using two preset convolutional neural networks; processing the first image feature sequence and the second image feature sequence by using a gated recurrent unit (GRU) to output a first tensor and a second tensor; and generating three channels corresponding to an RGB color space based on a splicing tensor of the first tensor and the second tensor, so as to generate light source information for correcting image colors according to the three-channel features.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and image processing, and in particular to a method for calculating color constancy of sequential images based on a convolutional neural network. Background Art

[0002] With the continuous development of computer vision and digital image processing, the application of color images is becoming increasingly popular. In this context, the processing of color information has become a research focus. The stability of color information is a prerequisite for visual tasks such as feature extraction, object detection, semantic segmentation, and image recognition.

[0003] However, in practical applications, due to the influence of the object's own material and lighting conditions, the color in the image often deviates, posing challenges to the work and practical applications of computer vision. Therefore, color correction has become an essential preprocessing step. Its purpose is to eliminate the influence of lighting on the color of the object and make the color in the image more realistic and stable. Achieving color constancy is not only an important component of the digital imaging system, but also a key step in improving the accuracy of image analysis. Through color correction, color deviation can be effectively eliminated, making the image closer to the real scene and providing more reliable data support for subsequent image analysis tasks. At the same time, the application of color constancy algorithms can also improve the stability and adaptability of computer vision systems, achieving consistent recognition results under various complex lighting conditions, thereby further improving the accuracy and robustness of image algorithms.

[0004] The color of an object is the result of the combined effects of its material's reflective properties and the color of the light source illuminating it. However, the human visual system has a remarkable ability to adaptively adjust its response to changes in light sources, allowing it to perceive the true color of an object under standard white light in any lighting environment. This allows us to consistently identify the color of an object under varying lighting conditions, such as switching from artificial lighting indoors to natural light outdoors. The study of color constancy is a research area that connects computer vision and human vision. It explores how the human eye effectively ignores the effects of light source changes on an object's color and integrates this adaptability into computer vision systems. However, traditional color constancy algorithms, based on simplified assumptions, struggle to adapt to complex lighting conditions. Summary of the Invention

[0005] This application provides a method for calculating the color constancy of sequential images based on a convolutional neural network to solve the problem that traditional color constancy algorithms are difficult to adapt to complex lighting.

[0006] The first aspect of the present application provides a method for calculating the color constancy of sequence images based on a convolutional neural network, comprising the following steps: reading an original image sequence using a preset composite model, and performing image correction and cropping on the original image sequence to obtain a sequence of images to be processed that meets preset image standard conditions, wherein the preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks, the two preset convolutional neural networks are both SqueezeNet networks constructed by the RFCM module, and the two preset recurrent neural networks are both gated recurrent units GRU; performing a preset transformation processing on the key frame images of the image sequence to be processed to obtain a simulated image sequence, and using the two preset convolutional neural networks to respectively extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence; using the gated recurrent unit GRU to respectively process the first image feature sequence and the second image feature sequence to output a first tensor and a second tensor, and generating three channels corresponding to the RGB color space based on the spliced ​​tensor of the first tensor and the second tensor to generate light source information for correcting the image color according to the three-channel features.

[0007] Optionally, the key frame images of the image sequence to be processed are subjected to a preset transformation processing to obtain a simulated image sequence, including: determining an initial cropping position of a target cropped image, and cropping the key frame image according to the initial cropping position; rotating the cropped key frame image by a preset rotation angle, and cropping the rotated key frame image to obtain a square image, scaling the square image to the target pixel, and assembling the scaled square images to obtain the simulated image sequence.

[0008] Optionally, the RFCM module is:

[0009] X out =ω*CAM(Conv(X in ))+Fire(X in )

[0010] Among them, X out is the preset number of channels, ω is the weight vector, X in is the number of input channels, Conv(X in ) is the input tensor, CAM(Conv(X in )) is X in The tensor obtained by convolution and attention module, Fire(X in ) is the output tensor of Fire.

[0011] Optionally, the gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is:

[0012] z t =σ(W xz x t +W hz h t-1 +b z )

[0013] Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters, b z is the door offset;

[0014] The reset gate is:

[0015] r t =σ(W xr x t +W hr h t-1 +b r )

[0016] Among them, r t To reset the door switch, b r is the gate bias, W xr is the tensor x t Parameters, W hr is the tensor h t-1 Parameters.

[0017] The second aspect of the present application provides a device for calculating the color constancy of sequence images based on a convolutional neural network, including: a reading module for reading an original image sequence using a preset composite model, and performing image correction and cropping on the original image sequence to obtain a sequence of images to be processed that meets preset image standard conditions, wherein the preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks, the two preset convolutional neural networks are SqueezeNet networks constructed by the RFCM module, and the two preset recurrent neural networks are gated recurrent units GRU; an extraction module for performing a preset transformation processing on the key frame images of the image sequence to be processed to obtain a simulated image sequence, and using the two preset convolutional neural networks to respectively extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence; a generation module for using the gated recurrent unit GRU to respectively process the first image feature sequence and the second image feature sequence to output a first tensor and a second tensor, and generate three channels corresponding to the RGB color space based on the spliced ​​tensor of the first tensor and the second tensor, so as to generate light source information for correcting the image color according to the three-channel features.

[0018] Optionally, the extraction module is also used to: determine the initial cropping position of the target cropped image, and crop the key frame image according to the initial cropping position; rotate the cropped key frame image by a preset rotation angle, and crop the rotated key frame image to obtain a square image, scale the square image to the target pixel, and assemble the scaled square images to obtain the simulated image sequence.

[0019] Optionally, the RFCM module is:

[0020] X out =ω*CAM(Conv(X in ))+Fire(X in )

[0021] Among them, X out is the preset number of channels, ω is the weight vector, X in is the number of input channels, Conv(X in ) is the input tensor, CAM(Conv(X in )) is X in The tensor obtained by convolution and attention module, Fire(X in ) is the output tensor of Fire.

[0022] Optionally, the gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is:

[0023] zt =σ(W xz x t +W hz h t-1 +b z )

[0024] Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters, b z is the door offset;

[0025] The reset gate is:

[0026] r t =σ(W xr x t +W hr h t-1 +b r )

[0027] Among them, r t To reset the door switch, b r is the gate bias, W xr is the tensor x t Parameters, W hr is the tensor h t-1 Parameters.

[0028] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the method for calculating the color constancy of sequential images based on a convolutional neural network as described in the above embodiment.

[0029] The fourth aspect of the present application provides a computer program product having a computer program stored thereon, which is executed by a processor to implement the method for calculating the color constancy of sequential images based on a convolutional neural network as described in the above embodiment.

[0030] In the above embodiment, a preset composite model is used to read the original image sequence, and the original image sequence is subjected to image correction and cropping to obtain a to-be-processed image sequence that meets the preset image standard conditions. The key frame images of the to-be-processed image sequence are subjected to a preset transformation to obtain a simulated image sequence. Two preset convolutional neural networks are used to extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence, respectively. The gated recurrent unit (GRU) is used to process the first image feature sequence and the second image feature sequence, respectively, to output a first tensor and a second tensor. Based on the concatenated tensor of the first tensor and the second tensor, three channels corresponding to the RGB color space are generated to generate light source information for correcting the image color based on the three-channel features. This solves the problem that traditional color constancy algorithms are difficult to adapt to complex lighting. The model can understand the changes in the scene at different time scales, thereby achieving color correction in complex actual shooting environments.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A flowchart of a method for calculating color constancy of sequential images based on a convolutional neural network according to an embodiment of the present application;

[0034] Figure 2 Schematic diagram of the Fire Module structure according to one embodiment of the present application;

[0035] Figure 3 Schematic diagram of the structure of an improved RFCM module based on the Fire Module according to one embodiment of the present application;

[0036] Figure 4 Schematic diagram of a convolutional neural network constructed based on SqueezeNet and RFCM according to one embodiment of the present application;

[0037] Figure 5 Schematic diagram of a gated recurrent unit GRU used according to one embodiment of the present application;

[0038] Figure 6 A flowchart of constructing a simulation sequence according to one embodiment of the present application;

[0039] Figure 7 is a schematic diagram of a simulation sequence according to one embodiment of the present application;

[0040] Figure 8 is a structural diagram of a composite model according to an embodiment of the present application;

[0041] Figure 9 This is an example diagram of a device for calculating color constancy of sequential images based on a convolutional neural network according to an embodiment of the present application;

[0042] Figure 10 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0043] Reference numerals:

[0044] 10-Computing device for color constancy of sequential images based on convolutional neural network; 100-Reading module; 200-Extraction module; 300-Generation module; 1001-Memory; 1002-Processor; 1003-Communication interface. DETAILED DESCRIPTION

[0045] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0046] The following describes a method for calculating color constancy of sequential images based on a convolutional neural network according to an embodiment of the present application, with reference to the accompanying drawings. In response to the problem mentioned in the background art above that traditional color constancy algorithms are difficult to adapt to complex lighting, the present application provides a method for calculating color constancy of sequential images based on a convolutional neural network. In this method, a preset composite model is used to read the original image sequence, and the original image sequence is subjected to image correction and cropping to obtain a sequence of images to be processed that meets preset image standard conditions. A preset transformation is performed on the keyframe images of the processed image sequence to obtain a simulated image sequence. Two preset convolutional neural networks are used to extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence, respectively. A gated recurrent unit (GRU) is used to process the first image feature sequence and the second image feature sequence, respectively, to output a first tensor and a second tensor. Based on the concatenation of the first and second tensors, three channels corresponding to the RGB color space are generated to generate light source information for correcting the image color based on the three-channel features. This solves the problem that traditional color constancy algorithms are difficult to adapt to complex lighting. The model can understand scene changes at different time scales, thereby achieving color correction in complex actual shooting environments.

[0047] The rise of deep learning, particularly convolutional neural networks (CNNs), has brought new opportunities for color constancy calculations. CNNs can directly process images and, by learning from a large number of samples, automatically extract color constancy features and predict light source information for image correction. This approach offers greater adaptability, improving computational accuracy and robustness. Sequential images provide continuous image data, enabling color constancy algorithms based on these images to more accurately analyze light sources, reduce errors, and enhance model robustness. With the increasing popularity and performance of smart devices, sequential image color constancy algorithms are becoming more aligned with user photography habits, providing hardware support for light source calculation and image correction on mobile devices.

[0048] Specifically, Figure 1 A flowchart of a method for calculating color constancy of sequential images based on a convolutional neural network provided in an embodiment of the present application.

[0049] like Figure 1 As shown, the method for calculating color constancy of sequential images based on convolutional neural network includes the following steps:

[0050] In step S101, the original image sequence is read using a preset composite model, and the original image sequence is corrected and cropped to obtain a sequence of images to be processed that meets the preset image standard conditions, wherein the preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks, both of which are SqueezeNet networks constructed by the RFCM module, and both of which are gated recurrent units GRU.

[0051] Optionally, in some embodiments, the RFCM module is:

[0052] X out =ω*CAM(Conv(X in ))+Fire(X in )

[0053] Among them, X out is the preset number of channels, ω is the weight vector, X in is the number of input channels, Conv(X in ) is the input tensor, CAM(Conv(X in )) is X in The tensor obtained by convolution and attention module, Fire(X in ) is the output tensor of Fire.

[0054] Optionally, in some embodiments, the gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is:

[0055] zt =σ(W xz x t +W hz h t-1 +b z )

[0056] Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters, b z is the door offset;

[0057] Reset gate to:

[0058] r t =σ(W xr x t +W hr h t-1 +b r )

[0059] Among them, r t To reset the door switch, b r is the gate bias, W xr is the tensor x t Parameters, W hr is the tensor h t-1 Parameters.

[0060] In order to solve the problem of learning sequential images, this application adopts convolutional neural networks to extract image features and uses its complex network structure to learn the potential information in image features. In order to process sequential images, recurrent neural networks (RNNs) are introduced. RNNs have internal states and can consider previous information when processing new inputs, thereby being able to learn long-term dependencies in sequential data.

[0061] The convolutional neural network uses SqueezeNet as the backbone network to extract image features. Building on the Fire Module in SqueezeNet, the model introduces a channel attention mechanism and a residual module, called the Residual Fire with Channel Attention Module (RFCM). Because processing sequential images significantly increases image information entropy, the introduction of the channel attention mechanism emphasizes key channels in the feature map and adaptively adjusts the weights of these channels, thereby increasing the model's sensitivity to color information.

[0062] The recurrent neural network in this embodiment uses a gated recurrent unit (GRU) to process sequential images. Traditional RNN structures often encounter vanishing or exploding gradients when processing long sequences, severely limiting their performance in learning long-term dependencies. The gated recurrent unit (GRU) effectively addresses these issues by introducing update and reset gates to memorize long-term sequence information. This maintains a stable gradient flow during training.

[0063] In order to make full use of the key information in the leading frame and assist the model in capturing the core content of the captured frame, this application proposes a color constancy calculation method based on dual-head input. The core of this method is to construct a composite model consisting of two convolutional neural networks and two recurrent neural networks. This structure enables the model to extract contextual information from the leading frame that is helpful for color constancy estimation, while also being able to understand the instantaneous features of the captured frame, thereby more comprehensively understanding the dynamic changes of the scene. In order to improve the learning efficiency of sequential images, this application introduces a sequence generation mechanism that increases the information of the sequential image by simulating the movement of key frames in the leading frame, thereby improving the performance of the model in sequential image learning.

[0064] The model in this application integrates a dual-head mechanism, a modified RFCM module, and a gated recurrent unit (GRU). Compared to previous models, the new dual-head model improves performance while also achieving a balance of efficiency. The RFCM-based SqueezeNet serves as the network's feature extractor, responsible for extracting high-level features from images, while the gated recurrent unit (GRU) focuses on understanding the input of image sequences and capturing temporal dynamics within them. By fusing the dual-head input, the model is able to parse sequential image inputs as well as sequences composed of simulated blocks of keyframes.

[0065] This application combines the channel attention mechanism, the SqueezeNet of the residual module and the gated recurrent unit GRU to process a given image sequence and realize global illumination estimation. This application uses convolutional neural networks (CNN) and gated recurrent units GRU to process sequence images. The core of the model is to combine the feature extraction capabilities of CNNs with the sequence processing capabilities of the gated recurrent unit GRU, making full use of the spatial and temporal characteristics of the image sequence to realize the color constancy calculation of the image. In order to make full use of the key information in the leading frame and assist the model in capturing the core content of the shooting frame, a color constancy calculation method based on dual-head input is proposed. By constructing a preset composite model consisting of two convolutional neural networks and two recurrent neural networks, the model can extract contextual information that is helpful for color constancy estimation from the leading frame, and can also understand the instantaneous characteristics of the shooting frame in time, so as to more comprehensively understand the dynamic changes of the scene.

[0066] Among them, SqueezeNet is a lightweight convolutional neural network model that aims to significantly improve processing speed and computational efficiency while maintaining model accuracy. Its core design concept is to reduce model parameters and computational complexity while maintaining performance, so that it can run in resource-constrained environments, such as mobile devices and embedded systems. The core module of SqueezeNet is the Fire Module, such as Figure 2 As shown in the figure, this module consists of three main parts: Squeeze layer, Expand layer and splicing operation.

[0067] Squeeze layer: The input feature map is compressed through a 1×1 convolution kernel to reduce the number of feature map channels, thereby reducing model parameters and computational complexity.

[0068] Expand layer: The compressed feature map is expanded by combining a 1×1 convolution kernel and a 3×3 convolution kernel. The 1×1 convolution kernel further reduces the number of channels, and the 3×3 convolution kernel increases the nonlinear processing capability.

[0069] Splicing operation: The outputs of the Squeeze layer and the Expand layer are spliced ​​to form a new feature map, which serves as the input of the next module.

[0070] By replacing most 3×3 convolution kernels with 1×1 convolution kernels, SqueezeNet is able to achieve delayed downsampling while reducing the number of model parameters. This means that smaller convolution kernels are used in the early layers of the network, while larger convolution kernels are used in deeper layers, reducing information loss caused by downsampling and improving network performance.

[0071] This application designs a Fire Module (RFCM) based on the residual module and channel attention mechanism, such as Figure 3 As shown in Figure 2, the attention mechanism is an important component of neural networks. Its role is to guide the model to focus on the most relevant parts of the input data, thereby improving the efficiency and accuracy of information processing.

[0072] exist Figure 3 In the implementation shown, the input tensor first passes through a 1×1 convolutional layer to modify the number of input channels to the specified number of output channels. The feature map after adjusting the number of channels is average pooled to reduce the spatial dimension and retain important feature information. The feature map after average pooling is sent to two fully connected layers to generate a channel weight vector. This vector is compressed to a real value between 0 and 1 by the Sigmoid function. Finally, the channel weight vector is multiplied element-by-element with the original input tensor to ensure that the output dimension is consistent with the original input. The result of the multiplication is consistent with the Fir e The module output tensors are summed and the result is used as the input of the next module.

[0073] This application uses CNN built based on RFCM module to extract features from image sequences. The convolutional neural network built by RFCM is as follows: Figure 4 Each RFCM is labeled with four parameters. These parameters refer to the number of input channels, the number of filters in the 1×1 convolution in the Squeeze layer, the number of 1×1 convolutions in the Expand layer, and the number of 3×3 convolutions in the Expand layer.

[0074] CNN finally outputs a tensor of size n×512×13×13, where n is the number of images in the sequence. These features fully reflect the spatial information in the image and help to deeply understand the image content. Through multi-layer convolution and pooling operations, CNN can effectively capture feature representations from low-level to high-level. After feature extraction is completed, it is input into the gated recurrent unit (GRU) module. GRU is an improved recurrent neural network that can alleviate the gradient vanishing problem of standard RNN. It uses update gates and reset gates to remember long-term information and simplify the model structure. The overall architecture of GRU is as follows: Figure 5 shown.

[0075] Update Gate:

[0076] z t =σ(W xz x t +W hz h t-1 +b z )

[0077] Among them, [h t-1 , x t ] is the splicing operation, which converts the hidden state h of the previous time step t-1 Connected to the input of the current time step.

[0078] Reset the gate:

[0079] r t =σ(W xr x t +W hr h t-1 +b r )

[0080] New memory content:

[0081]

[0082] is the new memory content, tanh is the hyperbolic tangent activation function, W h and b are the weight and bias of the gate, and ⊙ represents element-by-element multiplication (Hadamard product).

[0083] Hidden state:

[0084]

[0085] Among them, h t is the hidden state at the current time step.

[0086] Since x in the formula t Corresponding to the image feature map input, the image feature map is two-dimensional. In order to better process the two-dimensional image information, it is necessary to use convolution operations to replace the update gate and reset gate parameters in the GRU. In the formula z of the gated recurrent unit GRU t 、r t 、 In the xz\r\t 、W hz\r\t and b z 、b r 、b h And so on. Use convolution instead of vector operation, for W xz\r\t The input channel is the number of channels in the feature map, the output channel is the set number of hidden layer channels (hidden_size), and the padding size is set according to the convolution kernel size. The convolution kernel size and the number of hidden layer channels are variable hyperparameters that are configured at the beginning of model training. The default sizes are 5 and 128.

[0087] W hz\r\t and W xz\r\t Similar, but as a parameter of the hidden layer, the number of input and output channels is hidden_size, and the size of the output is consistent with the size of the input feature map; for b z 、b r and b h The three parameters are initialized to a two-dimensional tensor of all zeros before training begins by calling the function. The size is the same as that of a single feature map, and the number of channels is hidden_size. Finally, the hidden layer h0 is initialized, and its size and number of channels are set to the same as b z 、b r and b h So far, the convolution improvement of the gated recurrent unit GRU has been realized.

[0088] The gated recurrent unit (GRU) ultimately generates a time series descriptor that contains the illumination information in the sequence data. To achieve end-to-end learning, the model uses two convolutional layers to map the high-dimensional GRU descriptor to a three-dimensional chromaticity vector (R, G, B). This chromaticity vector represents the illumination conditions of the image and is a key output for global illumination estimation. Through this mapping, the model can obtain illumination information for image color correction directly from the output of the GRU. This convolution-based GRU implementation method not only improves the computational efficiency of the model, but also enhances the model's adaptability to image features through the introduction of convolution operations. The improved gated recurrent unit (GRU) demonstrates good performance in processing color constancy tasks for sequence images, helping to improve the accuracy of global illumination estimation.

[0089] According to the above content, the original image sequence is read through the preset composite model, and the original image sequence is image corrected and cropped to obtain the image sequence to be processed that meets the preset image standard conditions, so that the image size and quality of the image sequence to be processed meet the requirements of subsequent processing.

[0090] In step S102, a preset transformation process is performed on the key frame images of the image sequence to be processed to obtain a simulated image sequence, and two preset convolutional neural networks are used to extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence.

[0091] Optionally, in some embodiments, a preset transformation is performed on the key frame images of the image sequence to be processed to obtain a simulated image sequence, including: determining the initial cropping position of the target cropped image, and cropping the key frame image according to the initial cropping position; rotating the cropped key frame image by a preset rotation angle, and cropping the rotated key frame image to obtain a square image, scaling the square image to the target pixel, and assembling the scaled square images to obtain a simulated image sequence. The present application proposes a method for translating and scaling the captured frame by simulating the movement of the key frame in the leading frame. The purpose of this simulation is to enhance the model's ability to perceive dynamic changes in the scene, thereby improving the accuracy of color constancy estimation. The specific process is as follows: Figure 6 shown.

[0092] When obtaining a simulated image sequence, first obtain the length n of the image sequence to be processed and the pixel information of each frame of the image, then extract the key frame image, that is, the last frame image, from the image sequence to be processed. Next, perform the following processing on the key frame image:

[0093] Step S1: Calculate the size s of the newly generated square image: take the smaller value of the width (width) and height (height) of the key frame image, multiply it by 0.95, and obtain s.

[0094] Step S2: Determine the initial cropping position of the target cropped image: calculate two random numbers (x, y) in the range [max(width-s, height-s)], which represent the starting positions of the cropped image in the width and height directions respectively.

[0095] Step S3: cropping the image: cropping the key frame image according to the initial cropping position (x, y) determined in the previous step and the newly generated square image size s.

[0096] Step S4: Rotate the key frame image: randomly generate a preset rotation angle, the range of the preset rotation angle is between [-15°, 15°], and rotate the image according to the preset rotation angle.

[0097] Step S5: cropping into a square: cropping the rotated key frame image to obtain a square image.

[0098] Step S6: Scale the image: Scale the cropped and rotated square image to the target pixel size (e.g., 224×224 pixels) to meet the input requirements of the model.

[0099] Through the above scaling and cropping operations, the model can receive transformed key frames, thereby better learning the dynamic changes in the scene and improving the performance of color constancy estimation. The specific effects are as follows: Figure 7 shown.

[0100] Furthermore, a new SqueezeNet architecture in two preset convolutional neural networks is used as a feature extractor, and a first image feature sequence is extracted from the original image sequence and a second image feature sequence is extracted from the simulated image sequence through the new SqueezeNet architecture.

[0101] In step S103, the gated recurrent unit GRU is used to process the first image feature sequence and the second image feature sequence respectively to output a first tensor and a second tensor, and three channels corresponding to the RGB color space are generated based on the spliced ​​tensor of the first tensor and the second tensor to generate light source information for correcting the image color according to the three-channel features.

[0102] Specifically, the gated recurrent unit GRU processes the first image feature sequence and the second image feature sequence obtained by two preset convolutional neural networks CNN. The gated recurrent unit GRU focuses on the temporal information in the learning sequence, captures the dynamic changes in the image sequence, and outputs two tensors of size [1, 128, 13, 13]. That is, the first tensor and the second tensor are both tensors of [1, 128, 13, 13]. The two tensors are spliced ​​to generate a tensor of size [1, 256, 13, 13], which is then sent to a two-layer convolutional network. These convolutional networks are responsible for downsampling the number of feature channels to three channels, corresponding to the three channels of the RGB color space. Finally, the preset composite model generates the final light source information through three channels. This information is used to correct the color of the input image to achieve color constancy.

[0103] In order to enable those skilled in the art to further understand the method for calculating color constancy of sequential images based on convolutional neural networks in the embodiment of the present application, the following is a detailed description with reference to specific embodiments. Figure 8 shown.

[0104] Part 1: Input Data Processing

[0105] The preset composite model first reads the input original image sequence and performs necessary corrections and cropping to ensure that the image size and quality meet the requirements of subsequent processing, thereby obtaining the image sequence to be processed. Next, the preset composite model generates a simulated image sequence based on the image sequence to be processed. By performing transformations such as scaling, rotating, and cropping on the keyframe images, the motion of the keyframe content in the leading frame is simulated. Ultimately, the original image sequence and the simulated image sequence are processed into tensors of size [batch_size, n, channel, height, width], where batch_size is the batch size of each round of data during training, n is the number of images in the sequence, channel is the number of channels in the image, usually three channels of red, green, and blue, and height and width are the width and height of the image, which are set to 224 in this example.

[0106] Part II: Feature Extraction

[0107] The preset composite model uses the new SqueezeNet architecture as a feature extractor to extract the first image feature sequence from the original image sequence and the second image feature sequence from the simulated image sequence. Through this structure, the preset composite model can effectively capture the spatial information in the image sequence. Finally, the two convolutional neural networks (CNNs) convert the first image feature sequence and the second image feature sequence into two tensors of size [batch_size, n, 512, 13, 13], providing rich feature representation for subsequent sequence processing and light source information estimation.

[0108] Part III: Sequence Processing and Light Source Information Estimation

[0109] The gated recurrent unit GRU processes the first image feature sequence and the second image feature sequence obtained by two convolutional neural networks CNN, captures the dynamic changes in the first image feature sequence and the second image feature sequence, and outputs two tensors of size [1, 128, 13, 13]. These two tensors are spliced ​​to generate a tensor of size [1, 256, 13, 13], which is then fed into two layers of convolutional networks. These networks are responsible for downsampling the number of feature channels to three channels, corresponding to the three channels of the RGB color space. The preset composite model generates light source information for correcting the color of the input image through three channels to achieve color constancy.

[0110] Therefore, through this layered processing approach, the model is able to understand the changes in the scene at different time scales, thereby achieving color correction in complex real-world shooting environments.

[0111] According to the method for calculating the color constancy of sequential images based on a convolutional neural network proposed in an embodiment of the present application, a preset composite model is used to read the original image sequence, and the original image sequence is subjected to image correction and cropping to obtain a to-be-processed image sequence that meets the preset image standard conditions. The key frame images of the to-be-processed image sequence are subjected to a preset transformation process to obtain a simulated image sequence. Two preset convolutional neural networks are used to extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence respectively. The gated recurrent unit (GRU) is used to process the first image feature sequence and the second image feature sequence respectively to output a first tensor and a second tensor. Based on the concatenated tensor of the first tensor and the second tensor, three channels corresponding to the RGB color space are generated to generate light source information for correcting the image color based on the three-channel features. Thus, the problem that the traditional color constancy algorithm is difficult to adapt to complex lighting is solved. The model can understand the changes in the scene at different time scales, thereby achieving color correction in complex actual shooting environments.

[0112] Next, a device for calculating color constancy of sequential images based on a convolutional neural network according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0113] Figure 9 4 is a block diagram of a device for calculating color constancy of sequential images based on a convolutional neural network according to an embodiment of the present application.

[0114] like Figure 9 As shown, the device 10 for calculating color constancy of sequential images based on convolutional neural networks includes: a reading module 100, an extraction module 200 and a generation module 300.

[0115] The reading module 100 is configured to read the original image sequence by using a preset composite model, and to obtain a to-be-processed image sequence satisfying a preset image standard condition by performing image correction and cropping on the original image sequence. The preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks. The two preset convolutional neural networks are both SqueezeNet networks constructed by the RFCM module, and the two preset recurrent neural networks are both gated recurrent units (GRUs).

[0116] Optionally, in some embodiments, the extraction module is further configured to determine an initial cropping position of the target cropped image, and crop the key frame image according to the initial cropping position; rotate the cropped key frame image by a preset rotation angle, crop the rotated key frame image to obtain a square image, scale the square image to a target pixel, and splice the scaled square image to obtain the simulation image sequence.

[0117] Optionally, in some embodiments, the RFCM module is as follows:

[0118] X out =ω*CAM(Conv(X in ))+Fire(X in )

[0119] wherein X out is a preset channel number, ω is a weight vector, X in is an input channel number, Conv(X in ) is an input tensor, CAM(Conv(X in )) is a tensor obtained by performing convolution and attention module on X in , and Fire(X in ) is an output tensor of Fire.

[0120] Optionally, in some embodiments, the gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is as follows:

[0121] z t =σ(W xz x t+W hz h t-1 +b z )

[0122] Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters, b z is the door offset;

[0123] Reset gate to:

[0124] r t =σ(W xr x t +W hr h t-1 +b r )

[0125] Among them, r t To reset the door switch, b r is the gate bias, W xr is the tensor x t Parameters, W hr is the tensor h t-1 Parameters.

[0126] It should be noted that the above explanation of the embodiment of the method for calculating the color constancy of sequential images based on a convolutional neural network is also applicable to the device for calculating the color constancy of sequential images based on a convolutional neural network in this embodiment, and will not be repeated here.

[0127] According to the convolutional neural network-based sequential image color constancy calculation device proposed in the embodiment of the present application, a preset composite model is used to read the original image sequence, and the original image sequence is subjected to image correction and cropping to obtain a to-be-processed image sequence that meets the preset image standard conditions. The key frame images of the to-be-processed image sequence are subjected to a preset transformation process to obtain a simulated image sequence. Two preset convolutional neural networks are used to extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence respectively. The gated recurrent unit GRU is used to process the first image feature sequence and the second image feature sequence respectively to output a first tensor and a second tensor. Based on the concatenated tensor of the first tensor and the second tensor, three channels corresponding to the RGB color space are generated to generate light source information for correcting the image color based on the three-channel features. Thus, the problem that the traditional color constancy algorithm is difficult to adapt to complex lighting is solved. The model can understand the changes in the scene at different time scales, thereby achieving color correction in complex actual shooting environments.

[0128] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0129] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .

[0130] When the processor 1002 executes the program, the method for calculating the color constancy of sequential images based on a convolutional neural network provided in the above embodiment is implemented.

[0131] Furthermore, the electronic device further includes:

[0132] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0133] The memory 1001 is used to store computer programs that can be run on the processor 1002 .

[0134] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0135] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0136] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0137] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0138] An embodiment of the present application also provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for calculating the color constancy of sequential images based on a convolutional neural network.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0141] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0142] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer program product for use with, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer program product" can be any device that can contain, store, communicate, propagate, or transmit a program for use with, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer program products include the following: an electrical connection having one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer program product may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or, if necessary, processing it in another suitable manner, and then storing it in a computer memory.

[0143] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer program product, which, when executed, includes one or a combination of the steps of the method embodiment.

[0145] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer program product.

[0146] The computer program product mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for calculating color constancy of sequential images based on convolutional neural networks, characterized in that: The following steps are involved: Using a preset composite model to read an original image sequence, and performing image correction and cropping on the original image sequence to obtain a to-be-processed image sequence that meets preset image standard conditions, wherein the preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks, the two preset convolutional neural networks are both SqueezeNet networks constructed by RFCM modules, and the two preset recurrent neural networks are both gated recurrent units (GRUs); Performing a preset transformation process on the key frame images of the image sequence to be processed to obtain a simulated image sequence, and using the two preset convolutional neural networks to respectively extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence; The gated recurrent unit GRU is used to process the first image feature sequence and the second image feature sequence respectively to output a first tensor and a second tensor, and three channels corresponding to the RGB color space are generated based on the spliced ​​tensor of the first tensor and the second tensor to generate light source information for correcting the image color according to the three-channel features.

2. The method according to claim 1, characterized in that The step of performing a preset transformation on the key frame images of the image sequence to be processed to obtain a simulated image sequence includes: Determining an initial cropping position of a target cropping image, and cropping the key frame image according to the initial cropping position; The cropped key frame image is rotated by a preset rotation angle, and the rotated key frame image is cropped to obtain a square image. The square image is scaled to a target pixel, and the scaled square images are assembled to obtain the simulated image sequence.

3. The method according to claim 1, characterized in that The RFCM module is: X out =ω*CAM(Conv(X in ))+Fire(X in ) Among them, X out is the preset number of channels, ω is the weight vector, X in is the number of input channels, Conv(X in ) is the input tensor, CAM(Conv(X in )) is X in The tensor obtained by convolution and attention module, Fire(X in ) is the output tensor of Fire.

4. The method according to claim 1, wherein The gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is: z t =σ(W xz x t +W hz h t-1 +b z ) Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t The parameters learned in the neural network, W hz is the tensor h t-1 The parameters learned in the neural network, b z is the door offset; The reset gate is: r t =σ(W xr x t +W hr h t-1 +b r ) Among them, r t To reset the door switch, b r is the gate bias, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters.

5. A device for calculating color constancy of sequential images based on convolutional neural networks, characterized in that: include: A reading module, configured to read an original image sequence using a preset composite model, and perform image correction and cropping on the original image sequence to obtain an image sequence to be processed that meets preset image standard conditions, wherein the preset composite model includes two preset convolutional neural networks and two preset recurrent neural networks, the two preset convolutional neural networks are both SqueezeNet networks constructed by the RFCM module, and the two preset recurrent neural networks are both gated recurrent units (GRUs); an extraction module, configured to perform a preset transformation on the key frame images of the image sequence to be processed to obtain a simulated image sequence, and to respectively extract a first image feature sequence from the original image sequence and a second image feature sequence from the simulated image sequence using the two preset convolutional neural networks; A generation module is used to use the gated recurrent unit GRU to process the first image feature sequence and the second image feature sequence respectively to output a first tensor and a second tensor, and generate three channels corresponding to the RGB color space based on the spliced ​​tensor of the first tensor and the second tensor, so as to generate light source information for correcting the image color according to the three-channel features.

6. The device according to claim 5, characterized in that The extraction module is further used to: Determining an initial cropping position of a target cropping image, and cropping the key frame image according to the initial cropping position; The cropped key frame image is rotated by a preset rotation angle, and the rotated key frame image is cropped to obtain a square image. The square image is scaled to a target pixel, and the scaled square images are assembled to obtain the simulated image sequence.

7. The device according to claim 5, characterized in that The RFCM module is: X out =ω*CAM(Conv(X in ))+Fire(X in ) Among them, X out is the preset number of channels, ω is the weight vector, X in is the number of input channels, Conv(X in ) is the input tensor, CAM(Conv(X in )) is X in The tensor obtained by convolution and attention module, Fire(X in ) is the output tensor of Fire.

8. The device according to claim 5, characterized in that The gated recurrent unit GRU includes an update gate and a reset gate, wherein the update gate is: z t =σ(W xz x t +W hz h t-1 +b z ) Among them, z t is the switch of the update gate, σ is the Sigmoid activation function, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, W xz is the tensor x t Parameters, W hz is the tensor h t-1 Parameters, b z is the door offset; The reset gate is: r t =σ(W xr x t +W hr h t-1 +b r ) Among them, r t To reset the door switch, b r is the gate bias, W xr is the tensor x t Parameters, W hr is the tensor h t-1 Parameters.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for calculating color constancy of sequential images based on a convolutional neural network as described in any one of claims 1 to 4.

10. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the program is executed by a processor, the method for calculating color constancy of sequential images based on a convolutional neural network as described in any one of claims 1 to 4 is implemented.

Images