Industrial camera color constancy correction device system based on image cluster block expansion convolution

Abstract

The invention discloses an industrial camera color constancy correction device system based on image cluster block expansion convolution. The industrial camera color constancy correction device system comprises a PC end module and a camera end module. The PC end module completes training and optimization of an illumination model, and the camera end module realizes real-time reasoning and white balance correction through an embedded hardware FPGA (Field Programmable Gate Array); the construction method for generating the lightweight illumination estimation training model by the PC end module specifically comprises the following steps: step 1, expanding a data set; 2, constructing a sectional type multi-scale expansion convolutional network model; step 3, training the expanded data set obtained in the step 1 by using the segmented multi-scale expansion convolutional network model constructed in the step 2, and iteratively training parameters; color constancy data analysis dimensions are added, image data are analyzed from two dimensions of expanding a data set and constructing a lightweight illumination estimation training model, the average angle error in an industrial scene is reduced to 1.52 degrees, the frame rate is larger than or equal to 30 fps, power consumption is smaller than 2 W, and the method has the characteristics of high precision and low delay.

Description

Technical Field

[0001] The present invention relates to the technical field of image color correction, and in particular to a color constancy correction device system for industrial cameras based on image cluster block dilation convolution. Background Art

[0002] Industrial cameras are fundamental sensing devices for intelligent manufacturing and are widely used throughout the production process. White balance processing capability and color constancy are crucial parameters for industrial camera post-processing visual inspection. The camera converts the raw RGB sensor response into a final output image through multiple processing steps. Typically, in industrial camera front-end processing, scene illumination is assumed to be global and uniform, meaning that there is only a single light source in the scene. Statistical or learning methods are used to estimate the global illumination color as closely as possible and eliminate the resulting image color deviations, thereby achieving white balance correction.

[0003] In practical applications, white balance and image constancy processing in industrial cameras are constrained by a variety of factors. The most common issue is complex and variable scene lighting, which leads to large correction errors due to single-scale feature extraction. With the success of machine learning in image processing, learning-based methods are widely used in camera front-end processing due to their high accuracy. However, these methods require large, high-quality datasets, and camera production processes struggle to collect data with a balanced distribution of labeled knowledge to ensure dataset quality. Furthermore, the large number of machine learning model parameters currently available makes it difficult to achieve both high accuracy and real-time performance, severely impacting the performance of industrial cameras in actual production management.

[0004] Currently, the methods for processing white balance of the front-end image mainly include statistical methods and learning methods.

[0005] Statistical methods, such as the automatic white balance method and device described in patent CN200580026185.2, analyze the distribution of image color and brightness information to achieve white balance. This method assumes that the average value of all pixels in the image approaches gray and adjusts the gains of the RGB channels to achieve white balance. This method is simple but relies on the uniformity of scene color distribution and is susceptible to interference from large areas of monochrome.

[0006] Learning-based methods learn features from images through artificial design, thereby realizing image light source prediction. Feature learning network design and training data set construction are the core work of the learning method. Network generalization ability and learning efficiency are the main indicators based on the learning method. For example, patent CN118574015A, an image white balance processing method, device and image sensing device, divides the image into multiple blocks, and screens effective blocks based on color temperature judgment conditions to avoid interference from large areas of monochrome areas. This method improves robustness in complex scenes, but the block division strategy may miss key information. CN115938282B, an automatic white balance configuration method, device and system with machine learning, combines deep learning or traditional machine learning algorithms, uses historical data to optimize white balance parameter configuration, and gradually improves calibration accuracy. This method relies on the accumulation of historical data, has low initial efficiency, and has high model complexity.

[0007] The main problems of existing technologies include: dataset limitations: uneven label distribution and lack of complex lighting scene data; model complexity: the deep learning network has a large number of parameters and high deployment costs; insufficient real-time performance: traditional methods have high processing delays and cannot support real-time processing at 4K resolution. Summary of the Invention

[0008] The purpose of the present invention is to address the deficiencies of the prior art and provide an industrial camera color constancy correction device system based on image cluster block dilation convolution.

[0009] The technical solution of the present invention to solve the above problems is: an industrial camera color constancy correction device system based on image cluster block dilation convolution, including a PC end module and a camera end module;

[0010] The PC module completes the training and optimization of the lighting model, specifically by using high-performance GPU computing resources to train a multi-scale dilated convolutional network to generate a lightweight lighting estimation training model; quantizing the training model to 8-bit fixed-point numbers to generate an FPGA-compatible model IP core;

[0011] The camera-side module's embedded hardware FPGA implements real-time inference and white balance correction, specifically by integrating the model IP core into the FPGA to enable real-time RAW image block processing, lighting prediction, and correction; updating model parameters via OTA to adapt to new scene lighting conditions; and, at the same time, expanding the labeled images to the PC-side dataset for incremental training and optimized iterative models.

[0012] Furthermore, the PC module generates a lightweight illumination estimation training model, which specifically includes the following steps:

[0013] Step 1, expand the dataset;

[0014] Step 2: Construct a segmented multi-scale dilated convolutional network model;

[0015] Step 3: Use the extended dataset obtained in step 1 to train the segmented multi-scale dilated convolutional network model constructed in step 2, and iterate the training parameters.

[0016] Furthermore, step 1 specifically includes the following steps:

[0017] Step 101, input processing: removing the initial light source label from the original RAW image to generate a neutral image after white balance;

[0018] Step 102, constructing a random illumination matrix: randomly generating an RGB triplet illumination matrix, with the illumination color gamut covering normalized values ​​from 0.013 to 0.923;

[0019] Step 103, data expansion: For each neutral image, use the random illumination matrix constructed in step 2 to generate five random illumination images;

[0020] Step 104, cluster block processing: Each random illumination image generated in step 103 is resized and then cut into 16 local blocks with an overlap of ≥ 30%. The upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) are randomly set to ensure full image coverage. All local blocks share the illumination label of the original image, forming a mapping relationship of "1 original image → 16 cluster blocks".

[0021] Furthermore, the multi-scale dilated convolutional network model includes a feature network and a fusion network;

[0022] The feature network consists of five convolutional layers. The first four convolutional layers all use a 3×3 convolution kernel + ReLU activation function, and the stride of the convolution block is 2. A batch normalization layer is set between the third and fourth convolutional layers. In the last convolutional layer, a 7×7 maximum pooling layer is used with zero padding to obtain a 1×1×3 result, which represents the lighting characteristics of a block in the image.

[0023] The fusion network contains a fully connected layer, the input is a 48-dimensional feature vector, which is a combination of the features of 16 blocks in the image, and the output is a global illumination estimate.

[0024] The present invention has the beneficial effects:

[0025] The present invention provides an industrial camera color constancy correction device system based on image cluster block dilation convolution, which increases the dimension of color constancy data analysis and analyzes image data from two dimensions: expanding the data set and building a lightweight illumination estimation training model. By removing the initial color and adding new random illumination to optimize the original data set, not only the scale of the data set is appropriately expanded, but also the color gamut of the data set is expanded, and the amount of knowledge of the data set is expanded, thereby improving the generalization ability of the network; the image is randomly cut into small blocks, all of which correspond to the same color label, and are used as new data sets to establish a wider color space. These local small image blocks will greatly contain the underlying color information, which will support simple neural networks to perform effective feature extraction. In industrial scenarios, the present invention reduces the average angle error to 1.52°, the frame rate is ≥30fps, the power consumption is <2W, and it has high precision and low latency characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the hardware structure of the PC end module and the camera end module of the present invention.

[0027] Figure 2 Detailed flowchart for expanding the dataset during the construction of a lightweight illumination estimation training model. DETAILED DESCRIPTION

[0028] like Figure 1 As shown, an industrial camera color constancy correction device system based on image cluster block dilation convolution includes a PC end module and a camera end module.

[0029] The PC-side module completes the training and optimization of the lighting model, specifically including: using high-performance GPU computing resources to train a multi-scale dilated convolutional network to generate a lightweight lighting estimation training model; quantizing the training model into 8-bit fixed-point numbers to generate a hardware description file adapted to the FPGA, namely the model IP core.

[0030] The camera-side module's embedded hardware FPGA implements real-time inference and white balance correction, specifically by integrating the model IP core into the FPGA to enable real-time RAW image block processing, lighting prediction, and correction; updating model parameters via OTA to adapt to new scene lighting conditions; and, at the same time, expanding the labeled images to the PC-side dataset for incremental training and optimized iterative models.

[0031] like Figure 2 As shown, the method for constructing a lightweight illumination estimation training model generated by the PC module specifically includes the following steps:

[0032] Step 1, expand the dataset, specifically includes the following steps:

[0033] Step 101, input processing: remove the initial light source label from the original RAW image (such as the Color Checker dataset) to generate a neutral image after white balance;

[0034] Step 102, constructing a random illumination matrix: randomly generating an RGB triplet illumination matrix, with the illumination color gamut ranging from 0.013 to 0.923 normalized values, to ensure the color gamut diversity of the extended dataset.

[0035] Step 103, data expansion: For each neutral image, use the random illumination matrix constructed in step 2 to generate five random illumination images, expanding the dataset size by five times.

[0036] Step 104: Cluster processing: Each randomly illuminated image generated in step 103 is resized to 2041×1359 and then divided into 16 224×224 local blocks. The overlap between local blocks is ≥30%. The top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2) are randomly set to ensure full image coverage, where x1∈[0,1817], y1∈[0,1135], x2∈[224,2041], and y2∈[224,1359]. All local blocks share the illumination label of the original image, forming a "1 original image → 16 cluster blocks" mapping. This overlapping clustering method allows for as much underlying information as possible to be incorporated into network training, compensating for the dataset's vulnerability to overfitting. The overall dataset size is expanded to 16×5 times the original size.

[0037] Step 2: Construct a segmented multi-scale dilated convolutional network model, which includes two independent feature networks and a fusion network.

[0038] The feature network consists of five convolutional layers. The first four layers use a 3×3 convolution kernel with a ReLU activation function, and the stride of the convolution block is 2. A batch normalization layer is placed between the third and fourth convolutional layers. In the last convolutional layer, a 7×7 max pooling layer with zero padding is used to obtain a 1×1×3 result, which represents the lighting characteristics of a block in the image.

[0039] The fusion network contains a fully connected layer, the input is a 48-dimensional feature vector, which is a combination of the features of 16 blocks in the image, and the output is a global illumination estimate.

[0040] Multi-scale dilated convolutional network model, the network parameters of each layer are shown in Table 1.

[0041] Table 1

[0042] Hierarchy Input size Output size Convolution kernel Expansion rate Activation Function Input layer 224×224×3 112×112×9 3×3 1 ReLU Dilated convolution group 1 112×112×9 112×112×32 3×3 2 ReLU Dilated convolution group 2 112×112×32 112×112×32 3×3 4 ReLU Dilated convolution group 3 112×112×32 112×112×32 3×3 8 ReLU Feature concatenation layer 112×112×32×3 112×112×96 - - - Fully connected fusion layer 16×96 3 - - Linear

[0043] The dilated convolution groups 1-3 use dilated convolution layers with expansion rates of 2, 4, and 8 in parallel to capture short-range, medium-range, and long-range illumination features, respectively. The multi-scale information is then fused through the feature concatenation layer to generate a 96-dimensional feature vector. The fully connected fusion layer inputs the 96-dimensional feature vector of 16 blocks, learns the weight matrix through the fully connected layer, and outputs the weighted illumination estimation value of the three color channels of the world [R es ,G es ,B es ].

[0044] Step 3: Use the extended dataset obtained in step 1 to train the segmented multi-scale dilated convolutional network model constructed in step 2, and iterate the training parameters.

[0045] Training parameters include the optimizer and the loss function.

[0046] Optimizer: Adam, learning rate set to 7×10 -4 , decayed by 0.95 every 200 rounds, the batch size was set to 32, and the training was performed with 400,000 Adam optimization algorithms.

[0047] Loss function:

[0048] Where I ori Indicates the real light source of the tag, I es Represents the network's results. This loss function is used for iterative training to optimize the convolutional neural network by reducing the error between the true value and the network output. Each round of training uses three-fold cross-validation, with the dataset split into training, validation, and test sets in an 85%:5%:10% ratio.

[0049] Example 1: Using the Gehler dataset (568 images) and expanding it to 2840 images, on the NUS dataset, the average angular error was reduced from 2.36° to 1.52°, improving generalization by 35.6%. In highly reflective scenes, the error fluctuation was less than 0.5°.

[0050] The lightweight illumination estimation training model was quantized to 8-bit fixed-point numbers, and a hardware IP core was generated using Vivado HLS. This was then deployed to an FPGA. Processing a single 4K image (3840×2160) took 9.8ms, with a frame rate of ≥30fps.

[0051] Comparative Example 1: The difference from Example 1 is that a simple regression model is used to process the estimated illumination color of the image block, with an average angle error of 1.77°, and the average angle error of Example 1 is reduced by 14.1%.

[0052] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any person skilled in the art may utilize the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes for application in other fields. However, any simple modification, equivalent change, and modification of the above embodiments made in accordance with the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A color constancy correction device system for industrial cameras based on image cluster block dilation convolution, characterized by: Including PC end module and camera end module; The PC module completes the training and optimization of the lighting model, specifically by using high-performance GPU computing resources to train a multi-scale dilated convolutional network to generate a lightweight lighting estimation training model; quantizing the training model to 8-bit fixed-point numbers to generate an FPGA-compatible model IP core; The camera-side module's embedded hardware FPGA implements real-time inference and white balance correction, specifically by integrating the model IP core into the FPGA to enable real-time RAW image block processing, lighting prediction, and correction; updating model parameters via OTA to adapt to new scene lighting conditions; and, at the same time, expanding the labeled images to the PC-side dataset for incremental training and optimized iterative models.

2. The industrial camera color constancy correction device system based on image cluster block dilation convolution according to claim 1, characterized in that: The PC module generates a lightweight illumination estimation training model, which specifically includes the following steps: Step 1, expand the dataset; Step 2: Construct a segmented multi-scale dilated convolutional network model; Step 3: Use the extended dataset obtained in step 1 to train the segmented multi-scale dilated convolutional network model constructed in step 2, and iterate the training parameters.

3. The industrial camera color constancy correction device system based on image cluster block dilation convolution according to claim 2, characterized in that: Step 1 specifically includes the following steps: Step 101, input processing: removing the initial light source label from the original RAW image to generate a neutral image after white balance; Step 102, constructing a random illumination matrix: randomly generating an RGB triplet illumination matrix, with the illumination color gamut covering normalized values ​​from 0.013 to 0.923; Step 103, data expansion: For each neutral image, use the random illumination matrix constructed in step 2 to generate five random illumination images; Step 104, cluster block processing: Each random illumination image generated in step 103 is resized and then divided into 16 local blocks. The overlap between local blocks is ≥ 30%. The upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) are randomly set to ensure full image coverage. All local blocks share the illumination label of the original image, forming a "1 original image → 16 cluster blocks" mapping relationship.

4. The industrial camera color constancy correction device system based on image cluster block dilation convolution according to claim 2, characterized in that: The multi-scale dilated convolutional network model includes a feature network and a fusion network; The feature network consists of five convolutional layers. The first four convolutional layers all use a 3×3 convolution kernel + ReLU activation function, and the stride of the convolution block is 2. A batch normalization layer is set between the third and fourth convolutional layers. In the last convolutional layer, a 7×7 maximum pooling layer is used with zero padding to obtain a 1×1×3 result, which represents the lighting characteristics of a block in the image. The fusion network contains a fully connected layer, the input is a 48-dimensional feature vector, which is a combination of the features of 16 blocks in the image, and the output is a global illumination estimate.

Images