An image processing method and system based on near-infrared imaging

Abstract

The application discloses an image processing method and system based on near-infrared imaging, and particularly relates to the field of image data processing, which comprises the following steps: collecting continuous near-infrared image frames in the running process of a production line, and extracting corresponding pixel sets; obtaining pixel gray values of all image frames, and establishing a pixel gray distribution map; using a multi-branch convolutional neural network to learn spatial features of the distribution map, and separating background, target structure and dynamic residual image areas; performing spatial alignment on the image frames with the target structure area as the center, and superimposing to generate a global area distribution; then constructing a training set through historical image samples, and completing neural network training of an area distribution prediction model; using the model to predict the background and residual image area distribution of real-time images, and completing dynamic residual image correction through neighborhood repair, so that dynamic residual image fast fading processing of near-infrared images of a high-speed production line is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and more particularly, to an image processing method and system based on near-infrared imaging. BACKGROUND

[0002] The near-infrared image processing technology is widely used in defect detection, size measurement and surface quality analysis of continuously moving targets on industrial production lines. However, in a high-speed production environment, due to the rapid movement of the target and the integral characteristics of the image acquisition system, dynamic ghosting often occurs in the near-infrared image sequence. This phenomenon is manifested as the position superposition of the target edge or structure in multiple images, resulting in "smearing" or "ghost" residual signals in the image, which seriously affects the accuracy of defect detection. The existing image denoising or background modeling technology mainly faces static images or uniform motion blur, and it is difficult to effectively deal with the time domain accumulation effect and multi-frame dependence characteristics of dynamic ghosting, especially in high frame rate and pipeline detection scenarios. The traditional algorithm often has high computational complexity and poor real-time performance, and is easy to cause the loss of target structure information, so it is difficult to apply to specific production scenarios.

[0003] Therefore, there is an urgent need for a dynamic ghosting fast fading processing for near-infrared images on high-speed production lines, which can accurately separate dynamic ghosting from real target structure, avoid detection errors caused by ghosting interference, and improve the stability of target detection and the real-time performance of the system. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide an image processing method based on near-infrared imaging to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0006] An image processing method based on near-infrared imaging, comprising the following steps:

[0007] S1: based on the running speed of the production line, continuously acquiring near-infrared images of the production area to generate a pixel set of near-infrared image frames;

[0008] S2: obtaining the pixel gray value of the pixel set corresponding to all image frames, and establishing a pixel gray distribution map;

[0009] S3: using a multi-branch convolutional neural network to learn the spatial features of the pixel gray distribution map, and separating out the background, target structure and dynamic ghosting region distribution;

[0010] S4: taking the target structure region as the center, performing spatial alignment on the continuous image frames, and superimposing all image frames to generate a global region distribution;

[0011] S5: Obtain historical near-infrared image frame samples, train a neural network model through a training set constructed by the samples, and establish a region distribution prediction model;

[0012] S6: Predict the background and residual image region distribution of a real-time image based on the region distribution prediction model, perform neighborhood repair on the residual image region through the background region, and output the residual image after correction.

[0013] In a preferred embodiment, in S1, continuous near-infrared image acquisition is performed on the production region based on the running speed of the production line, and the pixel set of the near-infrared image frame specifically includes:

[0014] The real-time running speed of the production line is obtained, and the near-infrared image acquisition frame rate is dynamically adjusted based on a preset fixed total frame number;

[0015] The external light source parameters are obtained, continuous near-infrared image frame acquisition is performed on the target production region on the production line, the acquisition time stamp of each image frame is recorded, and an interpolation method is used to perform time axis uniform processing on the image frames;

[0016] The pixel numbering and row and column position calibration of the image frames corresponding to each time sequence index position on the time axis are performed, and the pixel set of each image frame is generated.

[0017] In a preferred embodiment, in S2, the pixel gray value of the pixel set corresponding to all image frames is obtained, and the pixel gray distribution map is established, specifically including:

[0018] In the pixel set corresponding to the image frame, the optical signal collected at each pixel row and column position is converted into an electronic signal to generate a corresponding pixel gray signal, and the pixel gray signal is quantized into a digital gray value;

[0019] The digital gray value is converted into a two-dimensional gray matrix based on the arrangement of the row and column positions of the pixels, and the two-dimensional gray matrix is marked as a pixel gray distribution map;

[0020] The continuous image frame sequence is read and indexed labeled, and the corresponding relationship between the pixel gray distribution map and its belonging image frame number is established.

[0021] In a preferred embodiment, in S3, the pixel gray distribution map is subjected to spatial feature learning by using a multi-branch convolutional neural network, and the background, target structure and dynamic residual image region distribution are separated, specifically including:

[0022] The pixel gray distribution map is scanned pixel by pixel, and the pixel gray values are arranged in row and column order to form a multi-dimensional input tensor;

[0023] The multi-branch convolutional neural network structure is set, a single-channel input format is reserved in a multi-dimensional input tensor, and a convolution kernel of a corresponding size is configured for each branch;

[0024] The multi-branch includes a background branch, a target structure branch and a dynamic residual branch;

[0025] The multi-dimensional input tensor is input into the multi-branch convolutional neural network, and a local convolution operation is performed in each branch based on the configured convolution kernel to extract pixel spatial features and construct feature maps of different branches, wherein the spatial features include texture features, edge features and continuous region features;

[0026] The feature maps of different branches are fused across branches, and the background, target structure and dynamic residual region distribution are separated through feature clustering.

[0027] In a preferred embodiment, the feature maps of different branches are fused across branches, and the background, target structure and dynamic residual region distribution are separated through feature clustering.

[0028] In the output of the multi-branch convolutional neural network, the feature maps of the background branch, the target structure branch and the dynamic residual branch are read in sequence;

[0029] The three types of feature maps are superimposed to construct a joint feature based on the background feature value, the target structure feature value and the dynamic residual feature value at each pixel position, forming a joint feature set at the pixel level;

[0030] A joint feature classification template of the background, the target structure and the dynamic residual region is preset, and a corresponding class index is established;

[0031] Unsupervised clustering is performed on the joint feature set in the entire pixel gray distribution map, a local joint feature set of each class cluster is extracted, an average feature of the local joint feature set is calculated, and the class index of each clustering cluster is labeled by the distance between the average feature and the joint feature classification template;

[0032] The labeled class index is mapped to the row and column positions of the pixel gray distribution map to generate the background region, the target structure region and the dynamic residual region distribution corresponding to a single image frame.

[0033] In a preferred embodiment, in S4, the spatial alignment is performed on the continuous image frames with the target structure region as the center, and all image frames are superimposed to generate a global region distribution, which specifically includes:

[0034] The target structure region is taken as the center position of the pixel distribution map, and the spatial alignment is performed on the pixel distribution maps corresponding to all image frames;

[0035] Based on the pixel row and column position correspondence relationship of the spatial aligned background region, target structure region and dynamic residual image region distribution, all image frames are cross-frame superimposed to generate global background region, global target structure region and global dynamic residual image region distribution.

[0036] In a preferred embodiment, in S5, historical near-infrared image frame samples are obtained, a neural network model is trained by a training set constructed by the samples, and the establishment of the region distribution prediction model specifically includes:

[0037] Historical near-infrared image frame sample data is obtained, and each set of sample data covers different external light source parameters and production line running speeds;

[0038] In each set of sample data, the pixel gray scale distribution map of all historical near-infrared image frames is determined for global background region, global target structure region and global dynamic residual image region distribution;

[0039] Based on the global background region, global target structure region and global dynamic residual image region distribution determination result, the sample data is executed for label marking processing, and the training set data of the region distribution prediction model is constructed;

[0040] A neural network model is selected, the training set data is input to train the region distribution prediction model, and after the model training is completed, the model is deployed to the production line end.

[0041] In a preferred embodiment, in S6, based on the region distribution prediction model, the background and residual image region distribution of the real-time image is predicted, the residual image region is repaired by the background region, and the image after residual image correction is output, specifically including:

[0042] The external light source parameters and real-time production line running speed of the production line end are input to the region distribution prediction model;

[0043] The model output result is obtained, and the pixel gray scale distribution map of the real-time collected near-infrared image frame is marked for background region and dynamic residual image region distribution;

[0044] The pixel position coordinates of the marked residual image region and background region are extracted in the pixel gray scale distribution map;

[0045] The pixel gray scale value of the pixel gray scale distribution map is repaired on the pixel position coordinates of the residual image region, the pixel gray scale of the residual image region is corrected according to the pixel gray scale value of the background region, and the image frame after residual image correction is output.

[0046] On the other hand, the application provides an image processing method system based on near-infrared imaging, including a gray value construction module, a convolution feature extraction module, a global distribution construction module, a prediction model training module and a dynamic residual image correction module:

[0047] Grayscale value construction module: Collects continuous near-infrared image frame samples during production line operation, dynamically adjusts the image frame rate in combination with real-time production line speed and external light source parameters, performs electronic conversion and quantization processing on the pixel light signals of the image frames, generates digital grayscale values ​​and marks the row and column positions, and constructs a pixel grayscale distribution map;

[0048] Convolutional feature extraction module: It uses a multi-branch convolutional neural network to perform spatial feature learning on the pixel grayscale distribution map, configures convolution kernels according to background, target structure and dynamic afterimage branches and extracts spatial features, generates branch feature maps, and assigns category indexes through feature fusion and unsupervised clustering, outputting the pixel region distribution of background, target structure and dynamic afterimage;

[0049] Global distribution construction module: Using the target structure region as an alignment reference, it performs spatial alignment on consecutive image frames. Based on the correspondence of pixel row and column positions, it overlays the background, target structure, and dynamic afterimage region distributions across frames to generate global background region, global target structure region, and global dynamic afterimage region distribution.

[0050] Prediction model training module: acquires historical near-infrared image frame sample data and corresponding global region distribution map, performs labeling processing on the sample data to construct a training set, selects a neural network model as input to the training set for training, and generates a region distribution prediction model after training and deploys it to the production line.

[0051] Dynamic image retention correction module: Input the external light source parameters and real-time production line running speed to the regional distribution prediction model, obtain the background and dynamic image retention region distribution, perform neighborhood repair on the image retention region, use the pixel grayscale values ​​of the background region to correct the image retention region, and output the corrected image frame.

[0052] The technical effects and advantages of the image processing method and system based on near-infrared imaging of this invention are as follows:

[0053] For real-time near-infrared image acquisition data from the production line, a multi-branch convolutional neural network is used to learn spatial features of the pixel grayscale distribution map. This effectively separates the background, target structure, and dynamic afterimage regions, enhancing the spatial resolution of dynamic afterimage areas. A spatial alignment strategy centered on the target structure region solves the problem of region distribution misalignment caused by target drift in consecutive image frames, improving the accuracy of multi-frame afterimage overlay analysis. A training set and region distribution prediction model constructed using historical sample data, combined with real-time light source parameters and production line operating speed, achieves the ability to predict dynamic afterimage distribution, significantly improving the model's adaptability to dynamic afterimage interference under various operating conditions. A neighborhood repair mechanism for afterimage regions effectively reduces the impact of dynamic afterimage interference on image quality, ensuring the image stability and accuracy of production line visual inspection and subsequent processes. Attached Figure Description

[0054] Figure 1 A schematic diagram of an image processing method based on near-infrared imaging according to the present application;

[0055] Figure 2 A structural schematic diagram of an image processing system based on near-infrared imaging according to the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0057] Embodiment 1, Figure 1 An image processing method based on near-infrared imaging according to the present application is given, which comprises the following steps:

[0058] S1: Continuously collecting near-infrared images of the production area based on the running speed of the production line to generate a pixel set of near-infrared image frames;

[0059] S2: Obtaining pixel gray value of the pixel set corresponding to all image frames to establish a pixel gray distribution map;

[0060] S3: Using a multi-branch convolutional neural network to learn spatial features of the pixel gray distribution map to separate the background, target structure and dynamic ghost region distribution;

[0061] S4: Taking the target structure region as the center, performing spatial alignment on the continuous image frames to generate a global region distribution by superimposing all image frames;

[0062] S5: Obtaining historical near-infrared image frame samples, training the neural network model through a training set constructed by the samples, and establishing a region distribution prediction model;

[0063] S6: Predicting the background and ghost region distribution of the real-time image based on the region distribution prediction model, performing neighborhood repair on the ghost region through the background region, and outputting the image after ghost correction.

[0064] In S1, continuously collecting near-infrared images of the production area based on the running speed of the production line to generate a pixel set of near-infrared image frames.

[0065] Real-time running speed information is acquired on the production line. The running speed of the production line is described by the length of the target area (such as a fixed tray, a fixed device, a specific station on the material conveying line, or an assembly area) passing through per unit time, with units of meters per second or millimeters per second. The preset fixed total frame number is set to 100 frames by default. It is necessary to ensure the complete image acquisition period of the target area. This value is determined before the image acquisition process is started and is stored in the parameter configuration, and does not change with the running speed. The setting of the fixed total frame number supports adjustment according to actual needs, for example, it can be adjusted to 50 frames, 150 frames, or 300 frames in special process scenarios. According to the length of the target area and the real-time running speed, the target area passing time is calculated, and the passing time is divided by the fixed total frame number to obtain the frame rate value per second. This frame rate value is used to dynamically set the time interval in the image acquisition process, realizing dynamic synchronous acquisition of image frames. The frame rate calculation method is: the length of the target area (unit: meters) is divided by the real-time running speed (meters per second) to obtain the target area passing time (seconds), and then the target area passing time is divided by the fixed total frame number to calculate the time interval (seconds) of each frame acquisition. Finally, the reciprocal is taken to obtain the acquisition frame rate (frames per second). For example, in the case of a target area length of 2 meters and a real-time running speed of 0.2 meters per second, the target area passing time is 10 seconds, and if the fixed total frame number is set to 100 frames, the acquisition frame rate is 10 frames per second. This calculation method ensures that the frame acquisition time interval in the target area is uniform, thereby making the subsequent pixel gray value extraction step have high time axis consistency.

[0066] The external light source parameters of the target production area are collected and input, including light source intensity, color temperature, and illumination angle. The default light source intensity setting value is 500lx, the default color temperature setting value is 5000K, and the default illumination angle setting value is 45°. The above values are configured before the image acquisition process is started, and can be adjusted according to different natural light effects. The specific value range can be referred to: light source intensity 300lx to 1000lx, color temperature 3500K to 6000K, and illumination angle 30° to 60°. Each image acquisition refers to the external light source parameters. For the target production area, continuous near-infrared image frame acquisition is performed. Each frame of image is labeled with a unique timestamp during the acquisition process. The timestamp uses a unified format of year, month, day, hour, minute, second, and millisecond, and is directly attached to the header or data structure of each frame of image file, ensuring the integrity and traceability of the acquisition order.

[0067] After the acquisition of the continuous image frames is completed, image frame uniformization processing is performed for the possible inter-frame time interval unevenness. In the time axis index, the inter-frame interval is detected, and when the inter-frame acquisition interval exceeds a preset threshold, for example, exceeds 1000 milliseconds (the threshold can be set according to the needs during the production line debugging stage and is noted in the parameter document), time axis uniformization processing is performed. The time axis uniformization processing method uses an interpolation algorithm to correct the time index between the frames, so that the image frame sampling time distribution is uniform. The specific interpolation method uses a previous frame image replication method, which directly replicates the previous frame image to the vacant time index position, to ensure the continuity of the time axis and the integrity of the data. After the time axis uniformization processing is completed, the pixels in each frame image in the two-dimensional space are numbered and labeled according to the row and column positions. Each pixel has a unique row and column index, and the pixel number is arranged in a row-by-row incremental manner, indexed from left to right and from top to bottom, and combined with the row and column positions to construct a pixel set.

[0068] In S2, the pixel gray scale values of the pixel set corresponding to all image frames are acquired, and a pixel gray scale distribution map is established.

[0069] After each near-infrared image frame is acquired, the light signals collected at the row and column positions of each pixel in the image frame are directly converted electronically, a standard photoelectric conversion process is used to convert the received light signals into electrical signals, and then corresponding pixel gray scale signals are generated. Each pixel gray scale signal is digitally processed by a quantizer to quantize the analog electrical signal into a digital gray scale value, which is an unsigned integer format and can be represented by 8 bits, 10 bits or 12 bits. In near-infrared image processing, 8-bit format is usually selected, with a range of 0-255. After the quantization of the pixel gray scale value is completed, the digital gray scale value is arranged in row and column order based on the row and column positions of the pixel to construct a two-dimensional gray scale matrix. The number of rows of the two-dimensional gray scale matrix is consistent with the vertical resolution of the image frame, and the number of columns is consistent with the horizontal resolution, ensuring the integrity of the spatial correspondence of the pixels. Each element in the two-dimensional gray scale matrix contains its corresponding row and column index and digital gray scale value. The two-dimensional gray scale matrix is marked as a pixel gray scale distribution map in the image processing process, which facilitates subsequent steps such as spatial feature extraction and convolution analysis. When the number of pixels is large (such as 1024x1024 or higher resolution), the calculation resources or storage resources are tight, and a matrix simplification strategy is used to simplify the pixel gray scale matrix to reduce the calculation burden. The matrix simplification process of the present scheme includes two ways of row and column downsampling and regional mean replacement, and the specific steps are as follows:

[0070] (1) Row and column downsampling is performed according to a preset simplification factor, for example, the simplification factor can be set to 2, which means that every other pixel is retained and the adjacent pixels are removed to form a downsampled pixel gray scale matrix. If the simplification factor is set to 4, then every 4 pixels retain 1 pixel, and the simplification ratio is increased in this way.

[0071] (2) In some scenarios, to preserve the average gray feature of the pixels, a region mean replacement method can also be used, in which adjacent pixels are divided into small blocks (such as 4x4 or 8x8), and the mean value of the pixel gray of each small block is calculated, and the mean value is used to replace the original pixel value, which simplifies the matrix size while ensuring the integrity of the overall gray distribution feature. The above simplification factor and block size can be set in the process design stage, for example, the default value of the simplification factor is 2, and the region mean replacement block size is 4x4 by default, which can be adjusted according to actual application.

[0072] After the construction of the pixel gray distribution map is completed, a sequence of continuous image frames is read, and each frame of image is indexed and labeled with a frame number, which is numbered in ascending order according to the image acquisition sequence, such as the first frame numbered 1, the second frame numbered 2, and so on. Each pixel gray distribution map is bound to the corresponding image frame number.

[0073] In S3, a multi-branch convolutional neural network is used to learn the spatial features of the pixel gray distribution map, and the background, target structure and dynamic residual image area distribution are separated.

[0074] The pixel gray distribution map formed after photoelectric conversion and digital processing is scanned pixel by pixel. Pixel by pixel scanning means that starting from the top left corner of the image, the pixels are scanned column by column from left to right along the horizontal direction, and after scanning a row of pixels, the next row of pixels is scanned downward, until the entire image is scanned row by row and column by column. In the process of pixel by pixel scanning, the digital gray value of each pixel is read and temporarily stored in a sequential array, keeping its row and column position unchanged in the two-dimensional plane. Then, the pixel gray values in the sequential array are recombined in row and column order to construct a multi-dimensional input tensor. In this embodiment, the multi-dimensional input tensor constructed mainly includes three dimensions: row dimension, column dimension and channel dimension. Among them, the row dimension is consistent with the vertical resolution of the pixel gray distribution map, the column dimension is consistent with the horizontal resolution, and the channel dimension is set to single channel, that is, each pixel point corresponds to a digital gray value.

[0075] The convolutional neural network structure adopts a multi-branch strategy, and simultaneously transmits the multi-dimensional input tensor to multiple branches, each branch corresponding to a target component, including a background branch, a target structure branch, and a dynamic ghost branch. The input format of each branch is kept as a single channel format, that is, each input tensor only contains one number gray scale value data, to ensure the simplicity of the data structure and the efficiency of the calculation. To ensure the focus of each branch on the target features, a convolution kernel of corresponding size is configured for each branch. The size of the convolution kernel of the background branch is set to 5x5 pixels, the size of the convolution kernel of the target structure branch is set to 3x3 pixels, and the size of the convolution kernel of the dynamic ghost branch is set to 7x7 pixels. The selection of the convolution kernel size is based on the spatial distribution characteristics of each target component: the background region usually has a large spatial distribution, so a larger convolution kernel is suitable; the target structure region is relatively compact, so a medium-sized convolution kernel is used; the dynamic ghost region often has a large area of disturbance, so it is more advantageous to use a larger convolution kernel for feature extraction. The above-mentioned convolution kernel size can be fine-tuned according to actual needs during the engineering debugging stage, for example, the background convolution kernel size can be set between 3x3 and 9x9.

[0076] The constructed multi-dimensional input tensor is directly input into the multi-branch convolutional neural network. The input layer of the network maintains a single channel format, and the gray scale value of each pixel point corresponds to a separate channel dimension in the tensor, ensuring the corresponding relationship with the image gray scale distribution. Each branch receives the input tensor and performs independent convolution operations, and the convolution operation is performed pixel by pixel using the configured convolution kernel. The weight of each convolution kernel is determined through a large amount of data analysis and engineering tuning during the development and verification of the network. Specifically, combining historical image data, the convolution kernel of each branch is repeatedly convolved and analyzed, and the weight distribution of the convolution kernel is determined through the statistical law of the gray scale distribution, so that it can stably extract the features of the respective target components in different branches. In the background branch, the target structure branch, and the dynamic ghost branch, the convolution operation is performed with the respective configured convolution kernel size for sliding window convolution, and the convolution step is set to 1 by default to ensure complete extraction of pixel information. After the convolution operation is completed, the output result is taken as the feature map of the branch. In the local convolution operation, the extracted pixel spatial features mainly include texture features, edge features, and continuous region features. Texture features are reflected through local changes in pixel gray scale values, edge features are expressed through pixel gray scale gradient changes, and continuous region features are reflected through the similarity of adjacent pixel gray scale values. Finally, through the local convolution operation of the multi-branch convolutional neural network, the feature maps of the background, target structure, and dynamic ghost branches are constructed.

[0077] The feature maps of different branches are cross-branch feature fused, and the background, target structure and dynamic residual area distribution are separated by feature clustering. The feature maps of the background branch, the target structure branch and the dynamic residual branch are read in the output results of the multi-branch convolutional neural network in turn. The feature map of each branch is a two-dimensional matrix format, which maintains the row and column dimensions consistent with the input multi-dimensional input tensor, and each pixel position contains a feature value output by the convolution kernel. The reading order of the feature map strictly follows the definition order of the branch structure, avoiding feature correspondence offset caused by order error. When reading, the pixel position of each feature map corresponds to the row and column position of the original image frame, ensuring the consistency of the spatial position. After the feature map is read, the background branch feature value, the target structure branch feature value and the dynamic residual branch feature value at the same pixel position are numerically superimposed to form a joint feature in the form of a three-dimensional vector. The first dimension of the joint feature vector represents the background feature value, the second dimension represents the target structure feature value, and the third dimension represents the dynamic residual feature value. Through the construction of this joint feature, the complete fusion of features of different branches at the same pixel position is ensured. The joint features of all pixels are stored in order of row and column position, forming a pixel-level joint feature set of the entire image frame.

[0078] According to the feature distribution characteristics of the target region, a joint feature classification template of the background, target structure and dynamic residual area is preset. The classification template is completed by manual sampling and labeling, that is, typical background areas, target structure areas and dynamic residual areas are selected in historical samples, and the joint feature vectors of these areas are collected in turn, and the numerical features are recorded. The joint feature template of each region is formed by the average value of the joint feature vectors of multiple samples, which ensures the representativeness and engineering repeatability of the template. On the basis of establishing the joint feature classification template, the corresponding class index is established according to the region category (background, target structure, dynamic residual). The class index is represented by an integer value, for example, the background region class index is 0, the target structure region class index is 1, and the dynamic residual region class index is 2. The class index corresponds to the joint feature template. The constructed pixel-level joint feature set performs unsupervised clustering analysis in the entire pixel gray distribution map range. The unsupervised clustering process adopts the K-means clustering method, and the pixel-level joint feature set is divided into several category clusters, and the joint feature vectors in each category cluster have high similarity, which ensures the local consistency of the pixel features. The number of clusters in the clustering is preset to 3 in the engineering, corresponding to the three types of target components of background, target structure and dynamic residual.

[0079] In each category cluster, a local joint feature set is extracted, and the average feature of all joint feature vectors in the set is calculated. The average feature is calculated in a normalized average manner, that is, each feature value is converted to a range of 0-1, then the pixels in the category cluster are summed and divided by the number of pixels to obtain each average feature value, and then an average feature vector is formed. The average feature vector of each category cluster is distance measured with a preset joint feature classification template, and the numerical distance between the average feature vector and each template is calculated using the Euclidean distance or other distance measurement algorithms. The template category with the smallest distance is selected as the category index of the category cluster. The category index of each category cluster is mapped to the row and column positions of the pixel gray distribution map. The mapping process strictly follows the pixel row and column order, and each pixel's category index is directly assigned to the corresponding pixel row and column position to ensure the integrity and accuracy of the mapping relationship. The pixel position corresponding to each category index represents the region category to which the pixel belongs. The background region corresponds to category index 0, the target structure region corresponds to category index 1, and the dynamic residual image region corresponds to category index 2. Through this mapping, the pixel-level automatic labeling of the background region, target structure region and dynamic residual image region distribution of a single image frame is completed.

[0080] In S4, the target structure region is taken as the center, and spatial alignment is performed on the continuous image frames to generate a global region distribution by superimposing all image frames.

[0081] According to the extracted target structure region distribution, the spatial position of the target structure region in each image frame is determined. The spatial position of the target structure region is described by pixel row and column coordinates. Specifically, the center pixel point of the target structure region is taken as the reference point, and the row and column coordinates of the center pixel point can be obtained by traversing the pixel set of the target structure region and calculating the average of the upper and lower boundaries and the left and right boundaries, ensuring the repeatability and consistency of the spatial position of the center pixel point. After determining the center pixel point of the target structure region, it is taken as the alignment reference center of the pixel distribution map. For each image frame, the offset (i.e. row offset and column offset) of the center pixel point of the target structure region relative to the preset reference center is calculated, and then the pixel distribution map of the entire image frame is translated as a whole to align the center pixel point of the target structure region with the preset reference center. The translation operation is performed by adjusting the row and column index values of each pixel, and the row offset and column offset are added to the original row and column coordinates of each pixel to complete the spatial alignment at the pixel level. For pixels outside the image boundary, a boundary clipping strategy is used in engineering, that is, the pixel values outside the image boundary range are directly discarded to ensure that the row and column dimensions of the pixel distribution map after spatial alignment are consistent with the initial input, avoiding data dimension mismatch. In addition, to prevent image region loss or black edges caused by spatial translation, a boundary compensation mechanism can be set for the pixel distribution map in engineering, such as filling with the gray values of adjacent pixels, thereby ensuring the consistency and integrity of the image frame.

[0082] For each region category, the pixel class index of the same row and column position in all image frames is statistically summarized according to the correspondence of the pixel row and column position, to generate the region category probability distribution of the row and column position under the cross-frame superposition. Specifically, for the background region, the target structure region and the dynamic residual image region, the superposition counters of the pixel row and column position are established respectively, and the pixel position of the corresponding category in each frame of image is counted by one. After accumulating and counting all frames, the region category superposition distribution of each pixel row and column position is obtained. Through the statistical summary process, the cross-frame pixel region distribution analysis is realized. After completing the superposition statistics of the pixel region category, for the pixel category distribution of the same row and column position, the category index with the most cumulative number is selected as the final region category of the row and column position, avoiding the problem of category conflict in the cross-frame region distribution. Finally, based on the superposition results of all frames, the distribution maps of the global background region, the global target structure region and the global dynamic residual image region are output. Each distribution map is represented in the form of a two-dimensional matrix, the row and column dimensions of the matrix are consistent with the row and column dimensions of the input image frame, and each matrix element contains the final category index information, ensuring the integrity of the spatial correspondence of the region distribution and strictly matching the row and column position of each image frame.

[0083] In S5, historical near-infrared image frame samples are obtained, and a neural network model is trained through a training set constructed by the samples to establish a region distribution prediction model.

[0084] Collect historical near-infrared image frame sample data, which needs to cover different external light source parameters and production line running speeds to fully reflect the image acquisition characteristics under the actual running conditions of the production line. The external light source parameters include light source intensity, color temperature and illumination angle. The specific value range can refer to: light source intensity 300lx to 1000lx, color temperature 3500K to 6000K, and illumination angle 30° to 60°. The production line running speed range covers 0.2 meters per second to 1.5 meters per second, ensuring the diversity and representativeness of the sample data. Each group of sample data is numbered and managed, and each group of data contains a number of continuous image frame samples and their corresponding external light source parameters and production line running speed information, which facilitates subsequent data grouping and label annotation. In each group of sample data, the pixel gray scale distribution map of all historical near-infrared image frames is read and processed frame by frame, and each frame of image has pixel gray scale distribution information and frame number index. For the image frames in each group of sample data, the cross-frame alignment of the pixel row and column position is performed according to the extracted background region, target structure region and dynamic residual image region distribution, and the region category index of the same row and column position is summarized by combining the spatial superposition statistics method, to determine the spatial distribution of the global background region, the global target structure region and the global dynamic residual image region in each group of sample data.

[0085] For the generated global area distribution result, label marking processing is performed on the sample data. The label marking process matches each pixel point in the sample data with the class index in the global area distribution according to its row and column coordinates in units of pixel-level labels. After the label marking processing is completed, the labeled data is bound with the original pixel gray distribution data to construct the training set data of the area distribution prediction model. The training set data includes pixel gray distribution features and corresponding class index labels, and has high engineering operability and repeatability.

[0086] The neural network model adopts a convolutional neural network structure and is composed of multiple convolutional layers, activation function layers, and an output layer. The convolutional layer is responsible for extracting the spatial features of the input data, and the activation function layer uses a ReLU function to extract nonlinear features. The output layer uses a Softmax function to convert the model output into a probability distribution of each class. In the model training process, a cross-entropy loss function is used as the objective function of the model to calculate the difference between the model prediction result and the actual label. In the training process, a small batch stochastic gradient descent optimization algorithm is used to iteratively update the model weights, with an initial learning rate of 0.001. The learning rate can be dynamically adjusted according to the loss change in the training process to ensure the convergence speed and training stability of the model. In each training round, the training set data is randomly shuffled and input into the model for forward propagation and backward propagation. In the forward propagation process, the input data passes through the convolutional layer, the activation function layer, and the output layer in turn, and outputs the probability distribution of the area class. In the backward propagation process, the gradient information is calculated according to the cross-entropy loss function, and the convolution kernel weights and the bias parameters of each layer of the neural network are updated to make the model prediction result gradually approach the true label distribution. The entire training process is set to multiple training rounds (epochs), and each round iterates through the entire training set data once. The number of training rounds is usually set to 100 rounds or more, and the specific number of rounds can be dynamically adjusted according to the convergence curve of the model. When the validation set loss value of the model tends to be stable and reaches the expected convergence accuracy, the training process ends, and the trained model parameters are saved to form the area distribution prediction model.

[0087] After training, the model is deployed to the production line and integrated into the real-time image processing process to ensure that the pixel gray distribution map of each frame of image can be predicted in real time during actual acquisition.

[0088] In S6, the background and residual image area distribution of the real-time image is predicted based on the area distribution prediction model, the residual image area is repaired by the background area, and the residual image corrected image is output.

[0089] The obtained external light source parameters and real-time production line running speed are input as input features into the area distribution prediction model.

[0090] The pixel gray value of the pixel gray distribution graph is processed by neighborhood repair on the pixel position coordinates of the residual image area. The repair processing adopts the pixel gray value of the background area as a reference, calculates the average gray value and standard deviation of the background area by establishing a pixel gray value statistical model (local weighted average) in the background area, and combines the pixel position of the residual image area to correct the pixel gray value of the residual image area by using a weighted interpolation method. The weighted interpolation method adopts the adjacent pixels of the background area as a reference, calculates the corrected gray value of the residual image area pixel according to a distance weight function. Or the average gray value of the background area is directly used to replace the pixel gray value of the residual image area, and this method is used as a simple correction method in the case of excessive system operation amount. After the repair is completed, the corrected pixel gray value is replaced with the original pixel value of the residual image area, the pixel values of other areas remain unchanged, the gray structure of the whole image is ensured to be complete. The output image frame is saved in the form of a two-dimensional gray matrix, contains the corrected gray value of all pixel positions, and the row and column indexes are consistent with the original input image, avoiding image alignment error and area offset.

[0091] Through the neighborhood repair processing, the dynamic correction of the residual image area is completed, the consistency and integrity of the corrected image frame in the visual and engineering application are ensured, and reliable data support is provided for subsequent image analysis and product detection. The finally output residual image frame after correction has high image quality and engineering applicability, and meets the engineering requirements of dynamic residual image correction.

[0092] Embodiment 2, the difference between the embodiment 2 and the embodiment 1 of the present application is that the embodiment 2 is to introduce an image processing system based on near-infrared imaging.

[0093] Figure 2 The structure schematic diagram of the image processing system based on near-infrared imaging is given, the image processing system based on near-infrared imaging comprises a gray value construction module, a convolution feature extraction module, a global distribution construction module, a prediction model training module and a dynamic residual image correction module.

[0094] The gray value construction module: collects continuous near-infrared image frame samples during the running of the production line, dynamically adjusts the image frame rate in combination with the real-time production line speed and external light source parameters, and performs electronic conversion and quantization processing on the pixel light signals of the image frames to generate digital gray values and mark the row and column positions, and construct a pixel gray distribution map;

[0095] The convolution feature extraction module: a multi-branch convolutional neural network is used to perform spatial feature learning on the pixel gray distribution map, configure convolution kernels according to the background, target structure and dynamic residual image branches, and extract spatial features to generate branch feature maps, and then assign class indexes through feature fusion and unsupervised clustering, and output the pixel region distribution of the background, target structure and dynamic residual image;

[0096] The global distribution construction module: the target structure region is taken as the alignment reference, the spatial alignment of the continuous image frames is performed, the corresponding relationship of the pixel row and column positions is used, and the background, target structure and dynamic residual image region distribution is superimposed across frames to generate the global background region, global target structure region and global dynamic residual image region distribution;

[0097] The prediction model training module: historical near-infrared image frame sample data and corresponding global region distribution maps are obtained, the sample data is labeled to construct a training set, a neural network model is selected to input the training set for training, and after the training is completed, a region distribution prediction model is generated and deployed to the production line end;

[0098] The dynamic residual image correction module: the external light source parameters and real-time production line running speed of the production line end are input to the region distribution prediction model, the background and dynamic residual image region distribution is obtained, the residual image region is repaired by the neighborhood, the residual image region is corrected by the pixel gray value of the background region, and the corrected image frame is output.

[0099] The above formulas are all dimensionless numerical calculations, the formula is obtained by software simulation of a large amount of data to obtain the most real situation, and the preset parameters and threshold values in the formula are set by the person skilled in the art according to the actual situation.

[0100] The above-described embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the above-described embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0101] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0102] Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device, and module can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0103] In several embodiments provided in the present application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed ones can be indirect coupling or communication connection through some interfaces, devices, or modules, which can be electrical, mechanical, or other forms.

[0104] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0105] In addition, the functional modules in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0106] The functions, if realized in the form of software function modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0108] Finally: the above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. An image processing method based on near-infrared imaging, characterized by, The method comprises the following steps: S1: continuously collecting near-infrared images of the production area based on the running speed of the production line to generate a pixel set of the near-infrared image frame; S2: obtaining pixel gray values of the pixel set corresponding to all image frames to establish a pixel gray distribution map; S3: using a multi-branch convolutional neural network to learn spatial features of the pixel gray distribution map to separate background, target structure and dynamic residual image area distribution; S4: taking the target structure area as the center, performing spatial alignment on the continuous image frames to generate a global area distribution by superimposing all image frames; S5: obtaining historical near-infrared image frame samples, training a neural network model through a training set constructed by the samples, and establishing an area distribution prediction model; S6: predicting the background and residual image area distribution of the real-time image based on the area distribution prediction model, performing neighborhood repair on the residual image area through the background area, and outputting the residual image after correction; In S3, the multi-branch convolutional neural network is used to learn the spatial features of the pixel gray distribution map to separate the background, target structure and dynamic residual image area distribution, which specifically comprises: pixel-by-pixel scanning of the pixel gray distribution map, arranging the pixel gray values in row and column order to form a multi-dimensional input tensor; setting the multi-branch convolutional neural network structure, retaining the single-channel input format in the multi-dimensional input tensor, and configuring convolution kernels of corresponding sizes for different branches; wherein the multi-branch includes background, target structure and dynamic residual image branches; inputting the multi-dimensional input tensor into the multi-branch convolutional neural network, and performing local convolution operation in each branch based on the configured convolution kernel to extract pixel spatial features and construct feature maps of different branches, wherein the spatial features include texture features, edge features and continuous region features; cross-branch feature fusion of the feature maps of different branches to separate the background, target structure and dynamic residual image area distribution through feature clustering; the cross-branch feature fusion of the feature maps of different branches to separate the background, target structure and dynamic residual image area distribution through feature clustering specifically comprises: reading the feature maps of the background branch, the target structure branch and the dynamic residual image branch in the output of the multi-branch convolutional neural network in sequence; superimposing the three types of feature maps to construct joint features based on the background feature value, the target structure feature value and the dynamic residual image feature value at each pixel position to form a joint feature set at the pixel level; predefining a joint feature classification template for the background, the target structure and the dynamic residual image area, and establishing corresponding class indexes; performing unsupervised clustering on the joint feature set in the entire pixel gray distribution map to extract the local joint feature set of each class cluster, calculating the average feature of the local joint feature set, and labeling the class index of each clustering cluster through the distance between the average feature and the joint feature classification template; mapping the labeled class index to the row and column positions of the pixel gray distribution map to generate the background area, the target structure area and the dynamic residual image area distribution corresponding to a single image frame; in S6, the background and residual image area distribution of the real-time image is predicted based on the area distribution prediction model, the residual image area is repaired through the background area, and the residual image after correction is output, which specifically comprises: Input the external light source parameters at the production line end and the real-time production line running speed into the area distribution prediction model; Obtain the model output result, and perform background area and dynamic residual image area distribution labeling on the pixel gray scale distribution graph of the real-time collected near-infrared image frame; Extract the pixel position coordinates of the labeled residual image area and background area in the pixel gray scale distribution graph; Perform neighborhood repair processing on the pixel gray scale values of the pixel gray scale distribution graph at the pixel position coordinates of the residual image area, perform pixel gray scale correction on the residual image area according to the pixel gray scale values of the background area, and output the residual image corrected image frame.

2. The image processing method based on near-infrared imaging according to claim 1, characterized in that, In S1, continuous near-infrared image acquisition is performed on the production area based on the running speed of the production line to generate a pixel set of the near-infrared image frame, which specifically includes: Obtain the real-time running speed of the production line, and dynamically adjust the near-infrared image acquisition frame rate based on a preset fixed total frame number; Obtain the external light source parameters, continuously collect near-infrared image frames on the target production area of the production line, record the collection time stamp of each image frame, and simultaneously perform time axis uniform processing on the image frames using an interpolation method; Label the pixel number and row and column position of the image frame corresponding to each time sequence index position on the time axis to generate a pixel set of each image frame.

3. The image processing method based on near-infrared imaging according to claim 1, characterized in that, In S2, the pixel gray scale values of the pixel set corresponding to all image frames are obtained to establish a pixel gray scale distribution graph, which specifically includes: In the pixel set corresponding to the image frame, the optical signal collected at each pixel row and column position is converted into an electronic signal to generate a corresponding pixel gray scale signal, and the pixel gray scale signal is quantized into a digital gray scale value; Convert the digital gray scale value into a two-dimensional gray scale matrix based on the arrangement of the row and column positions of the pixels, and mark the two-dimensional gray scale matrix as a pixel gray scale distribution graph; Read the continuous image frame sequence and perform image frame number index labeling to establish the correspondence between the pixel gray scale distribution graph and its belonging image frame number.

4. The image processing method based on near-infrared imaging according to claim 1, characterized in that, In S4, the target structure area is taken as the center to perform spatial alignment on the continuous image frames, and all image frames are superimposed to generate a global area distribution, which specifically includes: Take the target structure area as the center position of the pixel distribution graph to perform spatial alignment on the pixel distribution graphs corresponding to all image frames; Based on the pixel row and column position correspondence relationship of the spatially aligned background area, target structure area and dynamic residual image area distribution, perform cross-frame superposition on all image frames to generate a global background area, a global target structure area and a global dynamic residual image area distribution.

5. The image processing method based on near-infrared imaging according to claim 1, characterized in that, In S5, historical near-infrared image frame samples are obtained, a neural network model is trained through a training set constructed by the samples, and a region distribution prediction model is established, which specifically includes: Obtain historical near-infrared image frame sample data, and each group of sample data covers different external light source parameters and production line running speeds; In each group of sample data, the global background area, the global target structure area and the global dynamic residual image area distribution of all historical near-infrared image frames are determined; Based on the global background area, the global target structure area and the global dynamic residual image area distribution determination result, perform label marking processing on the sample data to construct a training set data of the region distribution prediction model; A neural network model is selected, and a training set data is input to train a region distribution prediction model, and the model is deployed to a production line end after being trained.

6. An image processing system based on near infrared imaging for implementing the method according to any one of claims 1 to 5, characterized in that The method comprises a gray value construction module, a convolution feature extraction module, a global distribution construction module, a prediction model training module, and a dynamic residual image correction module. The gray value construction module: collects continuous near-infrared image frame samples during production line operation, dynamically adjusts the image frame rate in combination with real-time production line speed and external light source parameters, and performs electronic conversion and quantization processing on the pixel light signals of the image frames to generate digital gray values and mark the row and column positions, and construct a pixel gray distribution map; The convolution feature extraction module: uses a multi-branch convolutional neural network to perform spatial feature learning on the pixel gray distribution map, configures convolution kernels according to the background, target structure, and dynamic residual image branches, and extracts spatial features to generate branch feature maps, and assigns class indexes through feature fusion and unsupervised clustering, and outputs the pixel region distribution of the background, target structure, and dynamic residual image; The global distribution construction module: takes the target structure region as the alignment reference, performs spatial alignment on the continuous image frames, and based on the correspondence of the pixel row and column positions, superimposes the background, target structure, and dynamic residual image region distribution across frames to generate global background region, global target structure region, and global dynamic residual image region distribution; The prediction model training module: obtains historical near-infrared image frame sample data and corresponding global region distribution maps, labels the sample data to construct a training set, selects a neural network model to input the training set for training, generates a region distribution prediction model after training, and deploys it to the production line end; The dynamic residual image correction module: inputs the external light source parameters and real-time production line running speed of the production line end to the region distribution prediction model, obtains the background and dynamic residual image region distribution, performs neighborhood repair on the residual image region, corrects the residual image region using the pixel gray value of the background region, and outputs the corrected image frame.

Images