Small sample generation method of infrared image of conveyor roller and its fault simulation device

Abstract

The present invention discloses a method for generating a small sample of infrared images of conveyor rollers and a fault simulation device thereof. The method uses a generative adversarial network model including a generator and a discriminator, and includes a simple task learning stage and a complex task learning stage. In the simple task learning stage, only infrared images of completely stuck rollers are used to learn their global temperature distribution and local high temperature area distribution, and the generative adversarial network model is trained in combination with the loss of global temperature distribution and local high temperature area distribution. In the complex task learning stage, infrared images of other preset stuck rollers are used to learn their global label distribution and local landmark area label distribution, and the generative adversarial network model is trained in combination with the loss of global label distribution and local landmark area label distribution. The trained generative adversarial network model is used to generate a small sample of infrared images of conveyor rollers. The present invention can generate high-quality infrared images of rollers, thereby improving the accuracy and efficiency of fault detection.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for generating a small sample of infrared images of conveyor rollers and a fault simulation device thereof. Background Art

[0002] In industrial production, especially in the operation of material handling and transportation systems such as conveyors, the health of rollers and conveyor belts has a vital impact on the efficiency and safety of the entire production process. Failures of rollers, such as overheating, abnormal friction or jamming, and failures of conveyor belts, such as wear and tear, may lead to equipment damage, production stagnation and even safety accidents. Therefore, timely detection and diagnosis of these potential failures is the key to maintaining production continuity and equipment safety.

[0003] Infrared image detection technology is widely used in roller fault detection. However, in actual applications, the image representation of weak fault features of key conveyor components is poor, the richness of fault image samples is poor, and the collection and analysis of infrared images have problems such as insufficient samples and unclear features. Existing technologies are difficult to generate high-quality infrared images under small sample conditions, resulting in reduced detection accuracy and efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating a small sample of infrared images of conveyor rollers and a fault simulation device thereof, so as to generate high-quality infrared images of the rollers, thereby improving the accuracy and efficiency of fault detection.

[0005] To achieve the above-mentioned object, according to a first aspect of the present invention, a method for generating a small sample of infrared images of conveyor rollers is provided, the method using a generative adversarial network model including a generator and a discriminator, and including two stages: a simple task learning stage and a complex task learning stage;

[0006] In the simple task learning stage, only infrared images of completely stuck rollers are used to learn the global temperature distribution and local high temperature area distribution of each infrared image of completely stuck rollers, and the generative adversarial network model is trained by combining the losses of the global temperature distribution and the local high temperature area distribution;

[0007] In the complex task learning stage, infrared images of other preset stuck rollers are used to learn the global label distribution and local landmark area label distribution of each infrared image of other preset stuck rollers, and the generative adversarial network model is trained by combining the losses of the global label distribution and the local landmark area label distribution;

[0008] The trained generative adversarial network model is used to generate small samples of conveyor roller infrared images.

[0009] Furthermore, we study the global temperature distribution, including:

[0010] The range of setting temperature value is [ , ] and is divided into K intervals, each of which has a width of , as shown below:

[0011]

[0012] Constructing a temperature histogram of an infrared image , represents the global temperature distribution of the infrared image, and the formula is as follows:

[0013]

[0014] In the formula, i and j represents the pixel position, represents the temperature value at the corresponding position, M and N Indicates the size of the infrared image; Indicates when Falling in Temperature range Take 1 when it is, otherwise take 0;

[0015] The temperature histogram Convert to probability density function :

[0016]

[0017] right Gaussian smoothing is performed, and the formula is as follows:

[0018]

[0019] In the formula, represents the global temperature distribution after Gaussian smoothing, represents pi, represents a natural constant, represents the standard deviation of the Gaussian kernel;

[0020] Calculate the global temperature distribution of the real image :

[0021]

[0022] In the formula, Indicates o The temperature value of each pixel, O Represents the total number of image pixels; Represents the Gaussian kernel function, which is used to adjust the temperature offset Assign different weights to achieve smoothing.

[0023] Further, study the distribution of local high temperature areas, including:

[0024] Calculate the average temperature of an infrared image and temperature standard deviation , define the high temperature threshold :

[0025]

[0026] In the formula, is a constant used to adjust the high temperature threshold;

[0027] Based on high temperature threshold Generate high temperature mask :

[0028]

[0029] In the formula, express i and j The temperature value at the location;

[0030] Use sliding windows to divide the infrared image into multiple small areas , and calculate the local high temperature area Temperature variance ;in, Indicates The set of pixels in a window;

[0031] Calculate the local high temperature area distribution , the temperature in the local high temperature area is Number of pixels:

[0032]

[0033] In the formula, m and n express The middle pixel position, Indicates the temperature value at the corresponding position; is the Dirac function, when The value is 1 when it is, otherwise it is 0; represents the high temperature mask at the corresponding position;

[0034] Smooth out local high temperature areas And perform probability density function conversion, in each temperature range Construct a smoothing window ranging from arrive , use Gaussian smoothing to assign different weights to each temperature value in the local smoothing window, perform smooth weighted summation, and obtain the local high temperature area distribution of the real image :

[0035]

[0036] In the formula, represents the half width of the smoothing window, Indicates the temperature Offset on foundation The temperature value, is the Gaussian kernel function, is the standard deviation of the Gaussian kernel.

[0037] Furthermore, the generative adversarial network model is trained by combining the loss of global temperature distribution and local high temperature area distribution, including:

[0038] Global temperature distribution of the image generated by the corresponding calculation generator , Local high temperature area distribution And local high temperature areas Temperature variance ;

[0039] Define global temperature distribution losses :

[0040]

[0041] Definition of local high temperature area distribution loss :

[0042]

[0043] And the temperature variance loss in the local high temperature area:

[0044]

[0045] Constructing a comprehensive loss function for the generator :

[0046]

[0047] In the formula, represents the loss of the generator; , and is the weight coefficient;

[0048] Directly construct the discriminator loss , the generator and discriminator are trained using only infrared images of completely stuck rollers.

[0049] Furthermore, we learn the global label distribution, including:

[0050] The type of roller jam is used as a global label to determine the global labels of all pixels in the infrared image;

[0051] Calculate the global label distribution of real images :

[0052]

[0053] In the formula, i Indicates i pixels, N Represents the total number of image pixels, Indicates i The global label of pixels, c Indicates c Class global label; Indicates when For the c It takes 1 if it is a global label, otherwise it takes 0.

[0054] Furthermore, we learn the distribution of local landmark region labels, including:

[0055] Use SLIC to segment the infrared image into multiple super pixels, each super pixel represents a local area in the image;

[0056] Calculate the temperature feature, shape feature and position feature of each super pixel in the image; the temperature feature is the average temperature of the super pixel, the shape feature is the area of ​​the super pixel, and the position feature is the centroid of the super pixel;

[0057] Define the temperature feature labels of super pixels, including normal, low temperature area, medium temperature area and high temperature area; define the shape feature labels of super pixels, including small area, medium area and large area; define the position feature labels of super pixels, including center area, edge area and corner area; use the temperature feature labels, shape feature labels and position feature labels as local labels;

[0058] Computing local landmark region label distribution of infrared images :

[0059]

[0060] In the formula, Indicates j local tags, Indicates the image m super pixels, M represents the total number of super pixels in the image, Indicates when The local label is It takes 1 when it is, otherwise it takes 0.

[0061] Furthermore, the generative adversarial network model is trained by combining the loss of global label distribution and local landmark region label distribution, including:

[0062] The global label distribution of the images generated by the corresponding calculation generator And local landmark area label distribution ;

[0063] Define the loss of the global label distribution :

[0064]

[0065] In the formula, C Indicates the total number of global tags;

[0066] Defining local landmark region label distribution loss :

[0067]

[0068] In the formula, J Represents the total number of local labels;

[0069] Constructing a comprehensive loss function for the generator :

[0070]

[0071] In the formula, represents the loss of the generator, and represents the weight coefficient;

[0072] Constructing the label distribution loss of the global discriminator :

[0073]

[0074] In the formula, Represents a real image x The global label in is the target global label c The probability distribution of Represents the discriminator for the real image x Belongs to the global tag c The predicted probability of Represents the image generated by the discriminator to the generator Belongs to the global tag c The predicted probability of Represents the expectation of real image data; Represents the expectation of generating image data;

[0075] Constructing the iconic region label distribution loss of the local discriminator :

[0076]

[0077] In the formula, M is the total number of super pixels in the image, Represents a real image x In local area The local label is the target local label j The probability distribution of Represents the discriminator for the real image x In local area The local label is the target local label j The predicted probability of Represents the image generated by the discriminator to the generator In local area The local label is the target local label j The predicted probability of

[0078] Constructing a comprehensive loss function for the discriminator :

[0079]

[0080] In the formula, represents the loss of the discriminator; and is the weight coefficient;

[0081] The generator and discriminator are trained using infrared images of other preset stuck types of rollers.

[0082] Furthermore, other preset stuck types include semi-stuck, slight stuck and normal.

[0083] Furthermore, the method also includes:

[0084] Collect infrared images of rollers with different stuck types and perform image preprocessing, including: converting color infrared images into grayscale images, performing Gaussian smoothing on the grayscale images, performing calculations using the Laplace operator, and finally normalizing them;

[0085] Perform image data enhancement on the preprocessed infrared images, including vertical flipping, 90-degree random rotation, horizontal flipping, and transposition;

[0086] Use the multi-layer perceptron MLP classification model to predict the global label of the image; including:

[0087] The global label of the infrared image of the roller is obtained, and the multi-layer perceptron MLP classification model is trained. The trained multi-layer perceptron MLP classification model is used to predict the global label of the image generated by the generator, and then the global label distribution of the image generated by the generator is calculated.

[0088] According to a second aspect of the present invention, there is provided a conveyor roller fault simulation device, which is applied to any one of the above-mentioned conveyor roller infrared image small sample generation methods, the device comprising a roller, a stuck ring, a friction plate with adjustable thickness and a bolt with variable tightening degree;

[0089] The locking ring includes two semicircles, and the two ends of the semicircles are connected by bolts with variable tightening degrees; the roller is located inside the locking ring; and the friction plate with adjustable thickness is located between the roller and the locking ring;

[0090] The thickness of the friction plate with adjustable thickness and the tightening degree of the bolts with variable tightening degree are adjusted to simulate different degrees of roller jamming failure.

[0091] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0092] By constructing a CDA-GAN model, the present invention can effectively learn and simulate global and local temperature distribution and label distribution, thereby optimizing the detail performance of the generated image. This method uses adversarial training and curriculum learning to ensure that the generated image is highly consistent with the real data in terms of global temperature distribution and label distribution, significantly improving the quality of small sample image generation and meeting the needs of complex tasks. This technology has significant advantages in improving the quality and detail performance of small sample image generation, and helps to achieve better results in various image processing applications.

[0093] In addition, the design of the present invention can adjust the friction plate and bolt configuration, providing an effective means of simulating different degrees of roller jamming conditions, making fault testing more flexible and practical, and helping to better understand and optimize the operation and maintenance strategies of rollers and conveyor belts. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is an overall flow chart of the method for generating small samples of infrared images of conveyor rollers of the present invention;

[0095] Figure 2 It is a structural schematic diagram of a conveyor roller fault simulation device of the present invention;

[0096] Figure 3 Learn the structure diagram for the simple task of the present invention;

[0097] Figure 4 Learn flow charts for complex tasks of the present invention;

[0098] Figure 5 It is a flow chart of the CDA-GAN model of the present invention;

[0099] Figure 6 It is a structural diagram of the generator model of the present invention;

[0100] Figure 7 This is a structural diagram of the discriminator model of the present invention.

[0101] In the figure: 1-locking ring, 2-roller, 3-bracket, 4-angle code, 5-support plate, 6-friction plate with adjustable thickness, 7-bolt with variable tightening degree. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0103] This paper designs a small sample generation method of weak fault feature images of conveyor rollers based on curriculum learning adversarial training network (CAD-GAN). By learning global and local features to optimize the detailed performance of the generated image, it is suitable for applications such as high-precision image synthesis and small sample image generation. Through fault simulation, curriculum domain learning, and image sample generation, this paper aims to improve the early recognition and early warning of potential abnormal types such as stuck overheating, semi-stuck overheating, and friction abnormalities of three rollers.

[0104] First, in view of the poor performance of weak fault feature images of key components, the present invention designs a set of roller fault simulation devices with different jamming degrees to simulate the normal operation, slight jamming, semi-jamming, and complete jamming of three rollers, and optimizes the layout and angle of infrared cameras to capture thermal image anomalies caused by failures of key components of conveyors; then the present invention adds a curriculum learning method to the adversarial network GAN model, first identifying the global and local features of fault images in simple tasks, then gradually introducing complex task image features, and combining adversarial loss to comprehensively optimize the objective function, continuously optimizing the generator and discriminator parameters, thereby generating high-quality roller infrared images. The present invention can effectively solve the generation of small samples of infrared images under weak fault features, improve the authenticity and diversity of generated images, and enhance the stability and generalization ability of the model.

[0105] Generally speaking, the present invention includes the following four parts:

[0106] 1. Multi-fault simulation device:

[0107] In order to better understand and prevent the early warning of failure of key components of conveyors, the present invention designs a set of fault simulation devices that can simulate different degrees of roller jamming. The roller fault simulation device allows users to set the running resistance of the roller according to their needs through adjustable friction plates and bolt configurations.

[0108] 2. Image acquisition module:

[0109] This module uses an infrared camera to continuously capture infrared images of the three rollers and the conveyor belt. The infrared images can reflect in detail the temperature distribution of the rollers and conveyor belt during operation, because abnormal operating conditions of the rollers and conveyor belt (such as overheating or increased friction) will cause changes in their temperature distribution.

[0110] 3. Image preprocessing and data enhancement:

[0111] The captured infrared images were preprocessed, including image denoising, contrast enhancement, and normalization, to ensure that the image data met the input requirements for subsequent processing. At the same time, image enhancement was implemented using the open source Albumentations package, and a combination of different data enhancement techniques, namely: vertical flip, 90-degree random rotation, horizontal flip, and transposition, were applied to the thermal image dataset.

[0112] 4. Small sample generation of complex task images:

[0113] The CDA-GAN model is constructed to optimize the detailed performance of the generated image by learning the global and local temperature distribution and label distribution. First, the temperature histogram is calculated and smoothed to understand the overall characteristics, and then the details of the high-temperature area are detected and optimized. The SLIC algorithm is used to segment the roller thermal image into super pixels, and the features and label distribution are calculated. Through adversarial training (GAN) and curriculum domain adaptation learning (CDA), the parameters of the generator and discriminator are updated alternately, and the objective function is optimized in combination with adversarial loss to improve the quality of the generated image and ensure that the generated image is consistent with the real data in terms of global temperature and label distribution. Finally, the training ends when the loss of the discriminator and generator tends to be stable.

[0114] Among them, by calculating the global temperature distribution and probability density function (PDF) of each roller infrared image, the infrared image generated by GAN is made more realistic in temperature distribution and used as prior knowledge for the generator image. By learning the distribution of local high temperature areas, defining the high temperature threshold, detecting the high temperature area in the image, using the sliding window technology, dividing the image into multiple small areas, calculating the high temperature characteristics of each small area, calculating the temperature distribution of the high temperature area, and in the training process of the generative adversarial network (GAN) model, combining the loss function of the local high temperature area distribution, optimizing the parameters of the generator and the discriminator. Learning the local landmark area label distribution, using the SLIC algorithm to segment the feature area image of the roller thermal image abnormality into multiple super pixels, and calculating the features of each super pixel, including temperature features, shape features and position features, combined with the global label distribution and local label distribution, to ensure that the generated image is consistent with the real data in global and local features.

[0115] Specifically, if Figure 1 As shown, the present invention mainly comprises the following steps:

[0116] Step 1: Design a roller fault infrared image simulation device.

[0117] like Figure 2 As shown, a set of adjustable friction plates and bolt configurations includes a stuck ring 1, rollers 2, brackets 3, angle brackets 4, support plates 5, friction plates 6 with adjustable thickness, and bolts 7 with variable tightening degrees. This embodiment takes three rollers as an example, and places rollers 2 in the stuck ring 1. By adjusting the thickness of the friction plates and the tightening degree of the bolts, roller failures of different degrees from slight to complete stuck are simulated. This design allows the device to accurately set the running resistance of the rollers according to actual needs, and facilitates the simulation and research of various roller stuck and different stuck degree failures.

[0118] The layout and angle of the infrared camera are then optimized to ensure that the thermal image anomaly caused by the stuck roller can be accurately captured. The camera system is configured for the typical thermal distribution when the fault occurs to enhance the accuracy and response speed of fault detection. The specialized infrared camera system can continuously monitor and record infrared images to provide data support for subsequent fault analysis.

[0119] Step 2: Define the course learning task hierarchy.

[0120] According to the roller fault simulation device designed earlier, the method of gradually introducing complex tasks requires detailed definition and implementation of each task to ensure that the model can effectively learn and improve its capabilities at each step. The definition of the task of the course domain learning method for the thermal images of rollers with different degrees of stuck is as follows:

[0121] (1) Simple task: Use thermal images of completely stuck rollers for training. Due to the obvious degree of fault, large temperature difference, clear features, and consistent fault mode, thermal images of completely stuck rollers usually have consistent fault modes, and the fault feature areas have certain regularity, which enables the model to capture the commonality of the patterns more quickly in the initial learning stage, helping the model to quickly learn basic fault features.

[0122] (2) Complex tasks: Introduce thermal images of other stuck rollers. The fault features in these images are more subtle, the temperature anomaly is smaller, the temperature distribution area is irregular, the fault features are not prominent, and the recognition is difficult. However, through the early training, the model has a certain fault recognition ability and can better cope with complex situations.

[0123] Here, the infrared images of rollers with different faults are defined in the course domain learning task hierarchy. The infrared images of completely stuck rollers are defined as simple course learning tasks, and the infrared images of slightly stuck or semi-stuck rollers are defined as complex course learning tasks.

[0124] In this embodiment, the roller jam type is mainly set to include complete jam, semi-jamming, slight jam and normal. It should be noted that the roller jam type can be set as needed, for example, adding severe jam between complete jam and semi-jamming, adding moderate jam between semi-jamming and slight jam, etc.

[0125] Step 3: Collect infrared image data sets.

[0126] The infrared camera is used to continuously capture infrared images of the three rollers. These images can reflect the temperature distribution of the rollers during operation.

[0127] (1) Collecting images of simple fault tasks: By adjusting the tightening degree of the bolts of the fault simulation device and adding a friction plate with a thick thickness and a large friction coefficient inside the semi-stuck ring, the roller is completely stuck. An infrared thermal imager is set up to collect images of the roller. 200 data are automatically collected through the infrared thermal imager.

[0128] (2) Collecting complex fault task images: The existing equipment has difficulty in collecting complex fault image data sets (slightly stuck images, semi-stuck images), and it is difficult to simulate fault types, resulting in a small number of data samples. A small number of thermal images of rollers with slight stuck and semi-stuck are collected by infrared thermal imager. The thickness is adjusted and a friction plate with a low friction coefficient is added. At the same time, rubber sheets are added to the two halves of the stuck ring and tightened by bolts to achieve complex fault image simulation.

[0129] Step 4: Image preprocessing and data enhancement.

[0130] (1) Image preprocessing: Strong noise, uneven brightness, and poor contrast are common characteristics of thermal images taken in harsh environments. In order to solve the above problems, the original thermal images need to go through different preprocessing steps, and the infrared images of obvious faults (such as complete roller jamming faults) and a small number of weak faults (such as slight jamming and semi-jamming faults) are preprocessed separately.

[0131] In order to reduce the complexity of the captured scene and improve the accuracy of the segmentation results, the color thermal image is converted into a grayscale 8-bit image as follows:

[0132]

[0133] The intensity of the RGB channels in the extracted hot frame can be described as , , .

[0134] Gaussian smoothing is performed on the grayscale image as follows:

[0135]

[0136]

[0137] In the formula, represents the image after Gaussian smoothing, represents the Gaussian filter kernel, represents the standard deviation of the Gaussian kernel, Represents a convolution operation.

[0138] Then apply the Laplacian operator to calculate:

[0139]

[0140] In the formula, represents the image after Laplace transformation, represents the Laplace operator.

[0141] In order to reduce the computational burden and improve the detection results, a 256-pixel region of interest on a 256× grayscale thermal frame is defined. In order to reduce the seasonal variation in the extracted frame and define statistical parameters applicable to different temperature ranges, the present invention Normalize the grayscale image, normalize the temperature value of the image to a fixed range, and adjust all images to a uniform size to ensure data consistency. The formula is as follows:

[0142]

[0143] In the formula, refers to the average value, while Refers to the variance value of the processed grayscale thermal image.

[0144] (2) Image data enhancement: The purpose of data enhancement is to increase the model's adaptability to different heat distributions, temperature ranges, and background conditions, and to increase the number of samples used to train and test the model. This can alleviate model overfitting and improve the generalization ability of the trained model. Image enhancement was implemented using the open source Albumentations package, and a combination of different data enhancement techniques, namely vertical flip, 90-degree random rotation, horizontal flip, and transpose, were applied to the thermal image dataset.

[0145] Step 5: Learn simple tasks.

[0146] By learning the global temperature distribution and the distribution of local high-temperature areas, the detail expression and feature capture capabilities of the generated images are improved.

[0147] (1) Learning the global temperature distribution: Learn the global temperature distribution of each roller infrared image in order to gain a preliminary understanding of the overall characteristics of the image. In the generative adversarial network (GAN) model, by learning and understanding the global temperature distribution, the infrared images generated by GAN can be made more realistic in temperature distribution. As a priori knowledge, the global temperature distribution can guide the generator to generate images that conform to the expected temperature distribution, which helps to improve the diversity and quality of the generated images under small sample conditions. The generated images will be more consistent with the temperature characteristics of real infrared images. At the same time, the learning of the global temperature distribution can provide a stable training target, making the GAN model more stable during the training process and reducing the occurrence of mode collapse. The structure diagram of the simple task learning is shown in the figure below. Figure 3 shown.

[0148] a) Define global temperature distribution: Define the global temperature distribution of image I ,in It's the temperature.

[0149] b) For each image, calculate the temperature histogram :The range of setting temperature value is [ , ], dividing the temperature value into intervals, each interval has a width of ,as follows:

[0150]

[0151] c) Temperature distribution: Construct a histogram, record the number of pixels in each temperature interval, and define the global temperature distribution of image I as ,in is the temperature. For each image, calculate the temperature histogram , represents the temperature distribution, and the formula is as follows:

[0152]

[0153] in is the Dirac function, Indicates when Falling in It takes 1 if the interval is 0, otherwise it takes 0. express i and j The temperature value at the pixel location.

[0154] d) Probability Density Function (PDF): The probability density function allows the temperature distribution between different images to be directly compared, eliminating the influence of image size and total number of pixels. PDF provides a statistical description of the temperature distribution, which can be used for further statistical analysis and feature extraction. Convert the histogram to a probability density function:

[0155]

[0156] In the formula, Indicates The probability density of the temperature interval.

[0157] e) Smoothing and fitting: Gaussian smoothing is used, and the formula is as follows:

[0158]

[0159] in, is the standard deviation of the Gaussian kernel.

[0160] f) Calculate the smoothed global temperature distribution to provide a smooth, continuous temperature distribution curve, which is helpful for further statistical analysis and feature extraction. Calculate the smoothed global temperature distribution of the real image :

[0161]

[0162] In the formula, Indicates o The temperature value of each pixel, O Represents the total number of image pixels; Represents the Gaussian kernel function, which is used to adjust the temperature offset Different weights are assigned to achieve smoothing. The temperature offset and Gaussian kernel function are introduced to perform weighted smoothing on the temperature values, reduce the impact of noise and local outliers, and thus obtain a smoother temperature distribution that better reflects the global trend. This is very important when processing temperature data (such as thermal imaging images) and helps improve the accuracy and reliability of the analysis results.

[0163] (2) Learning the distribution of local high temperature areas: In the generative adversarial network (GAN) model, this process can improve the detail expression and feature capture capabilities of generating small sample images. By minimizing the difference in local temperature distribution between real images and generated images, the generator's detail generation capability can be further improved. For images with complex local features, learning the distribution of local high temperature areas can optimize the processing capabilities of GAN and enhance its adaptability to complex scenes.

[0164] a) Local high temperature area detection: Calculate the average temperature of each image and temperature standard deviation :

[0165]

[0166]

[0167] in, is the number of pixels in the image, It is The temperature value of a pixel.

[0168] b) Define high temperature threshold: Set the high temperature threshold Used to mark high temperature areas:

[0169]

[0170] in, is a constant used to adjust the high temperature threshold.

[0171] c) Segmentation using sliding windows: Divide the image into multiple small regions, e.g. The coordinate range of the sliding window is: ,For each small area detection whether it belongs to the high temperature area, the high temperature threshold is applied to each image, a high temperature mask is generated, and all pixels above the threshold are marked.

[0172]

[0173] Learn the characteristics of the local high temperature area in each window:

[0174] ① Maximum temperature in the window : ,in It is The collection of pixels in a window.

[0175] ②Average temperature in the window : ,

[0176] ③Temperature variance within the window :

[0177] d) Calculation window The medium temperature is ,and Temperature distribution in the high temperature area: In each window In the formula The calculated temperature is The distribution of high temperature areas is obtained through the Dirac function and the high temperature mask matrix The temperature in the high temperature area is The number of pixels makes the temperature distribution between different windows comparable. The local temperature distribution formula is as follows:

[0178]

[0179] in is the Dirac function, is the temperature at a certain pixel point. The value is 1 when the temperature is The number of pixels. If The pixel belongs to the high temperature area; otherwise .

[0180] Then the local features are smoothed And perform probability density function (PDF) conversion, in each temperature range Construct a smoothing window ranging from arrive , use Gaussian smoothing to assign different weights to each temperature value in the local smoothing window, and perform smooth weighted summation to make the temperature distribution of the local high temperature area more continuous and smooth, and obtain the local high temperature area distribution of the real image :

[0181]

[0182] In the formula, represents the half width of the smoothing window, Indicates the temperature Offset on foundation The temperature value, is the Gaussian kernel function, is the standard deviation of the Gaussian kernel.

[0183] (3) Model training: Combine the losses of global and local high temperature area distribution and use these losses to optimize the generator in a generative adversarial network (GAN). Use the trained model to generate new infrared images, which should be similar to real images in terms of global temperature and local temperature distribution. In a generative adversarial network (GAN), the generator generates images and the discriminator is used to evaluate the authenticity of the images.

[0184] a) Calculate the global and local temperature distributions of the simple task image generated by the generator G:

[0185] Generate image global temperature distribution :

[0186]

[0187] Generate image local temperature distribution :

[0188]

[0189] b) Define global and local loss functions:

[0190] Calculate the global high temperature area distribution loss:

[0191] Calculate the local high temperature area distribution loss:

[0192] Calculate the temperature variance loss of the local high temperature area in each window:

[0193] c) Adversarial loss: During the training of the GAN model, the generator and discriminator are optimized by combining the loss function of the local high temperature area distribution. The adversarial loss includes the loss of the generator and the discriminator:

[0194] ① Loss function of the discriminator:

[0195] ② Comprehensive loss function of the generator: combining global high temperature area distribution loss, local high temperature area distribution loss and generation adversarial loss, local high temperature area temperature variance loss:

[0196]

[0197] In the formula, represents the loss of the generator; , and are the weight coefficients of global temperature distribution loss, local temperature distribution loss and temperature variance loss in local high temperature area, respectively.

[0198] Iterative training: By repeatedly training the discriminator and generator, the two models are continuously optimized until the loss function converges. In this way, the simple task images generated by the generator will become more and more realistic and closer to the real images in temperature distribution.

[0199] Step 6: Gradually introduce complex tasks.

[0200] like Figure 4 As shown in the figure, after using infrared images of completely stuck rollers for simple task learning, infrared images of other types of stuck rollers are used for complex task learning. Complex tasks are gradually introduced here, and the quality of data generated by the model is optimized by learning the global label distribution and the local landmark area label distribution.

[0201] (1) Learning global labels: Global labels are used to describe the main feature areas in the image. The frequency of each label is calculated by summing the labels of all pixels. In roller fault detection, the global label c categories are as follows: normal rollers are marked as 0, slightly stuck rollers are marked as 1, semi-stuck rollers are marked as 2, and completely stuck rollers are marked as 3.

[0202] Then calculate the label distribution : When calculating the label distribution, the Dirac function Used to count the frequency of each label in the image.

[0203] Define the global label distribution of the input image I:

[0204] in, It is The label of pixels, is the Dirac function, when When the label is 1, it is 0 otherwise. The label distribution pattern is learned through the classification model: the roller infrared image data and the corresponding labels are divided into a training set and a test set, and the multi-layer perceptron MLP classification model is selected. The classification model is trained with the training set data, and the performance of the model is verified with the test set data. The trained model is used to predict the global label distribution of the new image generated by the generator.

[0205] The multi-layer perceptron MLP classification model includes:

[0206] Input layer: The input layer receives the pixel values ​​of the image, and each input node corresponds to a pixel value. Assume that the image size is , then the input layer has nodes.

[0207] Hidden layer: The hidden layer is calculated by weighted summation and activation function. Assume that The output of the hidden layer nodes is ,but:

[0208] in, It is The value of the input node, Is an input node To the hidden layer nodes The weight of is a hidden layer node The bias, is the activation function ( Activation Function

[0209] Label prediction output layer: The feature vector of each pixel is input into the MLP model, and the probability distribution of the output layer is obtained through forward propagation. Each node in the output layer corresponds to a category, and the output value is obtained through The function calculates the probability distribution and selects the category with the largest output probability as the predicted label of the pixel .

[0210]

[0211] in, is the model's prediction that the nth sample belongs to the category The probability of is the jth component of the network output vector z, and C is the total number of categories.

[0212] Model training: In order to measure the difference between the predicted probability distribution and the true distribution, the cross entropy loss function is used to train the classification model:

[0213] in, is the sample size, is the number of label categories, is the true label of the class to which the sample belongs (One-hot encoding, if the sample belongs to this category, then =1, otherwise =0), is the model's prediction that the nth sample belongs to the category probability.

[0214] The model parameters are optimized by gradient descent to minimize the loss function and the weight update formula , bias update formula :

[0215]

[0216]

[0217] in, is the learning rate, is the loss function over weights The gradient of is the value of the i-th input node of the n-th sample, is the true label of the jth category of the nth sample.

[0218] The present invention also introduces pre-training learning in the multi-layer perceptron MLP classification model. In order to reduce the training time and improve the performance of the classifier, the pre-trained model of ResNet34 is used as the feature extraction model to transfer the image features from the source domain (ADASDataset) to the target domain (weak fault roller dataset). The common features between the two domain images are explored, and the parameters of the source domain model are transferred to the target domain model. Then, the model is trained for feature transfer using the enhanced dataset of the roller for a specific target task.

[0219] (2) Learning local landmark region labels: SLIC is used to segment the characteristic region image of the roller thermal image anomaly into multiple superpixels, each of which represents a local region in the image.

[0220] Calculate the features of each superpixel: For each superpixel , calculate its temperature characteristics, shape characteristics and position characteristics.

[0221] a) Temperature characteristics: reflects the average temperature in the super pixel area and captures the thermal characteristics of the area. Average temperature The calculation formula is:

[0222] in, Super Pixel The number of pixels in .

[0223] b) Shape features: Shape features (such as area) can describe the size of superpixels. Different types of regions (such as normal regions and faulty regions) may have significant differences in shape. Shape features can assist the model in classification and recognition, helping the model understand the scale of different regions in the image. Superpixels Area The calculation formula is: .

[0224] c) Positional features: Positional features (such as centroid position) can describe the spatial distribution of superpixels in an image and provide the positional relationship of regions. This is very important for capturing the positional relationship between different regions in an image. The centroid The calculation formula is: .

[0225] Calculate and divide super pixel feature labels based on the above features (true image label) or (Generate image labels):

[0226] a) Define temperature feature labels: normal is marked as 0, low temperature area is marked as 1, medium temperature area is marked as 2, and high temperature area is marked as 3; temperature feature labels as follows:

[0227]

[0228] in, , , is the set average temperature threshold.

[0229] b) Define shape feature labels: small area label is 4, medium area label is 5, large area label is 6, as follows:

[0230]

[0231] in, , , The shape area size threshold is set.

[0232] c) Define the region position feature labels: 7 for the center region, 8 for the edge region, and 9 for the corner region, as follows:

[0233]

[0234] The label distribution is calculated for each superpixel. The core of the SLIC algorithm is a clustering-based method that clusters the pixels in the image according to color and position:

[0235]

[0236] in, Indicates Super pixels.

[0237] The clustering-based SLIC algorithm clusters the pixels in the image into multiple super pixels, and finds the optimal super pixel segmentation scheme by minimizing the objective function D, so that each super pixel is as consistent as possible in color and space. The objective function of the SLIC algorithm is:

[0238]

[0239] in, It is the color distance based on temperature characteristics, used to measure the color value of the pixel The center color value of the super pixel The difference between It is a spatial distance based on position features, used to measure the position of pixels. Super pixel center position The difference between is a balance parameter between the color space distance and the position space distance, is the size of the superpixel.

[0240] Step 7: Build the CDA-GAN model.

[0241] Construct a CDA-GAN model, combine adversarial training and curriculum domain adaptation learning (CDA) methods, combine adversarial loss and optimized objective functions, optimize the parameters of the generator and discriminator, optimize the quality of model-generated data, ensure that the generated images are consistent with the real data in global temperature distribution and label distribution, and improve the stability and generalization ability of the model.

[0242] Here, the CDA-GAN network is used to realize the generation of small samples of roller infrared images. The generator generates images from random noise, and the discriminator judges the authenticity of the image. On the basis of the traditional GAN ​​network, the course domain learning strategy is used to help the model gradually adapt to the complexity of the generation task and continuously optimize the parameters of both. It can not only distinguish between real images and generated images, but also learn and predict the global label distribution and local label distribution of images, so as to generate images that conform to the real label distribution and improve the quality of generated images. Figure 5 shown.

[0243] (1) Generator ( ): Initialize the generator , input random noise vector , output composite infrared images of rollers for complex tasks . The 100-dimensional random noise The generator model is constructed by taking condition (c) (a small number of images with fault feature labels) as input. The noise is converted into an 8192-dimensional vector through a fully connected layer, and then converted into a 4×4×512 feature map through a reshape function. It is upsampled through 1-4 layers of deconvolution to finally generate 64×64×3 images. Each deconvolution is followed by batch normalization and ReLU activation function. The output layer uses the Tanh activation function to map the output to the range of [-1,1]. The generator model structure is as follows: Figure 6 shown.

[0244] (2) Discriminator D): The 64×64×3 images generated by the generator and condition (c) (a small number of images with weak fault features, input of real slightly stuck and semi-stuck roller infrared images) are used as input to form a discriminator model. After downsampling using convolutional layers 1-4, the 64×64×3 images are converted into 4×4×512 feature maps. Each convolutional layer is followed by a Leaky ReLU activation function. The 4×4×512 feature map is reshaped into a 4×4×512-dimensional vector in the reshaping layer. The 4×4×512-dimensional vector is converted into a classification probability vector through a fully connected layer. The softmax activation function is used to output the category probability to distinguish the probability values ​​of normal rollers, slightly stuck rollers, semi-stuck rollers, and completely stuck rollers. The model structure of the discriminator is as follows: Figure 7 shown.

[0245] (3) Label distribution calculation:

[0246] a) Global label distribution calculation: The generated new images are input into the trained MLP model to calculate the global label distribution of each generated image in the complex task And the global label distribution of the input real image :

[0247]

[0248]

[0249] in, Indicates that the global pixel label is slightly stuck or semi-stuck in complex tasks. is the label of the i-th pixel in the real image, is the label of the i-th pixel in the generated image.

[0250] Construct global label distribution loss: Substitute the above global label distribution into the cross entropy loss function:

[0251]

[0252] b) Local label distribution calculation: According to the super pixel feature label or , represents the temperature, shape and position characteristics of the super pixel. For each super pixel Calculate its label distribution and count super pixels The frequency of each label in the image is used to generate the local label distribution of the image. , local label distribution of real images The calculation formula is:

[0253]

[0254]

[0255] in, Indicates the label, is the Dirac function, when The value is 1 when it is set, otherwise the value is 0. is the target label in the complex task image.

[0256] The loss function of the label distribution of the local landmark area is:

[0257] (3) Adversarial training: Alternately update the parameters of the generator and discriminator, update the discriminator (fix the generator, minimize the loss of the discriminator ), update the generator (fix the discriminator, minimize the loss of the generator ), continuously optimize the model through adversarial training and curriculum learning, and continuously improve the model capabilities and the quality of small sample image generation.

[0258] a) Loss function of the generator :The goal of the generator is to minimize the loss function, that is, to maximize the deception of the discriminator so that it thinks the generated data is real. The adversarial loss function of the generator as follows:

[0259]

[0260] Generator comprehensive loss :

[0261] in is a weight parameter used to balance the adversarial loss and label distribution loss.

[0262] b) Loss function of the discriminator:

[0263] ①Fighting losses :

[0264] ②Label distribution loss of the global discriminator : Calculate the probability distribution of the global label in the real data as the target label c , the discriminator predicts the probability that the real data x belongs to the global label c , the discriminator generates data The predicted probability of belonging to the global label c , the calculation formula is as follows:

[0265]

[0266] ③Label distribution loss of local discriminator , respectively calculate the probability distribution of the target local label c in the real data , the discriminator predicts the probability that the real data x belongs to the local label c , the discriminator generates data The predicted probability of belonging to local label c , the calculation formula is as follows:

[0267]

[0268] ④ Comprehensive discriminator loss function:

[0269] in is the weight parameter of the comprehensive discriminator loss function. In complex task learning, the generator and discriminator are trained using infrared images of other preset stuck types of rollers other than completely stuck.

[0270] Step 8: Iterate training.

[0271] Through iterative training, the model capability and the quality of small sample image generation are continuously improved to generate realistic infrared images. The optimizer updates the parameters of the generator and discriminator alternately based on the gradient information, and continuously optimizes the model through adversarial training and curriculum learning, continuously improving the model capability and the quality of small sample image generation.

[0272] Observe the loss functions of the discriminator and generator. If the losses of both tend to stabilize within several rounds and no longer decrease significantly, it can be considered that the model has converged and the training can be ended.

[0273]

[0274] In summary, the present invention provides a conveyor roller multi-fault simulation device and a method for generating small sample infrared images of rollers. By combining the generative adversarial network (GAN) and the course domain learning method (CDA-GAN), the problem of infrared image generation under small sample conditions is solved, the stability and generalization ability of the model are improved, and a new technical means is provided for roller fault detection. The present invention can simulate three rollers stuck to different degrees, monitor the temperature distribution of the rollers in real time through an infrared camera, and use the generative adversarial network (GAN) and the course domain learning (CDA-GAN) method to generate high-quality infrared images of rollers, thereby improving the accuracy and efficiency of fault detection.

[0275] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0276] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0277] It will be easily understood by those skilled in the art that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating small samples of infrared images of conveyor rollers, characterized in that: The method uses a generative adversarial network model including a generator and a discriminator, and includes two stages: a simple task learning stage and a complex task learning stage; In the simple task learning stage, only infrared images of completely stuck rollers are used to learn the global temperature distribution and local high temperature area distribution of each infrared image of completely stuck rollers, and the generative adversarial network model is trained by combining the losses of the global temperature distribution and the local high temperature area distribution; In the complex task learning stage, infrared images of other preset stuck rollers are used to learn the global label distribution and local landmark area label distribution of each infrared image of other preset stuck rollers, and the generative adversarial network model is trained by combining the losses of the global label distribution and the local landmark area label distribution; The trained generative adversarial network model is used to generate small samples of infrared images of conveyor rollers; The learning of global temperature distribution includes: constructing a temperature histogram of the infrared image to represent the global temperature distribution of the infrared image; converting the temperature histogram into a probability density function and performing Gaussian smoothing to obtain a global temperature distribution function after Gaussian smoothing. ; Calculate the global temperature distribution of the real image : In the formula, Indicates o The temperature value of each pixel, O Represents the total number of image pixels; Indicates temperature range; Represents the Gaussian kernel function, which is used to adjust the temperature offset Assign different weights to achieve smoothing; Learning the distribution of local high temperature areas, including: defining a high temperature threshold, generating a high temperature mask based on the high temperature threshold; using a sliding window to divide the infrared image into multiple small areas , calculate the local high temperature area distribution , the temperature in the local high temperature area is Number of pixels: In the formula, Indicates The set of pixels in a window; m and n express The middle pixel position, Indicates the temperature value at the corresponding position; is the Dirac function, when The value is 1 when it is, otherwise it is 0; represents the high temperature mask at the corresponding position; Smooth out local high temperature areas And perform probability density function conversion, in each temperature range Construct a smoothing window ranging from arrive , use Gaussian smoothing to assign different weights to each temperature value in the local smoothing window, perform smooth weighted summation, and obtain the local high temperature area distribution of the real image : In the formula, represents the half width of the smoothing window, Indicates the temperature Offset on foundation The temperature value, is the Gaussian kernel function, is the standard deviation of the Gaussian kernel; Learning global label distribution, including: taking the type of roller jam as the global label, determining the global label of all pixels in the infrared image; calculating the global label distribution of the real image : In the formula, i Indicates i pixels, N Represents the total number of image pixels, Indicates i The global label of pixels, c Indicates c Class global label; Indicates when For the c It takes 1 if it is a global label, otherwise it takes 0; Learning the distribution of local landmark area labels includes: using SLIC to segment the infrared image into multiple superpixels, each superpixel represents a local area in the image; calculating the temperature feature, shape feature and position feature of each superpixel in the image; the temperature feature is the average temperature of the superpixel, the shape feature is the area of ​​the superpixel, and the position feature is the centroid of the superpixel; defining the temperature feature label, shape feature label and position feature label of the superpixel; using the temperature feature label, shape feature label and position feature label as local labels; Computing local landmark region label distribution of infrared images : In the formula, Indicates j local tags, Indicates the image m super pixels, M represents the total number of super pixels in the image, Indicates when The local label is It takes 1 when it is, otherwise it takes 0.

2. The method for generating a small sample of infrared images of conveyor rollers according to claim 1 is characterized in that: Learn the global temperature distribution, including: The range of setting temperature value is [ , ] and is divided into K intervals, each of which has a width of , as shown below: Constructing a temperature histogram of an infrared image , represents the global temperature distribution of the infrared image, and the formula is as follows: In the formula, i and j represents the pixel position, represents the temperature value at the corresponding position, M and N Indicates the size of the infrared image; Indicates when Falling in Temperature range Take 1 when it is, otherwise take 0; The temperature histogram Convert to probability density function : right Gaussian smoothing is performed, and the formula is as follows: In the formula, represents the global temperature distribution after Gaussian smoothing, represents pi, represents a natural constant, represents the standard deviation of the Gaussian kernel; Calculate the global temperature distribution of the real image : In the formula, Indicates o The temperature value of each pixel, O Represents the total number of image pixels; Represents the Gaussian kernel function, which is used to adjust the temperature offset Assign different weights to achieve smoothing.

3. The method for generating a small sample of infrared images of conveyor rollers according to claim 2 is characterized in that: Study the distribution of local high temperature areas, including: Calculate the average temperature of an infrared image and temperature standard deviation , define the high temperature threshold : In the formula, is a constant used to adjust the high temperature threshold; Based on high temperature threshold Generate high temperature mask : In the formula, express i and j The temperature value at the location; Use sliding windows to divide the infrared image into multiple small areas , and calculate the local high temperature area Temperature variance ;in, Indicates The set of pixels in a window; Calculate the local high temperature area distribution , the temperature in the local high temperature area is Number of pixels: In the formula, m and n express The middle pixel position, Indicates the temperature value at the corresponding position; is the Dirac function, when The value is 1 when it is, otherwise it is 0; represents the high temperature mask at the corresponding position; Smooth out local high temperature areas And perform probability density function conversion, in each temperature range Construct a smoothing window ranging from arrive , use Gaussian smoothing to assign different weights to each temperature value in the local smoothing window, perform smooth weighted summation, and obtain the local high temperature area distribution of the real image : In the formula, represents the half width of the smoothing window, Indicates the temperature Offset on foundation The temperature value, is the Gaussian kernel function, is the standard deviation of the Gaussian kernel.

4. The method for generating a small sample of infrared images of conveyor rollers according to claim 3 is characterized in that: The generative adversarial network model is trained by combining the loss of global temperature distribution and local high temperature area distribution, including: Global temperature distribution of the image generated by the corresponding calculation generator , Local high temperature area distribution And local high temperature areas Temperature variance ; Define global temperature distribution losses : Definition of local high temperature area distribution loss : And the temperature variance loss in the local high temperature area: Constructing a comprehensive loss function for the generator : In the formula, represents the loss of the generator; , and is the weight coefficient; Directly construct the discriminator loss , the generator and discriminator are trained using only infrared images of completely stuck rollers.

5. The method for generating a small sample of infrared images of conveyor rollers according to claim 1, characterized in that: The temperature feature labels of super pixels include normal, low temperature area, medium temperature area and high temperature area; the shape feature labels of super pixels include small area, medium area and large area; the position feature labels of super pixels include center area, edge area and corner area.

6. The method for generating a small sample of infrared images of conveyor rollers according to claim 5, characterized in that: The generative adversarial network model is trained by combining the loss of global label distribution and local landmark region label distribution, including: The global label distribution of the images generated by the corresponding calculation generator And local landmark area label distribution ; Define the loss of the global label distribution : In the formula, C represents the total number of global labels; Defining local landmark region label distribution loss : In the formula, J Represents the total number of local labels; Constructing a comprehensive loss function for the generator : In the formula, represents the loss of the generator, and represents the weight coefficient; Constructing the label distribution loss of the global discriminator : In the formula, Represents a real image x The global label in is the target global label c The probability distribution of Represents the discriminator for the real image x Belongs to the global tag c The predicted probability of Represents the image generated by the discriminator to the generator Belongs to the global tag c The predicted probability of Represents the expectation of real image data; Represents the expectation of generating image data; Constructing the iconic region label distribution loss of the local discriminator : In the formula, M is the total number of super pixels in the image, Represents a real image x In local area The local label is the target local label j The probability distribution of Represents the discriminator for the real image x In local area The local label is the target local label j The predicted probability of Represents the image generated by the discriminator to the generator In local area The local label is the target local label j The predicted probability of Constructing a comprehensive loss function for the discriminator : In the formula, represents the loss of the discriminator; and is the weight coefficient; The generator and discriminator are trained using infrared images of other preset stuck types of rollers.

7. The method for generating a small sample of infrared images of conveyor rollers according to claim 1, characterized in that: Other preset freeze types include semi-stuck, slight-stuck, and normal.

8. The method for generating a small sample of infrared images of conveyor rollers according to claim 6, characterized in that: The method further includes: Collect infrared images of rollers with different stuck types and perform image preprocessing, including: converting color infrared images into grayscale images, performing Gaussian smoothing on the grayscale images, performing calculations using the Laplace operator, and finally normalizing them; Perform image data enhancement on the preprocessed infrared images, including vertical flipping, 90-degree random rotation, horizontal flipping, and transposition; Use the multi-layer perceptron MLP classification model to predict the global label of the image; including: The global label of the infrared image of the roller is obtained, and the multi-layer perceptron MLP classification model is trained. The trained multi-layer perceptron MLP classification model is used to predict the global label of the image generated by the generator, and then the global label distribution of the image generated by the generator is calculated.

9. A conveyor roller fault simulation device, applied to the conveyor roller infrared image small sample generation method according to any one of claims 1 to 8, characterized in that: The device comprises a roller, a locking ring, a friction plate with adjustable thickness and a bolt with variable tightening degree; The locking ring includes two semicircles, and the two ends of the semicircles are connected by bolts with variable tightening degrees; the roller is located inside the locking ring; and the friction plate with adjustable thickness is located between the roller and the locking ring; The thickness of the friction plate with adjustable thickness and the tightening degree of the bolts with variable tightening degree are adjusted to simulate different degrees of roller jamming failure.

Images