Tiny foreign matter detection sample enhancement method for belt conveyor

Abstract

The invention relates to a small foreign matter detection sample enhancement method for a belt conveyor, and belongs to the technical field of image processing, and the method comprises the following steps: S1, collecting a foreign matter sample in a coal flow during the operation of the belt conveyor, marking a large target and a small target in the foreign matter sample, and carrying out the statistics of the size distribution of the small target; s2, cutting the large target picture, rotating, reducing and overturning the cut part, and resetting the target position to form an incomplete image; s3, constructing a generative adversarial network, wherein the generative adversarial network comprises a generator and a discriminator; the generator is used for generating a repaired image, and the discriminator is used for judging a real image and the repaired image; s4, training a generator by using the incomplete image, and adjusting and optimizing the generator according to a judgment result of the discriminator; and S5, repairing the small foreign matter detection sample of the belt conveyor by using the optimized generator.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a sample enhancement method for detecting tiny foreign objects on a belt conveyor. Background Art

[0002] The main coal flow transportation in coal mines mainly adopts belt conveyors, which have the characteristics of long transportation distance and large coal flow rate. The presence of non-coal foreign matter in the transportation process can easily lead to accidents such as belt tearing and fire, which seriously threatens the safe production of coal mines and can easily cause heavy casualties and economic losses.

[0003] Since the main coal flow has a long transportation distance and the belt conveyor runs at a high speed, it is inefficient to use manual points to check for foreign objects. Therefore, it is important to ensure the safety of coal flow transportation by performing video analysis on key points of the belt conveyor to detect whether there are foreign objects during the operation of the belt. At present, there are few foreign object samples and the targets are small, so the commonly used target detection algorithm cannot extract the features of foreign objects and identify them well.

[0004] At present, the method based on sample cropping mainly performs operations such as scaling and rotating the target to convert the target into a rich smaller target; the method based on image fusion is to combine the target with different backgrounds to form new samples to expand the training data set. Cropping the target is a basic method to increase the number of small targets, but the light in the coal mine is low and there is dust interference. Only using cropping or image background color fusion will cause errors in restoring the underground scene. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a belt conveyor tiny foreign body detection sample enhancement method, through image processing technology, cut out the foreign body marking target in the coal flow transported by the belt conveyor, enrich the small target data, and use the generative adversarial network based on the low-light environment underground to repair the cropped image and enhance the tiny target sample.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A belt conveyor micro foreign body detection sample enhancement method comprises the following steps:

[0008] S1: Collect foreign matter samples in the coal flow when the belt conveyor is running, mark the large and small objects in the foreign matter samples, and count the size distribution of the small objects;

[0009] S2: Crop the large target image, rotate, shrink, and flip the cropped part, and reset the target position to form a defective image;

[0010] S3: construct a generative adversarial network, including a generator and a discriminator; the generator is used to generate a repaired image, and the discriminator is used to distinguish between a real image and a repaired image;

[0011] S4: using the incomplete image to train the generator, and adjusting and optimizing the generator according to the judgment result of the discriminator;

[0012] S5: Use the optimized generator to repair the belt conveyor micro foreign body detection sample.

[0013] Furthermore, the labeling of large objects and small objects in the foreign body sample includes:

[0014] Foreign body samples with a proportion less than or equal to 1% are marked as small targets, and foreign body samples with a proportion greater than 1% are marked as large targets, thereby distinguishing foreign body samples into large target data sets D big And small target dataset D small .

[0015] Furthermore, the size distribution of the statistical small targets is specifically represented by a probability density function:

[0016]

[0017] where x small Represents a small target sample, image proportion 0<a≤b≤1%, S represents the pixel area of ​​the small target.

[0018] Furthermore, the generator is based on a 6-layer convolutional neural network and uses a deconvolution method to generate image layer by layer. f ;

[0019] In the generator, the input element matrix M input , after the sparse matrix representation K of the convolution kernel is calculated, the output matrix M is generated output :

[0020] M output =M input K

[0021] In the deconvolution layer, the deconvolution operation is the inverse operation of the matrix operation process. output and K to get M input , the deconvolution process is:

[0022] M input =M output K T

[0023] The incomplete image image of the cropped and collaged small target is used as input data, and after 2 layers of convolution feature extraction and 4 layers of deconvolution operation, it is repaired into a complete image image f ;

[0024] The generator's loss function uses the cross entropy function:

[0025]

[0026] z represents the input data, G(z) is the restored image output by the generator, D(G(z)) is the discriminator's judgment result on whether the generated image is real, the real sample is 1, and the fake sample is 0; H is the cross entropy loss function, n G Represents the number of generator training samples.

[0027] Furthermore, the discriminator is based on a 4-layer convolutional neural network, and its input image is the repaired image image generated by the generator f and the real image image t , the model structure is divided into two parts for feature extraction calculation; the first part extracts image features through multiple convolutional layers and outputs the result y with a fully connected layer d1 ; The second part first uses the local binary pattern LBP to extract the texture features of the image, then uses the convolution structure to calculate, and outputs y through the fully connected layer d2 ; Combine the two parts of the output, optimize the final output, and judge the image result;

[0028] The discriminator loss function is:

[0029] L D =H(1,D(x))+H(0,D(G(z)))

[0030] D(x) is the discrimination result of the real input image, and D(G(z)) is the discrimination result of the fake input image; H is the cross entropy loss function, n D Represents the number of generator training samples, that is:

[0031]

[0032] The output in D(x) is given by y d1 and d2 Composition, set threshold to adjust weight:

[0033] D(x)=α(y d1 )+(1-α)(y d2 ),0<α≤1.

[0034] Furthermore, the LBP operator is defined as taking the central pixel of the window as the threshold within a 3*3 window and comparing it with the grayscale values ​​of the surrounding 8 pixels. If it is greater than the threshold, it is 1, otherwise it is 0; the 8 points in the neighborhood generate an 8-bit binary number, which is converted into a decimal number, namely the LBP value of the pixel.

[0035] The beneficial effects of the present invention are:

[0036] The present invention crops the large foreign object in the original image, and pastes it into the original image by image transformation, so as to form an image with the small foreign object, thereby enriching the data sample.

[0037] The present invention uses a deconvolution module to generate an image, repairs the incomplete image after cutting, and transforms the image into a training sample containing only small objects, thereby improving the sample quality.

[0038] The present invention adds an LBP texture feature extraction layer to the discriminator, extracts original features through a convolution module, combines texture features with original image features, and optimizes the image effect output by the generator.

[0039] The tiny foreign body sample enhancement method designed by the present invention utilizes the image cropping method and the generative adversarial network to improve the quantity and quality of small target samples, optimize the recognition effect and foreign body tracing of the foreign body target detection algorithm, warn of abnormal conditions of the belt conveyor, and ensure safe production in coal mines.

[0040] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 Crop schematic for large targets;

[0043] Figure 2 This is the generator structure diagram;

[0044] Figure 3 It is the structure diagram of the discriminator;

[0045] Figure 4 This is a schematic diagram of LBP value calculation;

[0046] Figure 5 The overall structure diagram of the generative adversarial network enhanced for foreign small target samples. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0049] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0050] The present invention provides a belt conveyor micro-foreign matter detection sample enhancement method. When the belt conveyor is running, there are micro-foreign matters in the coal flow, and the target detection algorithm needs to improve the recognition effect based on rich samples. Collect foreign matter samples in the coal flow, mark the positions of the foreign matters, and count the size distribution of small targets. Crop the large target image, rotate, shrink, flip and other operations on the cropped part, and reset the target position. Considering the influence of lighting, extract the image texture features of the underground belt conveyor when it is running, design the feature extraction structure and optimize the loss function, train the generative adversarial network, repair the cropped incomplete image, and form a new training sample.

[0051] Grab the image samples of the belt conveyor during transportation (empty belt, running with little coal, running with a lot of coal). Usually, there are few tiny foreign objects (small anchors, sharp metals, etc.), and the foreign object samples need to be labeled. This scheme classifies the foreign object samples into large target datasets D according to the proportion of objects less than or equal to 1% as small targets. big And small target dataset D small .

[0052] Statistical small target sample D small The size distribution of is expressed using the probability density function:

[0053]

[0054] where x small Represents a small target sample, image proportion 0<a≤b≤1%, S represents the pixel area of ​​the small target.

[0055] According to the actual size distribution of small target foreign objects, the large target x big Crop and paste the target image by reducing, rotating, resetting the position, etc. Figure 1 shown.

[0056] After shrinking the large target into a small target, it is clipped and pasted to the new position. In order to improve the training effect, the training data set retains the small target, and there will be a gap in the position of the large target in the image. Therefore, the improved generative adversarial network method is used to repair and fill the image.

[0057] This scheme designs an image generator G based on a 6-layer convolutional neural network, which uses deconvolution to generate image layer by layer. f The discriminator D is designed based on a 4-layer convolutional neural network. Taking into account the low illumination and easy confusion of colors in the mine, it is necessary to extract clear texture features for discrimination.

[0058] The model structure designed by the generator G is as follows Figure 2 As shown. Commonly used convolutional network modules, input element matrix M input , after the sparse matrix representation K of the convolution kernel is calculated, the output matrix M is generated output .

[0059] M output =M input K

[0060] In the deconvolution layer, the deconvolution operation is the inverse operation of the matrix operation process. output and K to get M input , the deconvolution process is:

[0061] M input =M output K T

[0062] The incomplete image image of the cropped and collaged small target is used as input data, and after 2 layers of convolution feature extraction and 4 layers of deconvolution operation, it is repaired into a complete image image f The loss function of the generator G uses the cross entropy function:

[0063]

[0064] z represents the input data, G(z) is the restored image output by the generator, and D(G(z)) is the discriminator's judgment result on whether the generated image is real. The real sample is 1 and the fake sample is 0. H is the cross entropy loss function, n G Represents the number of generator training samples.

[0065] The model structure designed by the discriminator D is as follows Figure 3 As shown. The input image of the discriminator is the repaired image image generated by the generator f and the real image image t , the model structure is divided into two parts for feature extraction calculation. The first part extracts image features through multiple convolutional layers and outputs the result y with a fully connected layer d1 The second part first uses LBP (local binary pattern) to extract the texture features of the image, then uses the convolution structure to calculate, and outputs y through the fully connected layer d2 . Combine the two parts of the output, optimize the model output, and judge the image results.

[0066] Due to the low light and high dust environment underground, the image background is relatively dim. It is difficult to distinguish the details of true and false images by only using the convolution structure to extract image features. LBP has significant advantages such as rotation invariance and grayscale invariance, and can extract local texture features of images. The LBP operator is defined as a 3*3 window with the center pixel of the window as the threshold, and compares the grayscale value of the surrounding 8 pixels. If it is greater than this value, it is 1, otherwise it is 0. The 8 points in the neighborhood generate an 8-bit binary number, which is converted into a decimal number, that is, the LBP value of the pixel.

[0067] like Figure 4 As shown, the binary value 01101100 is converted to decimal 108, which is the LBP value x of the point. LBP , construct the LBP matrix M LBP .

[0068] The discriminator loss function is:

[0069] L D =H(1,D(x))+H(0,D(G(z)))

[0070] D(x) is the discrimination result of the real input image, and D(G(z)) is the discrimination result of the fake input image. H is still the cross entropy loss function, n D Represents the number of generator training samples, that is:

[0071]

[0072] The output in D(x) is given by y d1 and d2Composition, considering that LBP texture features are auxiliary features, the effect should be lower than the original image features, and the threshold is set here to adjust the weight:

[0073] D(x)=α(y d1 )+(1-α)(y d2 ),0<α≤1

[0074] Based on the generator G and the discriminator D, the training method of the generative adversarial network is used to iteratively train G and the discriminator D, such as Figure 5 As shown in the figure, the foreign body targets in the original image are cropped and collaged to form a partially incomplete image. The incomplete image is used as input data to train the generator, and the image generated by the generator is the filled image. The fake image and the real image are input, and then the discriminator is used to judge the true and false images, and the discriminator is optimized. After optimizing the discriminator, the generator is further adjusted so that the generator can generate more realistic images. The generator and discriminator are iteratively optimized in this way, and the threshold is set to stop the cycle to achieve a better image generation effect.

[0075] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0076] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0077] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0078] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0079] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0080] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0081] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0082] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0083] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0084] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0085] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A belt conveyor micro foreign body detection sample enhancement method, characterized in that: The following steps are involved: S1: Collect foreign matter samples in the coal flow when the belt conveyor is running, mark the large and small objects in the foreign matter samples, and count the size distribution of the small objects; S2: Crop the large target image, rotate, shrink, and flip the cropped part, and reset the target position to form a defective image; S3: construct a generative adversarial network, including a generator and a discriminator; the generator is used to generate a repaired image, and the discriminator is used to distinguish between a real image and a repaired image; S4: using the incomplete image to train the generator, and adjusting and optimizing the generator according to the judgment result of the discriminator; S5: Use the optimized generator to repair the belt conveyor micro foreign body detection sample.

2. The method for enhancing the sample size of a belt conveyor for detecting tiny foreign bodies according to claim 1, characterized in that: The marking of large objects and small objects in the foreign body sample includes: Foreign body samples with a proportion less than or equal to 1% are marked as small targets, and foreign body samples with a proportion greater than 1% are marked as large targets, thereby distinguishing foreign body samples into large target data sets D big And small target dataset D small .

3. The method for enhancing the sample size of a belt conveyor for detecting tiny foreign bodies according to claim 1, characterized in that: The size distribution of the statistical small targets is specifically represented by a probability density function: where x small Represents a small target sample, image proportion 0<a≤b≤1%, S represents the pixel area of ​​the small target.

4. The belt conveyor micro foreign body detection sample enhancement method according to claim 1, characterized in that: The generator is based on a 6-layer convolutional neural network and uses deconvolution to generate image layer by layer. f ; In the generator, the input element matrix M input , after the sparse matrix representation K of the convolution kernel is calculated, the output matrix M is generated output : M output =M input K In the deconvolution layer, the deconvolution operation is the inverse operation of the matrix operation process. output and K to get M input , the deconvolution process is: M input =M output K T The incomplete image image of the cropped and collaged small target is used as input data, and after 2 layers of convolution feature extraction and 4 layers of deconvolution operation, it is repaired into a complete image image f ; The generator's loss function uses the cross entropy function: z represents the input data, G(z) is the restored image output by the generator, D(G(z)) is the discriminator's judgment result on whether the generated image is real, the real sample is 1, and the fake sample is 0; H is the cross entropy loss function, n G Represents the number of generator training samples.

5. The method for enhancing the sample size of a belt conveyor for detecting tiny foreign bodies according to claim 1, characterized in that: The discriminator is based on a 4-layer convolutional neural network, and its input image is the repaired image image generated by the generator f and the real image image t , the model structure is divided into two parts for feature extraction calculation; the first part extracts image features through multiple convolutional layers and outputs the result y with a fully connected layer d1 ; The second part first uses the local binary pattern LBP to extract the texture features of the image, then uses the convolution structure to calculate, and outputs y through the fully connected layer d2 ; Combine the two parts of the output, optimize the final output, and judge the image result; The discriminator loss function is: L D =H(1,D(x))+H(0,D(G(z))) D(x) is the discrimination result of the real input image, and D(G(z)) is the discrimination result of the fake input image; H is the cross entropy loss function, n D Represents the number of generator training samples, that is: The output in D(x) is given by y d1 and d2 Composition, set threshold to adjust weight: D(x)=α(y d1 )+(1-α)(y d2 ),0<α≤1.

6. The method for enhancing the sample size of a belt conveyor for detecting tiny foreign bodies according to claim 1, characterized in that: The LBP operator is defined as taking the central pixel of the window as the threshold within a 3*3 window and comparing it with the grayscale values ​​of the surrounding 8 pixels. If it is greater than the threshold, it is 1, otherwise it is 0. The 8 points in the neighborhood generate an 8-bit binary number, which is converted into a decimal number, i.e. the LBP value of the pixel.

Images