Method and device for identifying foreign matters in conveyed materials, electronic equipment and storage medium

By combining the parameterless attention mechanism SimAM module and the convolutional neural network model, a lightweight foreign object recognition model is built, which solves the problems of large amount of calculation and long processing time in foreign object recognition in transported materials, and realizes efficient identification and is suitable for the lower edge of coal mines. Equipment ensures recognition accuracy and real-timeness.

CN119091178APending Publication Date: 2024-12-06TAIYUAN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410937432.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, foreign matter identification of conveyed materials has problems such as large data calculation and long processing time, which is difficult to meet the requirements of high real-time underground coal mines, especially on edge equipment with limited computing capabilities.

Method used

The parameterless attention mechanism SimAM module is used to combine with the convolutional neural network model to build a lightweight foreign object recognition model. Through deep separable convolution and feature fusion technology, the calculation complexity is reduced and the recognition efficiency is improved.

Benefits of technology

While ensuring identification accuracy, the calculation amount and processing time are significantly compressed, the recognition speed is improved, and it is suitable for edge computing equipment, reducing conveyor belt damage and improving coal mine safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091178A_ABST
    Figure CN119091178A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to the field of coal mining, and discloses a conveyed material foreign matter identification method and device, electronic equipment and a storage medium so as to solve the problems of large calculation amount and long processing time existing in conveyed material foreign matter detection. The identification method for the foreign matter in the conveyed material comprises the steps of obtaining a historical image of the conveyed material; a foreign matter recognition model is constructed based on the historical image, a parameter-free attention mechanism SimAM module and a convolutional neural network model, and the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM; acquiring a current image of the conveyed material; preprocessing the current image; and detecting whether foreign matters exist in the current image based on the foreign matter recognition model. According to the recognition method, under the condition that certain recognition precision is guaranteed, the calculation amount is compressed, the processing speed is increased, and the processing time is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of coal mining, and more specifically, to a method, device, electronic device and storage medium for identifying foreign matter in conveying materials. Background Art

[0002] As the main energy source for the development of human society, the coal industry occupies an important position in national production. Building a multi-energy integrated supply system and accelerating energy transformation have become important tasks for current energy development. In order to reduce the negative impact of coal on the environment during mining, production and utilization, developing intelligent coal mining equipment and realizing the green, clean and efficient utilization of coal resources have become the main development direction of the current coal industry. As a key link in the coal mining process, the intelligence level of coal flow transportation greatly affects the overall energy consumption and cost of coal production. As the main transportation equipment for coal transportation, the mining belt conveyor is developing towards large-scale, intelligent and energy-saving directions with the rapid development of science and technology and the increasing demand of the coal industry. The coal transported to the ground will inevitably be mixed with some non-coal foreign matter, such as six types of foreign matter (coal gangue, anchor rods, pallets, screws, nuts, angle steels), etc. These foreign matter will not only affect the quality of coal, but also may damage the transport belt and affect normal production. The normal and healthy operating state of the conveyor belt equipment is the prerequisite for ensuring the safe operation of the belt conveyor system. Abnormal working conditions will not only cause unnecessary damage or wear to the equipment, but also increase the energy consumption of the system, bring additional safety risks and economic pressure to the production of coal mining enterprises, thus seriously affecting the green and sustainable development of the enterprise.

[0003] The invention patent with the Chinese patent publication number CN115147392A and the name of "A coal gangue detection method and device based on improved YOLOv4": By introducing the Focal loss function to deal with the problem of uneven sample recognition difficulty, the K-means++ clustering algorithm is used to optimize the initial anchor box, the five convolution operations in PANet are replaced with the CSP structure, and the pyramid structure of the hollow convolution is introduced to reduce the model parameters and realize the lightweight model. The target detection model achieved an mAP of 98.14% on the validation set and an FPS of 28.3Hz.

[0004] However, the pyramid structure and CSP structure of the dilated convolution have brought about a certain increase in computational complexity, increased the computational load of the CPU and GPU, and reduced the processing speed of the network model.

[0005] The Chinese patent publication number is CN116494232A, and the name is "Design of a coal and gangue fast sorting robot arm system based on an improved mask-rcnn algorithm". The invention patent: an algorithm for detecting and segmenting coal and gangue is implemented through an improved mask-rcnn model, and then an algorithm for calculating the depth information of the coal and gangue images collected by a binocular camera, and an algorithm for calculating the optimal grasping angle of the segmented image.

[0006] However, the two-stage algorithm model used in this method is slower than the one-stage algorithm model, and the mask-rcnn network cannot meet the high real-time requirements of underground coal mine sites. At the same time, the computational load is too heavy for edge devices with limited computing power.

[0007] Therefore, it is necessary to analyze and study how to solve the problems of large amount of data calculation and long processing time while ensuring a certain recognition accuracy for foreign body identification in conveyed materials. Summary of the invention

[0008] In view of this, the embodiments of the present application are dedicated to providing a method, device, electronic device and storage medium for identifying foreign objects in conveyed materials, so as to solve the problems of large amount of data calculation and long processing time in identifying foreign objects in conveyed materials.

[0009] In a first aspect, this specification provides a method for identifying foreign matter in conveying materials, comprising:

[0010] Obtain historical images of conveyed materials;

[0011] Based on historical images, the parameter-free attention mechanism SimAM module and the convolutional neural network model, a foreign object recognition model is constructed, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM;

[0012] Get the current image of the conveyed material;

[0013] Preprocess the current image;

[0014] Based on the foreign object recognition model, detect whether there are foreign objects in the current image.

[0015] According to the first aspect, in one possible implementation, a parameter-free attention mechanism SimAM module is embedded in a downsampling unit and a basic unit of a convolutional neural network model, wherein the downsampling unit and the basic unit are designed by depthwise separable convolution.

[0016] According to the first aspect, in a possible implementation, a foreign object recognition model is constructed based on historical images, a parameter-free attention mechanism SimAM module and a convolutional neural network model, including:

[0017] Preprocess the acquired historical images;

[0018] The pre-processed historical images are labeled and classified, and the classified historical images form a training set, a validation set, and a test set;

[0019] Based on the training set and the validation set, a foreign body recognition model is obtained;

[0020] Based on the test set, the performance of the foreign object recognition model is evaluated.

[0021] According to the first aspect, in a possible implementation, a foreign body recognition model is obtained based on a training set and a validation set, including:

[0022] Based on the training set, the pre-trained model weights are imported into the convolutional neural network model through transfer learning;

[0023] Based on the validation set, the convolutional neural network model is fine-tuned to form a foreign object recognition model.

[0024] According to the first aspect, in a possible implementation manner, preprocessing the current image includes: performing image enhancement, image denoising, image size change, and image geometric transformation on the current image;

[0025] The historical images were classified according to the ratio of 7:2:1 and randomly divided to form the training set, validation set and test set respectively.

[0026] According to the first aspect, in a possible implementation, detecting whether there is a foreign object in a current image based on a foreign object recognition model includes:

[0027] Perform feature extraction and feature reconstruction on the current image to obtain a feature map;

[0028] Fusing the feature maps to obtain a fused feature map;

[0029] Based on the model weights, the fused feature maps are detected to output the recognized results, wherein the model weights include the 3-D attention weights obtained based on the parameter-free attention mechanism SimAM module.

[0030] According to the first aspect, in a possible implementation, feature extraction and feature reconstruction are performed on the current image, including: based on a downsampling unit and a basic unit, extracting foreign body features with attention information in the current image.

[0031] In a second aspect, this specification provides a device for identifying foreign matter in conveying materials, comprising:

[0032] A first acquisition unit, used for acquiring historical images of conveyed materials;

[0033] The first processing unit is used to build a foreign object recognition model based on historical images, a parameter-free attention mechanism SimAM module and a convolutional neural network model, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM;

[0034] A second acquisition unit, used to acquire a current image of the conveyed material;

[0035] A second processing unit, used for preprocessing the current image;

[0036] The recognition unit is used to detect whether there is a foreign object in the current image based on the foreign object recognition model.

[0037] In a third aspect, this specification provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform each step in any one of the described methods.

[0038] In a fourth aspect, the present specification provides a computer-readable storage medium, the computer-readable storage medium comprising a stored program, wherein the program executes each step in any one of the described methods.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] Different from the prior art, the method for identifying foreign bodies in conveyed materials provided by the present invention obtains historical images of conveyed materials; constructs a foreign body identification model based on historical images, a parameter-free attention mechanism SimAM module and a convolutional neural network model, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM; obtains the current image of the conveyed material; pre-processes the current image; and detects whether there are foreign bodies in the current image based on the foreign body identification model. The present invention suppresses the interference of redundant information on the network by introducing the parameter-free attention mechanism SimAM module, thereby more efficiently completing specific feature extraction and reconstruction, and achieving the technical effect of compressing the amount of calculation, improving the processing speed and reducing the processing time while ensuring a certain recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A schematic flow chart of a method for identifying foreign matter in conveying materials provided for at least one embodiment of this specification;

[0043] Figure 2 A schematic diagram of the structure of a basic unit and a down-sampling unit provided for at least one embodiment of this specification;

[0044] Figure 3 A weight allocation principle diagram based on the SimAM attention mechanism provided for at least one embodiment of this specification;

[0045] Figure 4 A schematic diagram of an example network model for identifying non-coal foreign matter provided for at least one embodiment of this specification;

[0046] Figure 5 A schematic diagram of analyzing experimental results based on an example model provided for at least one embodiment of this specification;

[0047] Figure 6 A visualization diagram of generalization experiment results based on an example model provided for at least one embodiment of this specification;

[0048] Figure 7(a)-Figure 7(b) A comparison chart of the results of the example network model of the present invention and other classic network models provided for at least one embodiment of this specification;

[0049] Figure 8 A schematic diagram of a device 800 for identifying foreign matter in conveyed materials provided in at least one embodiment of the present specification. DETAILED DESCRIPTION

[0050] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of this specification shall have the common meanings understood by persons with ordinary skills in the field to which this specification belongs. The words "first", "second" and similar words used in the embodiments of this specification do not indicate any order, quantity or importance, but are only used to avoid confusion of constituent elements.

[0051] Unless the context requires otherwise, throughout the specification, "plurality" means "at least two", and "including" is interpreted as an open, inclusive meaning, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" are intended to indicate that a particular feature, structure, material or characteristic associated with the embodiment or example is included in at least one embodiment or example of the specification. The schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0052] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0053] As mentioned in the background technology, the prior art of foreign body identification in conveyed materials has problems such as large amount of data calculation and long processing time. In order to solve the above problems, the embodiments of the present application provide a method, device, electronic device and storage medium for identifying foreign body in conveyed materials.

[0054] like Figure 1 As shown, a method for identifying foreign matter in conveyed materials provided by an embodiment of the present invention may specifically include the following steps.

[0055] Step S110: Acquire historical images of conveyed materials;

[0056] Step S120: constructing a foreign object recognition model based on historical images, a parameter-free attention mechanism SimAM module and a convolutional neural network model, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM;

[0057] Step S130: acquiring a current image of the conveyed material;

[0058] Step S140: preprocessing the current image;

[0059] Step S150: Based on the foreign object recognition model, detect whether there is a foreign object in the current image.

[0060] In the above method, firstly, the historical images of the conveyed materials can be obtained through the image acquisition device (such as the coal mine inspection robot), and then the foreign body recognition model is constructed based on the historical images, the parameter-free attention mechanism SimAM module and the convolutional neural network model, and then the current image of the conveyed material is obtained, and then the current image is preprocessed, and finally, based on the foreign body recognition model, whether there is a foreign body in the current image. The image of the conveyed material contains coal blocks and non-coal foreign bodies. By introducing the parameter-free attention mechanism SimAM module, the problems of large data calculation amount and long processing time in the foreign body recognition of the conveyed material are solved while ensuring a certain recognition accuracy, thereby reducing the possibility of coal mine shutdown caused by conveyor belt damage, while ensuring the life safety of coal miners.

[0061] Among them, the introduced parameter-free attention mechanism SimAM module enables the model to focus more on the extraction of important feature information of non-coal foreign body images, which is conducive to suppressing the interference of redundant information on the network, thereby more efficiently completing the extraction and reconstruction of specific features, improving network recognition accuracy, and reducing network complexity.

[0062] SimAM is an attention mechanism based on energy function, which can derive 3-D attention weights without additional parameters. Compared with other attention mechanisms, SimAM effectively avoids the problem of increasing model parameters caused by structural adjustment, allowing the network to focus more on the extraction of important feature information of non-coal foreign matter images, thereby completing specific feature extraction and reconstruction more efficiently and improving the detection accuracy of the network model. SimAM evaluates the importance of each neuron in the network by analyzing the energy function, and its energy function is defined as follows:

[0063]

[0064]

[0065]

[0066] In the formula, t represents the target neuron, x i represents the adjacent neurons, λ represents the hyperparameter, μ and σ represent the mean and variance of the neurons in the channel respectively, M represents the number of neurons on the channel, and energy The lower the value, the higher the linear separability of the target neuron and its neighboring neurons, and the higher its importance. Then, a unique weighted value is assigned to each neuron based on the attention regulation performance in the mammalian brain, and its calculation expression is as follows:

[0067]

[0068] In the formula, X is the input feature tensor, ⊙ is the dot product operation, E is the enhanced feature tensor, all The sigmoid function is added to limit the excessive value of E, and the sigmoid function does not affect the relative importance of each neuron.

[0069] It can be understood that image acquisition can directly acquire images, or can first acquire videos and then extract clear images through operations such as frame extraction.

[0070] Preprocessing the current image may include: image enhancement, image denoising, image size change and image geometric transformation of the current image, so as to achieve effects such as image enhancement.

[0071] In at least one embodiment of the present application, the parameter-free attention mechanism SimAM module is embedded in the downsampling unit and the basic unit of the convolutional neural network model, wherein the downsampling unit and the basic unit are designed by depth-separable convolution. The parameter-free attention mechanism SimAM module can be embedded after the depth-separable convolution of each downsampling unit and the basic unit. The downsampling unit further lightweights the network model mainly by reducing the amount of parameters in the spatial dimension, and the basic unit mainly extracts features by adopting depth-separable convolution with less parameters. In logical order, the downsampling unit is processed first, and then the basic unit is processed.

[0072] like Figure 2 As shown, it is a structural schematic diagram of the basic unit and the downsampling unit according to the embodiment of the present application. The left figure is the basic unit structure, and its process is as follows: the input features are divided into two branches through feature separation (i.e. channel segmentation) operation, and the number of feature channels of each branch is half of the original number of channels; then the right branch performs two 1×1 ordinary convolutions and one 3×3 depth-separable convolution in sequence, and embeds the SimAM attention mechanism after the convolution to extract important neurons. The left branch is not processed and merged with the convolution result of the right branch; finally, the feature map is output through feature splicing (i.e. channel merging) and channel shuffling operations.

[0073] The right side of the figure shows the downsampling unit structure, and its process is as follows: the feature map is first evenly divided into two and input to the branches on both sides. After each branch passes through the 1×1 ordinary convolution and 3×3 depth-separable convolution (step size is 2) modules, the SimAM attention mechanism is embedded to extract important neurons, and then the channel merging operation is performed to merge the feature maps of the two branches, and finally the merged feature map is channel shuffled.

[0074] By designing novel basic units and downsampling units, the model is made more delicate and can better capture the subtler shape features between different foreign objects. It can also fully learn more detailed information features of non-coal foreign objects and reduce the parameter redundancy of conventional convolution.

[0075] In at least one embodiment of the present application, a foreign body recognition model is constructed based on historical images, a parameter-free attention mechanism SimAM module, and a convolutional neural network model, including: preprocessing the acquired historical images; data annotation and data classification of the preprocessed historical images, forming a training set, a verification set, and a test set for the historical images after data classification; obtaining a foreign body recognition model based on the training set and the verification set; and evaluating the performance of the foreign body recognition model based on the test set. Among them, the training set is mainly used for the early training of the network model to obtain the optimal weight, the verification set is mainly used to verify the effect of the trained network model and play the role of fine-tuning the model, and the test set is mainly a measure of the final verification of the recognition effect of the network model (such as recognition accuracy). If the requirements are met, the foreign body recognition model is determined, otherwise the foreign body recognition model is obtained by reusing the training set and the verification set after adjustment.

[0076] In this solution, by preprocessing the historical images, the training data of the training set when building the recognition model can be guaranteed to be relatively accurate, and the performance evaluation of the foreign body recognition model based on the test set can ensure a certain recognition accuracy. Data annotation is mainly to provide a correct result for the input of the network model, and then the network model "masters" some rules of its recognition object through learning, so that it can provide reasonable and accurate output when facing new image data.

[0077] In at least one embodiment of the present application, a foreign body recognition model is obtained based on a training set and a validation set, including: based on the training set, importing the pre-trained model weights into the convolutional neural network model through transfer learning; based on the validation set, fine-tuning the convolutional neural network model to form a foreign body recognition model. The model weight is the weight coefficient in the convolution operation, that is, the parameter quantity.

[0078] like Figure 3 As shown, it is a weight allocation principle diagram based on the SimAM attention mechanism provided by the embodiment of the present application. The image data to be operated is input at the input end, and then the energy function is parsed to solve the energy function value of the weight near each pixel point in the image data, and then the corresponding 3-D weight value is parsed. Then the sigmoid function is used to limit some abnormal excessive values ​​of the energy function, and the sigmoid function will not affect the relative importance of each neuron. Finally, a feature map composed of important neurons is output.

[0079] In at least one embodiment of the present application, the current image is preprocessed, including: image enhancement, image denoising, image size change and image geometric transformation of the current image; historical images are classified according to the ratio of 7:2:1 and randomly divided to form a training set, a verification set and a test set respectively.

[0080] It can be understood that image preprocessing includes but is not limited to the above methods. By randomly dividing the training set, the validation set and the test set in the above proportions, a foreign body recognition model can be better generated and verified.

[0081] In at least one embodiment of the present application, based on a foreign object recognition model, detecting whether there is a foreign object in the current image includes: extracting features and reconstructing features of the current image to obtain a feature map; fusing the feature map to obtain a fused feature map; based on the model weights, detecting the fused feature map to output the recognized result, wherein the model weights include 3-D attention weights obtained based on the parameter-free attention mechanism SimAM module.

[0082] In this scheme, the feature map is obtained by performing convolution and maximum pooling operations on the image, and then fused. Based on the model weight, the fused feature map is detected to output the recognition result. The recognition result includes annotated images containing foreign objects, video streams, or other methods that can prompt the recognition of foreign objects. Feature extraction, through analysis and transformation to extract the required features, feature reconstruction can pay more attention to important neurons and allocate less attention to unimportant neurons, thereby reconstructing a new feature map composed of important neurons.

[0083] In at least one embodiment of the present application, feature extraction and feature reconstruction are performed on the current image, including: based on a downsampling unit and a basic unit, extracting foreign body features with attention information in the current image.

[0084] The foreign body recognition model based on the above recognition method achieved 95.6% mAP in the validation set, and the network parameter amount and model calculation amount were reduced by 56.9% and 42.0% respectively, realizing the rapid and accurate recognition of non-coal foreign bodies with high real-time performance and accuracy.

[0085] Based on the above recognition method, the YOLOv8 target detection network model is used as the basic network model. The schematic diagram of the non-coal foreign body recognition example network model is as follows: Figure 4 As shown. The network model mainly consists of three parts: backbone network, feature fusion and prediction layer. The backbone network contains two sub-units: stacked downsampling units and basic units. This module mainly extracts and reconstructs features of the input image data. Feature fusion mainly fuses the features extracted by the backbone network. The prediction layer is the output part of the network model, which is responsible for generating the results of non-coal foreign body detection. In the entire network model, the backbone network plays a basic and important role, so it is called the backbone network.

[0086] In the specific foreign body identification, the following steps are included:

[0087] (1) Perform data enhancement and other preprocessing operations on the belt conveyor non-coal foreign body image at the data input end, convert the image into 224×224×3 (width×height×number of feature map channels, the same below) and input it into the foreign body detection and recognition network;

[0088] (2) The image is subjected to convolution and maximum pooling operations to obtain a feature map. Three network modules containing downsampling units and basic units are used to further extract non-coal foreign body features with attention information, and a 7×7×192 feature map is obtained; among them, the number of stacking times of downsampling units and basic units in stages 2 and 4 is 1 and 3, and the number of stacking times of downsampling units and basic units in stage 3 is 1 and 9;

[0089] (3) After convolution operation, global pooling layer and fully connected layer, the feature extraction results are passed to the feature fusion layer for feature fusion. Finally, the non-coal foreign body image recognition results of the belt conveyor are obtained through three detection heads of different sizes in the prediction layer.

[0090] Max pooling layer: It is usually used in image processing tasks. It realizes feature extraction and dimensionality reduction by selecting the maximum value in each area. This method can retain the most significant features of the input data while reducing the amount of calculation. Global pooling layer: It compresses the entire image into a vector, and then inputs the vector into the fully connected layer for classification. This method can reduce the number of parameters and improve the generalization performance of the model. Fully connected layer: All feature matrices of the pooling layer are converted into a one-dimensional feature vector. The fully connected layer is generally placed at the end of the convolutional neural network structure to classify images. The three detection heads are connected in parallel, and the sizes of objects detected by detection heads of different sizes are different. According to the sizes of the three detection heads in the figure, they correspond to detecting large objects, medium objects, and small objects.

[0091] like Figure 5 As shown, the embodiment of the present application provides a schematic diagram of the analysis of experimental results based on the above example model. The figure shows the activation heat map of the foreign body recognition network model detection. The activation heat map (Grad-CAM, Gradient-weighted Class Activation Mapping) can provide a clear visualization basis for the classification results of the model. The darker the color of the heat map, the more attention the model has paid to the relevant area, and the more accurate the detection result of the foreign body target. Therefore, the activation heat maps before, during and after optimization were extracted respectively. The first row in the figure is the image of coal gangue to be detected, and the second row is the image of screws to be detected. It can be intuitively seen from the heat map that the recognition effect after optimization (i.e., the fourth column) is more significant. The network model only pays attention to the area where foreign bodies appear, and almost no attention is paid to the non-to-be-detected area. Thereby further verifying the significance of the non-coal foreign body recognition network model designed by the present invention.

[0092] In order to further test the adaptability of the model, Figure 6 As shown, an embodiment of the present application provides a visualization diagram of the generalization experimental results based on the above example model. The figure shows the generalization performance of the non-coal foreign matter recognition network model designed by the present invention. The first three rows and the last three rows in the figure are two groups of recognition results, which we can call the recognition results of data I and data II. In data I, the first row is the recognition result in the case of partial stains, the second row is the recognition result under low light conditions, and the third row is the recognition result under motion blur conditions (the first column in the figure represents a mild situation, the second column represents a moderate situation, and the last column represents a severe situation). The same is true for data II. By analyzing the visual detection effect, the anti-interference ability and generalization performance of the network model are verified. As can be seen from the figure, when faced with fresh sample data, the network model still has a high recognition accuracy.

[0093] Finally, in order to verify the technical effect of the model (such as the amount of calculation, processing speed, etc.), the embodiment of the present application provides a comparison chart of the results of the example network model of the present invention and other classic network models. Figure 7(a)-7(b) As shown, in order to further verify the effectiveness of the non-coal foreign body recognition network model designed by the present invention, a horizontal comparison is made with mainstream target detection algorithms such as YOLOv3, YOLOv5, and YOLOv7. In addition, several representative lightweight convolutional network models are compared, including Mobilenetv3, EfficientNet, MobileNext, PP-LCNet, GhostNetV2, and FasterNet. Among them, Figure 7 (a) is a comparison chart of the parameter quantity with other foreign body recognition network models, and Figure 7 (b) is a comparison chart of the detection accuracy (curve in the figure) and model calculation amount (bar chart in the figure) with other foreign body recognition network models. By comparison, it can be obtained that the non-coal foreign body recognition network model designed by the present invention further compresses the parameter quantity and calculation amount to 1.6M and 4.7G while ensuring the detection accuracy (mAP50: 95.6%). Computational amount FLOPs: refers to the number of floating-point operations per second. Theoretical floating-point calculation speed = main frequency * number of floating-point calculations per clock cycle * number of CPU cores per CPU * number of CPUs per node. Compared with many lightweight convolutional networks, it can still achieve higher detection accuracy and lower parameter and network calculation amount.

[0094] Compared with the prior art, the beneficial effects of the present invention are:

[0095] First, the method for identifying foreign objects in conveyed materials provided by the present invention designs a lightweight and flexible method for detecting and identifying foreign objects in conveyed materials on a belt conveyor that can be used in edge computing devices. On the basis of existing coal gangue identification, it enriches the six common types of foreign objects, and the detection types are more comprehensive.

[0096] Second, the present invention embeds the parameter-free attention mechanism SimAM module in the deep separable convolution and conventional convolution layers of the feature extraction network, so that the network can be more focused on extracting important feature information of non-coal foreign matter images, which is beneficial to suppress the interference of redundant information on the network, thereby more efficiently completing specific feature extraction and reconstruction, improving network recognition accuracy, and reducing network complexity.

[0097] Third, by designing novel basic units and downsampling units, the present invention makes the network more delicate, better captures the more subtle shape features between different foreign objects, and fully learns more detailed information features of non-coal foreign objects, reducing the parameter redundancy of conventional convolution.

[0098] Fourth, the present invention verifies the adaptability and technical effect of the network model through generalization experiments and comparison with the results of other classic network models.

[0099] like Figure 8 As shown, an embodiment of the present invention provides a device 800 for identifying foreign matter in conveying materials, comprising:

[0100] A first acquisition unit 810 is used to acquire historical images of conveyed materials;

[0101] The first processing unit 820 is used to construct a foreign object recognition model based on the historical image, the parameter-free attention mechanism SimAM module and the convolutional neural network model, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM;

[0102] A second acquisition unit 830 is used to acquire a current image of the conveyed material;

[0103] The second processing unit 840 is used to pre-process the current image;

[0104] The recognition unit 850 is used to detect whether there is a foreign object in the current image based on the foreign object recognition model.

[0105] In the above device, the first acquisition unit acquires the historical image of the conveyed material, and then the first processing unit builds a foreign body recognition model based on the historical image, the parameter-free attention mechanism SimAM module and the convolutional neural network model, and then the second acquisition unit acquires the current image of the conveyed material, and then the second processing unit preprocesses the current image, and finally the recognition unit detects whether there is a foreign body in the current image based on the foreign body recognition model. The image of the conveyed material contains coal blocks and non-coal foreign bodies. Through the above steps, while ensuring a certain recognition accuracy, the problems of large data calculation amount and long processing time in the foreign body recognition of the conveyed material are solved, thereby reducing the possibility of coal mine shutdown caused by conveyor belt damage, while ensuring the life safety of coal miners.

[0106] In one implementation of the present application, the first processing unit 820 includes an embedding module. The embedding module is used to embed the parameter-free attention mechanism SimAM module in the downsampling unit and the basic unit of the convolutional neural network model, wherein the downsampling unit and the basic unit are designed by depthwise separable convolution.

[0107] In another implementation of the present application, the first processing unit 820 further includes a first processing module, a second processing module, a determination module, and an evaluation module. The first processing module is used to preprocess the acquired historical images; the second processing module is used to perform data annotation and data classification on the preprocessed historical images, and the historical images after data classification form a training set, a verification set, and a test set; the determination module is used to obtain a foreign body recognition model based on the training set and the verification set; and the evaluation module is used to perform performance evaluation on the foreign body recognition model based on the test set.

[0108] In another implementation of the present application, the determination module is specifically used to: based on the training set, import the pre-trained model weights into the convolutional neural network model through transfer learning; based on the verification set, fine-tune the convolutional neural network model to form a foreign object recognition model.

[0109] In another implementation of the present application, the second processing unit 840 is specifically used to perform image enhancement, image denoising, image size change and image geometric transformation on the current image, and the second processing module is specifically used to classify the historical images according to the ratio of 7:2:1 and randomly divide them to form a training set, a verification set and a test set respectively.

[0110] In another implementation of the present application, the recognition unit 850 includes a third processing module, a fusion module and a recognition module. The third processing module is used to extract and reconstruct features of the current image to obtain a feature map; the fusion module is used to fuse the feature map to obtain a fused feature map; the recognition module is used to detect the fused feature map based on the model weight to output the recognized result, wherein the model weight includes a 3-D attention weight obtained based on the parameter-free attention mechanism SimAM module.

[0111] In another implementation of the present application, the third processing module is also used to extract foreign body features with attention information in the current image based on the downsampling unit and the basic unit.

[0112] The above-mentioned foreign body identification device for conveying materials includes a processor and a memory. The above-mentioned first acquisition unit, first processing unit, second acquisition unit, second processing unit, identification unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0114] Based on the embodiments of the present application, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute each step of the method for identifying foreign objects in conveyed materials of the aforementioned technical solution.

[0115] Based on the embodiments of the present application, the present application further provides a computer-readable storage medium, wherein computer instructions are used to enable a computer to execute each step of the method for identifying foreign matter in conveying materials of the aforementioned technical solution.

[0116] Based on the embodiments of the present application, the present application further provides a processor, which is used to run a program, wherein when the program is run, each step of the method for identifying foreign matter in conveying materials of the aforementioned technical solution is executed.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0119] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0121] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the described embodiment, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] A person skilled in the art may understand that all or part of the steps of implementing the method of the embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0124] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the embodiments are exemplary and cannot be construed as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the embodiments within the scope of the present invention.

[0125] The above is only a specific implementation of this specification, but the protection scope of this specification is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of this specification should be based on the protection scope of the claims.

Claims

1. A method for identifying foreign matter in conveying materials, characterized in that: include: Obtain historical images of conveyed materials; Based on the historical images, the parameter-free attention mechanism SimAM module and the convolutional neural network model, a foreign object recognition model is constructed, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM; Acquire a current image of the conveyed material; Preprocessing the current image; Based on the foreign object recognition model, detect whether there is a foreign object in the current image.

2. The identification method according to claim 1, characterized in that: The parameter-free attention mechanism SimAM module is embedded in the downsampling unit and the basic unit of the convolutional neural network model, wherein the downsampling unit and the basic unit are designed by depthwise separable convolution.

3. The identification method according to claim 1 or 2, characterized in that: The foreign body recognition model is constructed based on the historical image, the parameter-free attention mechanism SimAM module and the convolutional neural network model, including: Preprocessing the acquired historical images; Performing data annotation and data classification on the preprocessed historical images, and forming a training set, a verification set and a test set from the historical images after data classification; Based on the training set and the verification set, obtaining the foreign body recognition model; Based on the test set, the performance of the foreign object recognition model is evaluated.

4. The identification method according to claim 3, characterized in that: The foreign body recognition model is obtained based on the training set and the validation set, including: Based on the training set, the pre-trained model weights are imported into the convolutional neural network model through transfer learning; Based on the verification set, the convolutional neural network model is fine-tuned to form the foreign object recognition model.

5. The identification method according to claim 3, characterized in that: The preprocessing of the current image includes: performing image enhancement, image denoising, image size change and image geometric transformation on the current image; The historical images are classified according to a ratio of 7:2:1 and randomly divided to form the training set, the validation set and the test set, respectively.

6. The identification method according to claim 4, characterized in that: The detecting whether there is a foreign object in the current image based on the foreign object recognition model includes: Performing feature extraction and feature reconstruction on the current image to obtain a feature map; Fusing the feature maps to obtain the fused feature maps; Based on the model weights, the fused feature maps are detected to output the recognized results, wherein the model weights include 3-D attention weights obtained based on the parameter-free attention mechanism SimAM module.

7. The identification method according to claim 6, characterized in that: The feature extraction and feature reconstruction of the current image includes: extracting foreign body features with attention information in the current image based on the downsampling unit and the basic unit.

8. A device for identifying foreign matter in conveying materials, characterized in that: include: A first acquisition unit, used for acquiring historical images of conveyed materials; A first processing unit is used to construct a foreign object recognition model based on the historical image, the parameter-free attention mechanism SimAM module and the convolutional neural network model, wherein the parameter-free attention mechanism SimAM module is a module designed based on the parameter-free attention mechanism SimAM; A second acquisition unit, used to acquire a current image of the conveyed material; A second processing unit, configured to pre-process the current image; The recognition unit is used to detect whether there is a foreign object in the current image based on the foreign object recognition model.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform each step in the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes each step of the method according to any one of claims 1 to 7.