Conveyor real-time empty detection method and system based on improved ResNet network

Abstract

The application discloses a kind of based on improved ResNet network's conveyer belt real-time empty detection method and system, comprising: parallelly placed half circle protractor is placed on the shooting area of conveyer belt, pixel focal length of camera shooting is calculated in proportion;Image on the conveyer belt is divided into data, and the training set of sand and ore image is expanded in data enhancement;Improved ResNet network is constructed, and the network is trained;Test obtains the multi-classification model of network prediction conveyer belt empty;The feature information of the image area photographed on the conveyer belt is extracted by the improved ResNet network in the application, since the improved SE attention mechanism is introduced, the feature map output by residual module can be enhanced locally semantic information, not only greatly reduce the parameter amount and calculation amount of network, improve the detection effect of conveyer belt empty identification, but also effectively improve the production efficiency of conveyer belt and save power consumption, and have broad application prospect in industry.

Description

Technical Field

[0001] This invention relates to the technical field of image processing and image recognition, and in particular to a method and system for real-time empty detection of conveyor belts based on an improved ResNet network. Background Technology

[0002] With the rapid development of the building materials industry, the demand for sand and gravel aggregates is constantly increasing both domestically and internationally. The industry is placing increasing emphasis on the quality of building materials. Different building materials have specific requirements for the particle size of sand and gravel, and the industry has a strict set of quality control standards for the particle size of raw materials. Before mixing sand and gravel of different sizes, the particle size information of the sand and gravel passing on the conveyor belt is first detected by a camera. A good method for detecting and classifying sand and gravel particle size plays a crucial role in the accuracy of subsequent sand and gravel aggregate mixing and has a decisive impact on the quality of the building.

[0003] With the rapid development of deep learning, its application in industry has become a research hotspot. To achieve particle size detection of sand and gravel aggregates in dynamic industrial scenarios, most concrete companies use fixed cameras to monitor conveyor belts in real time. However, due to the harsh production environment on the assembly line, noise interference and vibrations from the equipment on the conveyor belt can cause the camera to lose focus, affecting the quality of the captured images and the accuracy of the algorithm. Furthermore, if the power is not cut off promptly after the sand and gravel aggregates have been conveyed, the conveyor belt will idle. Idle operation not only consumes a lot of electricity but also causes wear and tear on the conveyor belt, thus impacting production on the assembly line. Therefore, finding an efficient and automatic method for measuring distance and detecting empty conveyor belts is urgent and has become a key research focus in this field. Summary of the Invention

[0004] The primary objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a real-time empty detection method for conveyor belts based on an improved ResNet network. This method can effectively improve the production efficiency of conveyor belts and save power consumption. Before transporting sand and gravel aggregates on the conveyor belt, the method performs automatic focusing and distance measurement. It uses image morphology processing to convert the actual size of the reference object into the calibrated size of the image pixels to verify whether the focal length between the camera and the conveyor belt has changed. After verifying the focal length, the method identifies whether sand and gravel aggregates are entering the conveyor belt. The improved ResNet network is used to identify the images captured by the camera, and an improved SE attention mechanism is introduced to judge the feature information of the image area captured on the conveyor belt, thereby improving the accuracy of conveyor belt recognition.

[0005] The second objective of this invention is to provide a real-time empty detection system for conveyor belts based on an improved ResNet network.

[0006] The first objective of this invention is achieved through the following technical solution: a real-time empty detection method for conveyor belts based on an improved ResNet network. This improved ResNet network is an improvement on the original ResNet network in two parts. The first part is an improvement to the residual module of the original ResNet network: the original 3×3 convolutional layer is replaced with 3×3, 5×5, and 7×7 convolutional layers to construct parallel feature extraction. Different features are extracted through parallel convolution operations, and difference operations are performed to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the locally optimal feature information. The second part is the addition of an improved SE attention mechanism after the residual module. The improvement of the SE attention mechanism is to change the channel coefficient adjustment in the excitation operation to channel normalization activation, thereby avoiding parameter loss during channel adjustment. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, enabling the network to optimize the weights of the feature maps extracted at each stage from the perspective of global information.

[0007] The specific implementation of this real-time empty belt detection method includes the following steps:

[0008] 1) Before the conveyor belt operation, place the semicircular protractor parallel to the shooting area of ​​the conveyor belt. The camera takes a picture of the calibrated semicircular protractor through the monitoring screen. Calculate the number of unit pixels occupied by the calibrated object in the horizontal direction based on the picture. Calculate the distance between the camera and the conveyor belt according to the ratio, which is the pixel focal length of the camera.

[0009] 2) After determining the pixel focal length of the camera, images of the conveyor belt during startup and operation are captured by the camera. A custom conveyor belt dataset is constructed, which includes images of an empty conveyor belt, sand, and ore. Data augmentation is performed on the sand and ore images to expand the dataset. The augmented sand, ore, and empty conveyor belt images are divided into training, validation, and test sets according to a set ratio.

[0010] 3) The training set is fed into the improved ResNet network in batches for training. After the network extracts the features of the images in the training set, the probability value of the image classification is obtained by the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real value collected and labeled by humans. Finally, the weights are updated based on the loss value calculated by the network. The network parameters are adjusted by the optimizer according to the number of iterations during the training process, and the learning rate is updated. After each specific number of network trainings, the training effect of the network is verified by the validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained.

[0011] 4) Feed the test set into the optimal network obtained in step 3) to obtain the multi-classification model for predicting whether the conveyor belt is empty. Set the corresponding classification values: set the classification value of an empty conveyor belt to 0, the classification value of sand on the conveyor belt to 1, and the classification value of ore to 2. Record the top 1 test value during the test process. The evaluation index of the optimal accuracy value is obtained through the highest test value.

[0012] Further, in step 1), a CCD industrial camera is installed above the conveyor belt. The camera power supply is connected to the relay, and the relay and the central control system communicate via Socket through the same IP network segment. Before the conveyor belt starts operating, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is captured. Based on the captured image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. While enhancing the foreground of the calibration object, the low-frequency components of the background of the conveyor belt are weakened. This allows for the statistical analysis of the pixels in the connected area of ​​the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as:

[0013]

[0014] In the formula, S px S represents the pixel size of the calibrated image. l This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt.

[0015] Further, in step 2), after determining the pixel focal length of the camera, a custom conveyor belt dataset is constructed by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the original image using bilinear interpolation to a uniform size and resolution of 256×256; then, performing brightness and grayscale transformations on the image; next, performing median filtering on the transformed image to reduce noise and blur; and finally, performing morphological processing on the image, including closing... The operation reduces noise interference in the image during the conveyor belt operation, and finally normalizes the pixel values ​​of the image and scales them to the [0,1] range; among them, rotation, flipping, and cropping operations are performed on the sand and ore images in the training set to augment the data. The rotation values ​​are -30°, -15°, 15° and 30°, the flipping includes horizontal flipping, vertical flipping and horizontal and vertical flipping, and the cropping operation uses random aspect ratios of 3 / 4 or 4 / 3. The preprocessed images are divided into training set, validation set and test set according to the set ratio.

[0016] Furthermore, in step 3), the improved ResNet network consists of a six-layer architecture, as detailed below:

[0017] The first layer architecture consists of 7×7 convolutional operations and 3×3 max pooling operations, used to adjust the size of the input feature map and downsample the number of channels;

[0018] The architecture from the second to the fifth layer consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice. The improved residual module extracts features in five steps, as detailed below:

[0019] The first step is to reduce the channel dimension of the feature map by performing a 3×3 convolution operation, thereby reducing the number of parameters required for network training.

[0020] The second step involves forming three new feature maps in parallel using 3×3, 5×5, and 7×7 convolution operations, respectively. The process is shown in formulas (2), (3), and (4):

[0021] X0=H(x)*K3 (2)

[0022] X1=H(x)*K5 (3)

[0023] X2=H(x)*K7 (4)

[0024] In the formula, H(x) represents the feature map output after dimensionality reduction, X0 represents the feature map output after 3×3 convolution operation, K3 represents 3×3 convolution operation, X1 represents the feature map output after 5×5 convolution operation, K5 represents 5×5 convolution operation, X2 represents the feature map output after 7×7 convolution operation, and K7 represents 7×7 convolution operation.

[0025] The third step involves subtracting feature map X0 from feature map X1 obtained through parallel convolution to form a new feature map X3, and subtracting feature map X1 from feature map X2 to form a new feature map X4. This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: formulas (5) and (6):

[0026] X3 = X1 - X0 (5)

[0027] X4 = X2 - X1 (6)

[0028] In the formula, X3 represents the feature map obtained by subtracting feature map X0 from feature map X1, and X4 represents the feature map obtained by subtracting feature map X1 from feature map X2.

[0029] The fourth step involves fusing feature map X3 and feature map X4, and then using skip connections to concatenate the original feature maps of the improved residual module along channels to form a new feature map X5. This process fuses the feature maps obtained from the difference operation, enhancing the generalization ability between feature spaces and increasing the amount of information describing image features. Skip connections fuse the feature maps obtained from different convolution operations with the original feature maps, giving the network richer semantic information. The process is shown in formula (7):

[0030] X5=x+(X3+X4) (7)

[0031] In the formula, X5 represents the feature map obtained by skip connection fusion, and x represents the original feature map of the input residual module;

[0032] The fifth step is to increase the channel dimension of the feature map through a 1×1 convolution operation;

[0033] After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then augmented with a local information enhancement mechanism using an improved SE attention mechanism. This not only significantly reduces the number of network parameters and computational cost, but also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task, thus enabling better detection and real-time emptying of the conveyor belt. The improved SE attention mechanism involves feature extraction in three steps, as detailed below:

[0034] The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (8)

[0035]

[0036] In the formula, x c H represents the feature map obtained from the improved residual module. c The height of the feature map, W represents x c The width of the feature map, where i represents x c The pixel index position of the feature map height, where j represents x. c The pixel index position of the feature map width, x c (i,j) represents the coordinate position of a pixel in the feature map, F GAP This indicates that a global average pooling operation is performed on the feature map, Z. c Indicates the feature map x c The feature map after performing H×W global average pooling in the spatial dimension;

[0037] The second step is to adjust Z. c BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (9):

[0038] S c =σ(W2δ(W1Z) c (9)

[0039] In the formula, σ represents the sigmoid activation function, δ represents the ReLU activation function, and S c To obtain normalized weights for the activation function;

[0040] The third step is to transfer Z c With the obtained normalized weight S c Multiplication is performed, and the coefficients of the spatial local feature factors obtained by multiplication are weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (10) and (11):

[0041] F c =Z c *S c (10)

[0042]

[0043] In the formula, F c These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism;

[0044] The sixth layer architecture is a downsampling operation implemented through average pooling; the average value of all pixels in the feature map of each output channel is calculated to obtain the feature information corresponding to the category;

[0045] The feature maps output by the improved ResNet network are compressed into the range [0,1] using the Softmax function. The output probability values ​​are then normalized to predict the probability of the highest-scoring category. The loss is calculated using binary cross-entropy, and the network parameters are optimized through backpropagation using the Adam optimizer and loss calculation. During network training, a dynamic learning rate is used. After each specific number of training iterations, a validation set is used to verify the training effect of the network. The training terminates when the loss on the validation set reaches its minimum value, i.e., the total loss on the validation set no longer decreases after 10 consecutive iterations.

[0046] Further, in step 4), the test set is sent into the optimal network to obtain the classification value and probability predicted by the network. When the classification value is 0, the result is an empty conveyor belt, which is determined to be idle. When the classification value is 1, the result is sand, which is determined to be non-idle. When the classification value is 2, the result is ore, which is determined to be non-idle.

[0047] The second objective of this invention is achieved through the following technical solution: a real-time empty detection system for conveyor belts based on an improved ResNet network, comprising:

[0048] The data acquisition module captures images of the conveyor belt during startup and operation via a camera, and custom-builds a conveyor belt dataset, which includes images of an empty conveyor belt, sand, and ore. Data augmentation is performed on the sand and ore images to expand the dataset, and the augmented sand, ore, and empty conveyor belt images are divided into training, validation, and test sets according to a set ratio.

[0049] An improved ResNet network module is used to recognize photos taken by a camera, and an improved SE attention mechanism is introduced to judge the feature information of the image area captured on the conveyor belt, thereby improving the accuracy of conveyor belt recognition.

[0050] The training module is used to train and validate the improved ResNet network module to obtain the optimal network.

[0051] The empty conveyor detection module is used to feed the test set into the optimal network to obtain a multi-classification model for predicting whether the conveyor belt is empty. It sets corresponding classification values: 0 for an empty conveyor belt, 1 for the presence of sand, and 2 for the presence of ore. The module records the top-ranked test value during the testing process, using the highest test value as the evaluation metric for optimal accuracy. Specifically, when the identified classification value is 0, the result is an empty conveyor belt, indicating it is running idle; when the identified classification value is 1, the result is sand, indicating it is not running idle; and when the identified classification value is 2, the result is ore, indicating it is not running idle.

[0052] Furthermore, before the data acquisition module collects data, a semicircular protractor needs to be placed parallel to the imaging area of ​​the conveyor belt. The CCD industrial camera takes a picture of the calibrated semicircular protractor through the monitoring screen. The number of unit pixels occupied by the calibrated object in the horizontal direction is calculated from the picture. The distance between the camera and the conveyor belt is calculated proportionally, which is the pixel focal length of the camera. The specific details are as follows:

[0053] A CCD industrial camera is installed above the conveyor belt. The camera power is connected to the relay, and the relay communicates with the central control system via a socket exchange on the same IP network segment. Before the conveyor belt starts operating, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is captured. Based on the captured image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. This enhances the foreground of the calibration object while weakening the low-frequency components of the conveyor belt background. This allows for the counting of pixels in the connected areas of the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as:

[0054]

[0055] In the formula, S px S represents the pixel size of the calibrated image. l This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt.

[0056] After determining the pixel focal length of the camera, the data acquisition module constructs a custom conveyor belt dataset by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the original image using bilinear interpolation to uniformly size it to a resolution of 256×256; then, transforming the image's brightness and grayscale; next, performing median filtering on the transformed image to reduce noise and blur; then, performing morphological processing on the image to reduce noise interference during conveyor belt operation through closing operations; and finally, normalizing the pixel values ​​of the image and scaling it to the [0,1] range. The sand and ore images in the training set are augmented with rotation, flipping, and cropping operations. Rotation uses values ​​of -30°, -15°, 15°, and 30°; flipping includes horizontal flipping, vertical flipping, and horizontal-vertical flipping; and cropping uses random aspect ratios of 3 / 4 or 4 / 3. The preprocessed images are then divided into training, validation, and test sets according to a set ratio.

[0057] Furthermore, the improved ResNet network module is an improvement on the original ResNet network in two parts. The first part is an improvement to the residual module of the original ResNet network: the original 3×3 convolutional layer is replaced with 3×3, 5×5, and 7×7 convolutional layers to construct parallel feature extraction. Different features are extracted through parallel convolution operations and then differenced to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the local optimal feature information. The second part is the addition of an improved SE attention mechanism after the residual module. The improvement of the SE attention mechanism is to replace the channel coefficient adjustment in the excitation operation with channel normalization activation, thereby avoiding parameter loss during channel adjustment. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, enabling the network to optimize the weights of the feature maps extracted at each stage from the perspective of global information.

[0058] Furthermore, the improved ResNet network module consists of a six-layer architecture, as detailed below:

[0059] The first layer architecture consists of 7×7 convolutional operations and 3×3 max pooling operations, used to adjust the size of the input feature map and downsample the number of channels;

[0060] The architecture from the second to the fifth layer consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice. The improved residual module extracts features in five steps, as detailed below:

[0061] The first step is to reduce the channel dimension of the feature map by performing a 3×3 convolution operation, thereby reducing the number of parameters required for network training.

[0062] The second step involves forming three new feature maps in parallel using 3×3, 5×5, and 7×7 convolution operations, respectively. The process is illustrated in formulas (13), (14), and (15):

[0063] X0=H(x)*K3 (13)

[0064] X1=H(x)*K5 (14)

[0065] X2=H(x)*K7 (15)

[0066] In the formula, H(x) represents the feature map output after dimensionality reduction, X0 represents the feature map output after 3×3 convolution operation, K3 represents 3×3 convolution operation, X1 represents the feature map output after 5×5 convolution operation, K5 represents 5×5 convolution operation, X2 represents the feature map output after 7×7 convolution operation, and K7 represents 7×7 convolution operation.

[0067] The third step involves subtracting feature map X0 from feature map X1 obtained through parallel convolution to form a new feature map X3, and subtracting feature map X1 from feature map X2 to form a new feature map X4. This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: formulas (16) and (17):

[0068] X3 = X1 - X0 (16)

[0069] X4 = X2 - X1 (17)

[0070] In the formula, X3 represents the feature map obtained by subtracting feature map X0 from feature map X1, and X4 represents the feature map obtained by subtracting feature map X1 from feature map X2.

[0071] The fourth step involves fusing feature map X3 and feature map X4, and then using skip connections to concatenate the original feature maps of the improved residual module along channels to form a new feature map X5. This process fuses the feature maps obtained from the difference operation, enhancing the generalization ability between feature spaces and increasing the amount of information describing image features. Skip connections fuse the feature maps obtained from different convolution operations with the original feature maps, giving the network richer semantic information. The process is illustrated in formula (18).

[0072] X5=x+(X3+X4) (18)

[0073] In the formula, X5 represents the feature map obtained by skip connection fusion, and x represents the original feature map of the input residual module;

[0074] The fifth step is to increase the channel dimension of the feature map through a 1×1 convolution operation;

[0075] After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then augmented with a local information enhancement mechanism using an improved SE attention mechanism. This not only significantly reduces the number of network parameters and computational cost, but also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task, thus enabling better detection and real-time emptying of the conveyor belt. The improved SE attention mechanism involves feature extraction in three steps, as detailed below:

[0076] The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (19)

[0077]

[0078] In the formula, x c H represents the feature map obtained from the improved residual module. c The height of the feature map, W represents x c The width of the feature map, where i represents x c The pixel index position of the feature map height, where j represents x. c The pixel index position of the feature map width, x c (i,j) represents the coordinate position of a pixel in the feature map, F GAP This indicates that a global average pooling operation is performed on the feature map, Z. c Indicates the feature map x c The feature map after performing H×W global average pooling in the spatial dimension;

[0079] The second step is to adjust Z. c BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (20)

[0080] S c =σ(W2δ(W1Z) c (20)

[0081] In the formula, σ represents the sigmoid activation function, δ represents the ReLU activation function, and S c To obtain normalized weights for the activation function;

[0082] The third step is to transfer Z c With the obtained normalized weight S c Multiplication is performed, and the coefficients of the spatial local feature factors are obtained by multiplication and weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (21) and (22):

[0083] F c =Z c *S c (twenty one)

[0084]

[0085] In the formula, F c These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism;

[0086] The sixth layer architecture is a downsampling operation implemented through average pooling; the average value of all pixels in the feature map of each output channel is calculated to obtain the feature information corresponding to the category.

[0087] Furthermore, the training module specifically performs the following operations:

[0088] The training set is fed into the improved ResNet network module in batches for training. After the network extracts the features from the images in the training set, the probability value of image classification is obtained through the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real values ​​collected and labeled by humans. Finally, the weights are updated based on the loss value calculated by the network. The network parameters are adjusted by the optimizer according to the number of iterations during training, and the learning rate is updated. After each specific number of network training iterations, the training effect of the network is verified using a validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained.

[0089] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0090] 1. This invention can effectively reduce the impact of noise and vibration on the conveyor belt on the focal length. When the camera goes out of focus, the clarity of the acquired image decreases, and the detected focal length is less than the preset threshold, an alarm can be issued to adjust the focal length. Compared with other infrared and ultrasonic ranging methods on production lines, this solution does not require external equipment support and effectively avoids the impact of light and noise on the detection accuracy of the equipment, thereby ensuring the accuracy of sand and gravel aggregate detection results and the reliability of conveyor belt emptying detection, improving production efficiency, and reducing a large amount of manpower and material resources consumption.

[0091] 2. This invention is based on an improved ResNet network, using multiple convolutional layers of different sizes in parallel to extract different features, thereby allowing the network to converge to different values. An improved SE attention mechanism is introduced to enhance the local semantic information of the feature map output by the residual module, which not only significantly reduces the number of network parameters and computational cost, improving the detection effect of conveyor belt emptying identification, but also effectively improves the production efficiency of the conveyor belt and saves electricity consumption. The method of using difference fusion features not only allows weight parameters and edge features to be shared, but also enriches the feature information in shortcut branches, improving the accuracy of regional feature recognition on edge structures.

[0092] 3. Compared with other methods for detecting empty conveyor belts, this invention has a moderate number of model layers, a small number of parameters and computational load, fast recognition speed, high accuracy, and strong robustness. It is of great significance for subsequent sand and gravel aggregate detection. At the same time, it can reduce conveyor belt wear and power waste. Therefore, it has a wide range of application potential and broad application prospects in other practical application scenarios. Attached Figure Description

[0093] Figure 1 This is a hardware device architecture diagram of the present invention.

[0094] Figure 2 This is a schematic diagram of the network structure of the present invention; in the figure, Image represents an image, multiple_unit_blocks represents an improved residual module, and Seplus_blocks represents an improved SE attention mechanism.

[0095] Figure 3 This is a schematic diagram of the improved residual module structure of the present invention.

[0096] Figure 4 This is a schematic diagram of the improved SE attention mechanism structure of the present invention.

[0097] Figure 5 This is an architecture diagram of the system of the present invention. Detailed Implementation

[0098] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0099] Example 1

[0100] This embodiment is implemented using the PyTorch deep learning framework. The computer configuration includes an Intel Core i7-10750H processor, 16GB of RAM, an NVIDIA GeForce GTX1650 Ti graphics card, and a Windows operating system. This embodiment discloses a real-time empty belt detection method based on an improved ResNet network, the specific implementation of which includes the following steps:

[0101] 1) Install a CCD industrial camera above the conveyor belt. Connect the camera power supply to the relay. The relay and the central control system communicate via a socket exchange through the same IP network segment. See [link to documentation]. Figure 1 As shown, before the conveyor belt operation, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is taken. Based on the acquired image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. This enhances the foreground of the calibration object while weakening the low-frequency components of the background of the conveyor belt. This allows for the statistical analysis of the pixels in the connected regions of the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as:

[0102]

[0103] In the formula, S px S represents the pixel size of the calibrated image. l This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt.

[0104] 2) After determining the camera's pixel focal length, a custom conveyor belt dataset is constructed by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the original images using bilinear interpolation to a uniform resolution of 256×256; then, performing brightness transformation to adjust contrast and saturation; followed by grayscale transformation to adjust the R, G, and B pixel components, making the edges and contours of the sand and ore clearer; and finally, constructing a structuring element with a (3,3) convolution kernel to process the image. The image undergoes filtering, morphological processing, and a closing operation to reduce noise interference that may occur during the conveyor belt operation. Finally, the pixel values ​​are normalized and scaled to the [0,1] range. The training set's sand and ore images are augmented with rotation, flipping, and cropping operations. Rotation uses values ​​of -30°, -15°, 15°, and 30°, flipping includes horizontal flipping, vertical flipping, and horizontal-vertical flipping, and cropping uses random aspect ratios (3 / 4 or 4 / 3). The preprocessed images are then divided into training, validation, and test sets according to a set ratio.

[0105] 3) The training set is fed into the improved ResNet network in batches for training. After the network extracts the features of the images in the training set, the probability value of the image classification is obtained by the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real value collected and labeled by humans. Finally, the weights are updated based on the loss value calculated by the network. The network parameters are adjusted by the optimizer according to the number of iterations during the training process, and the learning rate is updated. After each specific number of network training iterations, the training effect of the network is verified by the validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained.

[0106] like Figure 2As shown, the improved ResNet network is a parallel residual structure, which improves the original ResNet network in two parts. The first part improves the residual module of the original ResNet network: the original 3×3 convolutional layer is replaced with 3×3, 5×5, and 7×7 convolutional layers to construct parallel feature extraction. Different features are extracted through parallel convolution operations and then differenced to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the local optimal feature information. The second part adds an improved SE attention mechanism after the residual module. The improvement of the SE attention mechanism is to replace the channel coefficient adjustment in the excitation operation with channel normalization activation, thereby avoiding parameter loss during channel adjustment. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, so that the network optimizes the weights of the feature maps extracted at each stage from the perspective of global information.

[0107] The improved ResNet network consists of a six-layer architecture, as detailed below:

[0108] The first layer architecture consists of 7×7 convolutional operations and 3×3 max pooling operations, used to adjust the size of the input feature map and downsample the number of channels;

[0109] The architecture from the second to the fifth layers consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice consecutively. The improved residual module extracts features in five steps, as shown in the structure below. Figure 3 As shown, the specific situation is as follows:

[0110] The first step is to reduce the channel dimension of the feature map by performing a 3×3 convolution operation, thereby reducing the number of parameters required for network training.

[0111] The second step involves forming three new feature maps in parallel using 3×3, 5×5, and 7×7 convolution operations, respectively. The process is shown in formulas (2), (3), and (4):

[0112] X0=H(x)*K3 (2)

[0113] X1=H(x)*K5 (3)

[0114] X2=H(x)*K7 (4)

[0115] In the formula, H(x) represents the feature map output after dimensionality reduction, X0 represents the feature map output after 3×3 convolution operation, K3 represents 3×3 convolution operation, X1 represents the feature map output after 5×5 convolution operation, K5 represents 5×5 convolution operation, X2 represents the feature map output after 7×7 convolution operation, and K7 represents 7×7 convolution operation.

[0116] The third step involves subtracting feature map X0 from feature map X1 obtained through parallel convolution to form a new feature map X3, and subtracting feature map X1 from feature map X2 to form a new feature map X4. This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: formulas (5) and (6):

[0117] X3 = X1 - X0 (5)

[0118] X4 = X2 - X1 (6)

[0119] In the formula, X3 represents the feature map obtained by subtracting feature map X0 from feature map X1, and X4 represents the feature map obtained by subtracting feature map X1 from feature map X2.

[0120] The fourth step involves fusing feature map X3 and feature map X4, and then using skip connections to concatenate the original feature maps from the input residual module along channels to form a new feature map X5. This process fuses the feature maps obtained from the difference operation, enhancing the generalization ability between feature spaces and increasing the amount of information describing image features. Skip connections fuse the feature maps obtained from different convolution operations with the original feature maps, giving the network richer semantic information. The process is shown in formula (7):

[0121] X5=x+(X3+X4) (7)

[0122] In the formula, X5 represents the feature map obtained by skip connection fusion, and x represents the original feature map of the input residual module;

[0123] The fifth step is to increase the channel dimension of the feature map through a 1×1 convolution operation;

[0124] After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then enhanced with a modified SE attention mechanism for local information enhancement. This not only significantly reduces the number of network parameters and computational cost, but also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task, thus enabling better detection and real-time emptying of the conveyor belt. The improved SE attention mechanism involves three steps for feature extraction, the structure of which is shown below. Figure 4 As shown, the specific situation is as follows:

[0125] The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (8)

[0126]

[0127] In the formula, x c H represents the feature map obtained from the improved residual module. c The height of the feature map, W represents x c The width of the feature map, where i represents x c The pixel index position of the feature map height, where j represents x. c The pixel index position of the feature map width, x c (i,j) represents the coordinate position of a pixel in the feature map, F GAP This indicates that a global average pooling operation is performed on the feature map, Z. c Indicates the feature map x c The feature map after performing H×W global average pooling in the spatial dimension;

[0128] The second step is to adjust Z. c BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (9):

[0129] S c =σ(W2δ(W1Z) c (9)

[0130] In the formula, σ represents the sigmoid activation function, δ represents the ReLU activation function, and S c To obtain normalized weights for the activation function;

[0131] The third step is to transfer Z c With the obtained normalized weight S c Multiplication is performed, and the coefficients of the spatial local feature factors obtained by multiplication are weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (10) and (11):

[0132] F c =Z c *S c (10)

[0133]

[0134] In the formula, F c These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism;

[0135] The sixth layer architecture is a downsampling operation implemented through average pooling; the average value of all pixels in the feature map of each output channel is calculated to obtain the feature information corresponding to the category;

[0136] The feature maps output by the improved ResNet network are compressed into the range [0,1] using the Softmax function. The output probability values ​​are then normalized to predict the probability of the highest-scoring category. The loss is calculated using binary cross-entropy, and the network parameters are optimized through backpropagation using the Adam optimizer and loss calculation. During network training, a dynamic learning rate is used. After each specific number of training iterations, a validation set is used to verify the training effect of the network. The training terminates when the loss on the validation set reaches its minimum value, i.e., the total loss on the validation set no longer decreases after 10 consecutive iterations.

[0137] 4) After the network training is completed, the test set is used as input and fed into the optimal network to obtain the multi-class model (classification value and probability) for predicting whether the conveyor belt is empty. The corresponding classification values ​​are set as follows: the classification value of an empty conveyor belt is set to 0, the classification value of sand on the conveyor belt is set to 1, and the classification value of ore on the conveyor belt is set to 2. The top 1 test value during the testing process is recorded, and the evaluation index of the optimal accuracy value is obtained through the highest test value. Specifically, when the identified classification value is 0, the result is that the conveyor belt is empty, and it is judged as running idle. When the identified classification value is 1, the result is that the result is that the conveyor belt is sand, and it is judged as not running idle. When the identified classification value is 2, the result is that the result is that the result is ore, and it is judged as not running idle.

[0138] Example 2

[0139] This embodiment discloses a real-time empty detection system for conveyor belts based on an improved ResNet network. See [link to relevant documentation]. Figure 5 As shown, the system includes the following functional modules:

[0140] The data acquisition module captures images of the conveyor belt during startup and operation via a camera, and custom-builds a conveyor belt dataset, which includes images of an empty conveyor belt, sand, and ore. Data augmentation is performed on the sand and ore images to expand the dataset, and the augmented sand, ore, and empty conveyor belt images are divided into training, validation, and test sets according to a set ratio.

[0141] An improved ResNet network module is used to recognize photos taken by a camera, and an improved SE attention mechanism is introduced to judge the feature information of the image area captured on the conveyor belt, thereby improving the accuracy of conveyor belt recognition.

[0142] The training module feeds the training set into the improved ResNet network module in batches for training. After the network extracts the features of the images in the training set, the probability value of image classification is obtained through the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real values ​​collected and labeled by humans. Finally, the weights are updated based on the loss value calculated by the network. The optimizer adjusts the network parameters and updates the learning rate according to the number of iterations during the training process. After each specific number of network training iterations, the training effect of the network is verified using a validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained.

[0143] The empty conveyor detection module is used to feed the test set into the optimal network to obtain a multi-classification model for predicting whether the conveyor belt is empty. It sets corresponding classification values: 0 for an empty conveyor belt, 1 for the presence of sand, and 2 for the presence of ore. The module records the top-ranked test value during the testing process, using the highest test value as the evaluation metric for optimal accuracy. Specifically, when the identified classification value is 0, the result is an empty conveyor belt, indicating it is running idle; when the identified classification value is 1, the result is sand, indicating it is not running idle; and when the identified classification value is 2, the result is ore, indicating it is not running idle.

[0144] Furthermore, before the data acquisition module collects data, a semicircular protractor needs to be placed parallel to the imaging area of ​​the conveyor belt. The CCD industrial camera takes a picture of the calibrated semicircular protractor through the monitoring screen. The number of unit pixels occupied by the calibrated object in the horizontal direction is calculated from the picture. The distance between the camera and the conveyor belt is calculated proportionally, which is the pixel focal length of the camera. The specific details are as follows:

[0145] A CCD industrial camera is installed above the conveyor belt. The camera's power supply is connected to a relay, and the relay communicates with the central control system via a socket exchange through the same IP network segment. Figure 1As shown, before the conveyor belt operation, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is taken. Based on the acquired image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. This enhances the foreground of the calibration object while weakening the low-frequency components of the background of the conveyor belt. This allows for the statistical analysis of the pixels in the connected regions of the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as:

[0146]

[0147] In the formula, S px S represents the pixel size of the calibrated image. l This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt.

[0148] After determining the pixel focal length of the camera, the data acquisition module constructs a custom conveyor belt dataset by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the original image using bilinear interpolation to unify the size to a resolution of 256×256; then, performing brightness transformation on the image to adjust its contrast and saturation; finally, performing grayscale transformation to adjust the R, G, and B pixel components to make the edges and contours of the sand and ore clearer; and finally, constructing a structuring element with a (3,3) convolution kernel to remove noise from the image. The image is first filtered, then morphologically processed. A closing operation is used to reduce noise interference that may occur during the conveyor belt operation. Finally, the pixel values ​​are normalized and scaled to the [0,1] range. The sand and ore images in the training set are augmented with rotation, flipping, and cropping operations. Rotation uses values ​​of -30°, -15°, 15°, and 30°. Flipping includes horizontal flipping, vertical flipping, and horizontal-vertical flipping. Cropping uses random aspect ratios (3 / 4 or 4 / 3). The preprocessed images are then divided into training, validation, and test sets according to a set ratio.

[0149] Furthermore, the improved ResNet network module is an improvement on the original ResNet network in two parts. The first part is an improvement to the residual module of the original ResNet network: the original 3×3 convolutional layer is replaced with 3×3, 5×5, and 7×7 convolutional layers to construct parallel feature extraction. Different features are extracted through parallel convolution operations and then differenced to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the local optimal feature information. The second part is the addition of an improved SE attention mechanism after the residual module. The improvement of the SE attention mechanism is to replace the channel coefficient adjustment in the excitation operation with channel normalization activation, thereby avoiding parameter loss during channel adjustment. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, enabling the network to optimize the weights of the feature maps extracted at each stage from the perspective of global information.

[0150] See Figure 2 As shown, the improved ResNet network module consists of a six-layer architecture, as detailed below:

[0151] The first layer architecture consists of 7×7 convolutional operations and 3×3 max pooling operations, used to adjust the size of the input feature map and downsample the number of channels;

[0152] The architecture from the second to the fifth layers consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice consecutively. The improved residual module extracts features in five steps, as shown in the structure below. Figure 3 As shown, the specific situation is as follows:

[0153] The first step is to reduce the channel dimension of the feature map by performing a 3×3 convolution operation, thereby reducing the number of parameters required for network training.

[0154] The second step involves forming three new feature maps in parallel using 3×3, 5×5, and 7×7 convolution operations, respectively. The process is illustrated in formulas (13), (14), and (15):

[0155] X0=H(x)*K3 (13)

[0156] X1=H(x)*K5 (14)

[0157] X2=H(x)*K7 (15)

[0158] In the formula, H(x) represents the feature map output after dimensionality reduction, X0 represents the feature map output after 3×3 convolution operation, K3 represents 3×3 convolution operation, X1 represents the feature map output after 5×5 convolution operation, K5 represents 5×5 convolution operation, X2 represents the feature map output after 7×7 convolution operation, and K7 represents 7×7 convolution operation.

[0159] The third step involves subtracting feature map X0 from feature map X1 obtained through parallel convolution to form a new feature map X3, and subtracting feature map X1 from feature map X2 to form a new feature map X4. This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: formulas (16) and (17):

[0160] X3 = X1 - X0 (16)

[0161] X4 = X2 - X1 (17)

[0162] In the formula, X3 represents the feature map obtained by subtracting feature map X0 from feature map X1, and X4 represents the feature map obtained by subtracting feature map X1 from feature map X2.

[0163] The fourth step involves fusing feature map X3 and feature map X4, and then using skip connections to concatenate the original feature maps of the improved residual module along channels to form a new feature map X5. This process fuses the feature maps obtained from the difference operation, enhancing the generalization ability between feature spaces and increasing the amount of information describing image features. Skip connections fuse the feature maps obtained from different convolution operations with the original feature maps, giving the network richer semantic information. The process is illustrated in formula (18).

[0164] X5=x+(X3+X4) (18)

[0165] In the formula, X5 represents the feature map obtained by skip connection fusion, and x represents the original feature map of the input residual module;

[0166] The fifth step is to increase the channel dimension of the feature map through a 1×1 convolution operation;

[0167] After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then enhanced with a modified SE attention mechanism for local information enhancement. This significantly reduces the number of network parameters and computational cost, and also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task. This enables better detection and real-time emptying of the conveyor belt. The improved SE attention mechanism involves three steps for feature extraction, the structure of which is shown below. Figure 4 As shown, the specific situation is as follows:

[0168] The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (19)

[0169]

[0170] In the formula, x c H represents the feature map obtained from the improved residual module. c The height of the feature map, W represents x c The width of the feature map, where i represents x c The pixel index position of the feature map height, where j represents x. c The pixel index position of the feature map width, x c (i,j) represents the coordinate position of a pixel in the feature map, F GAP This indicates that a global average pooling operation is performed on the feature map, Z. c Indicates the feature map x c The feature map after performing H×W global average pooling in the spatial dimension;

[0171] The second step is to adjust Z. c BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (20)

[0172] S c =σ(W2δ(W1Z) c (20)

[0173] In the formula, σ represents the sigmoid activation function, δ represents the ReLU activation function, and S c To obtain normalized weights for the activation function;

[0174] The third step is to transfer Z c With the obtained normalized weight S c Multiplication is performed, and the coefficients of the spatial local feature factors are obtained by multiplication and weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (21) and (22):

[0175] F c =Z c *S c (twenty one)

[0176]

[0177] In the formula, F c These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism;

[0178] The sixth layer architecture is a downsampling operation implemented through average pooling; the average value of all pixels in the feature map of each output channel is calculated to obtain the feature information corresponding to the category;

[0179] The feature maps output by the improved ResNet network module are compressed into the range [0,1] using the Softmax function. The output probability values ​​are then normalized to predict the probability of the highest-scoring category. The loss is calculated using binary cross-entropy, and the network parameters are optimized through backpropagation using the Adam optimizer and loss calculation. During network training, a dynamic learning rate is used. After each specific number of training iterations, a validation set is used to verify the training effect of the network. The training terminates when the loss on the validation set reaches its minimum value, i.e., the total loss on the validation set no longer decreases after 10 consecutive iterations.

[0180] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A real-time empty detection method for conveyor belts based on an improved ResNet network, characterized in that, This improved ResNet network is a two-part improvement on the original ResNet network. The first part improves the residual modules of the original ResNet network: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Convolutional layers were changed to utilize , , Convolutional layers are used to construct parallel feature extraction. Different features are extracted through parallel convolution operations and then difference operations are performed to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the local optimal feature information. The second part is to add an improved SE attention mechanism after the residual module. The improvement of the SE attention mechanism is to change the channel coefficient adjustment in the excitation operation to channel normalization activation, thereby avoiding parameter loss during the channel adjustment process. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, so that the network can optimize the weights of the feature maps extracted at each stage from the perspective of global information. The specific implementation of this real-time empty belt detection method includes the following steps: 1) Before the conveyor belt operation, place the semicircular protractor parallel to the shooting area of ​​the conveyor belt. The camera takes a picture of the calibrated semicircular protractor through the monitoring screen. Calculate the number of unit pixels occupied by the calibrated object in the horizontal direction based on the picture. Calculate the distance between the camera and the conveyor belt according to the ratio, which is the pixel focal length of the camera. 2) After determining the pixel focal length of the camera, images of the conveyor belt during startup and operation are captured by the camera. A custom conveyor belt dataset is constructed, which includes images of an empty conveyor belt, sand, and ore. Data augmentation is performed on the sand and ore images to expand the dataset. The augmented sand, ore, and empty conveyor belt images are divided into training, validation, and test sets according to a set ratio. 3) The training set is fed into the improved ResNet network in batches for training. After the network extracts the features of the images in the training set, the probability value of the image classification is obtained by the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real value collected and labeled by the human. Finally, the weights are updated based on the loss value calculated by the network. The network parameters are adjusted by the optimizer according to the number of iterations during the training process, and the learning rate is updated. After each preset number of network trainings, the training effect of the network is verified by the validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained. The improved ResNet network consists of a six-layer architecture, as detailed below: The first layer architecture is Convolution operations and The max pooling operation is used to adjust the size of the input feature map and downsample the number of channels. The architecture from the second to the fifth layer consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice. The improved residual module extracts features in five steps, as detailed below: The first step is Convolution operations reduce the channel dimension of feature maps, thus reducing the number of parameters required for network training. The second step is to separately... Convolution operation Convolution operation The convolution operation forms three branches of new feature maps in parallel, as shown in formulas (2), (3) and (4): （2）； （3）； （4）； In the formula, This represents the feature map output after dimensionality reduction. Indicates the process The feature map output after the convolution operation express Convolution operation, Indicates the process The feature map output after the convolution operation express Convolution operation, Indicates the process The feature map output after the convolution operation express Convolution operation; The third step involves obtaining feature maps through parallel convolution. With feature map Subtraction forms a new feature map Feature map With feature map Subtraction forms a new feature map This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: (5) and (6): （5）； （6）； In the formula, Representation of feature map With feature map The feature map obtained by subtraction, Representation of feature map With feature map The feature map obtained by subtraction; The fourth step is feature mapping. With feature map After fusion, the original feature maps of the input improved residual module are concatenated along channels using skip connections to form new feature maps. The feature maps obtained by the difference operation are fused together, which enhances the generalization ability between feature spaces and increases the amount of information describing image features. The skip connection fuses the feature maps obtained by different convolution operations with the original feature map, so that the network has richer semantic information. The process is as follows: Formula (7): （7）； In the formula, This represents the feature map obtained by skip connection fusion. This represents the original feature map of the input residual module; The fifth step is to pass. Convolution operations increase the channel dimension of the feature map; After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then enhanced with a modified SE attention mechanism for local information enhancement. This not only significantly reduces the number of network parameters and computational cost, but also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task. The improved SE attention mechanism involves feature extraction in three steps, as detailed below: The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (8) （8）； In the formula, This represents the feature map obtained from the improved residual module. express The height of the feature map, express The width of the feature map express The pixel index position of the feature map height. express The pixel index position of the feature map width. This represents the coordinate position of a pixel in the feature map. This indicates that a global average pooling operation is performed on the feature map. Indicates the feature map Execute in spatial dimension Feature map after global average pooling; The second step is to... BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (9): （9）； In the formula, This represents the sigmoid activation function. This represents the ReLU activation function. To obtain normalized weights for the activation function; The third step is to... With the obtained normalized weights Multiplication is performed, and the coefficients of the spatial local feature factors are obtained by multiplication and weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (10) and (11): （10）； （11）； In the formula, These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism; The sixth layer architecture implements downsampling through average pooling; for each output... The average value of all pixels in the feature map is calculated to obtain the feature information corresponding to the category. The feature maps output by the improved ResNet network are compressed into the range [0,1] using the Softmax function. The output probability values ​​are then normalized to predict the probability of the highest-scoring category. The loss is calculated using binary cross-entropy, and the network parameters are optimized through backpropagation using the Adam optimizer and loss calculation. During network training, a dynamic learning rate is used. After each preset number of training iterations, a validation set is used to verify the training effect of the network. The training terminates when the loss on the validation set reaches its minimum value, i.e., the total loss on the validation set no longer decreases after 10 consecutive iterations. 4) Feed the test set into the optimal network obtained in step 3) to obtain the multi-classification model for predicting whether the conveyor belt is empty. Set the corresponding classification values: set the classification value of an empty conveyor belt to 0, the classification value of sand on the conveyor belt to 1, and the classification value of ore to 2. Record the top 1 test value during the test process and obtain the evaluation index of the optimal accuracy value through the highest test value.

2. The real-time empty detection method for conveyor belts based on an improved ResNet network according to claim 1, characterized in that, In step 1), a CCD industrial camera is installed above the conveyor belt. The camera power supply is connected to the relay, and the relay and the central control system communicate via Socket through the same IP network segment. Before the conveyor belt starts operating, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is captured. Based on the captured image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. While enhancing the foreground of the calibration object, the low-frequency components of the background of the conveyor belt are weakened. This allows for the counting of pixels in the connected area of ​​the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as: （1）； In the formula, Indicates the pixel size of the calibrated image. This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt.

3. The real-time empty detection method for conveyor belts based on an improved ResNet network according to claim 1, characterized in that, In step 2), after determining the camera's pixel focal length, a custom conveyor belt dataset is constructed by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the captured original images using bilinear interpolation to uniformly adjust the size to a resolution of [resolution value missing]. The image undergoes brightness and grayscale transformations, followed by median filtering for noise reduction and deblurring. Morphological processing is then applied, using a closing operation to reduce noise interference from the conveyor belt. Finally, pixel values ​​are normalized and scaled to the [0,1] range. Furthermore, rotation, flipping, and cropping operations are performed on sand and ore images in the training set for data augmentation. Rotation is achieved using… , , and The value can be selected, and the flipping includes horizontal flipping, vertical flipping, and horizontal-vertical flipping. The cropping operation uses a random aspect ratio. or Cropping; dividing the preprocessed images into training, validation, and test sets according to a set ratio.

4. The real-time empty detection method for conveyor belts based on an improved ResNet network according to claim 1, characterized in that, In step 4), the test set is fed into the optimal network to obtain the classification value and probability predicted by the network. When the classification value is 0, the result is an empty conveyor belt, which is determined to be idle. When the classification value is 1, the result is sand, which is determined to be non-idle. When the classification value is 2, the result is ore, which is determined to be non-idle.

5. A real-time empty detection system for conveyor belts based on an improved ResNet network, characterized in that, include: The data acquisition module captures images of the conveyor belt during startup and operation via a camera, and custom-builds a conveyor belt dataset, which includes images of an empty conveyor belt, sand, and ore. Data augmentation is performed on the sand and ore images to expand the dataset, and the augmented sand, ore, and empty conveyor belt images are divided into training, validation, and test sets according to a set ratio. An improved ResNet network module is used to recognize photos taken by a camera, and an improved SE attention mechanism is introduced to judge the feature information of the image area captured on the conveyor belt. The training module is used to train and validate the improved ResNet network module to obtain the optimal network. The empty conveyor belt detection module is used to feed the test set into the optimal network to obtain a multi-classification model for predicting whether the conveyor belt is empty. It sets corresponding classification values: 0 for an empty conveyor belt, 1 for the presence of sand, and 2 for the presence of ore. The module records the top-performing test value during the testing process, using the highest test value as the evaluation metric for optimal accuracy. Specifically, when the identified classification value is 0, the result is an empty conveyor belt, indicating it is running idle; when the identified classification value is 1, the result is sand, indicating it is not running idle; and when the identified classification value is 2, the result is ore, indicating it is not running idle. The improved ResNet network module consists of a six-layer architecture, as detailed below: The first layer architecture is Convolution operations and The max pooling operation is used to adjust the size of the input feature map and downsample the number of channels. The architecture from the second to the fifth layer consists of an improved residual module enhanced by an improved SE attention mechanism. Each layer repeats the improved residual module and the improved SE attention mechanism twice. The improved residual module extracts features in five steps, as detailed below: The first step is Convolution operations reduce the channel dimension of feature maps, thus reducing the number of parameters required for network training. The second step is to separately... Convolution operation Convolution operation The convolution operation forms three branches of new feature maps in parallel, as shown in formulas (13), (14) and (15): （13）； （14）； （15）； In the formula, This represents the feature map output after dimensionality reduction. Indicates the process The feature map output after the convolution operation express Convolution operation, Indicates the process The feature map output after the convolution operation express Convolution operation, Indicates the process The feature map output after the convolution operation express Convolution operation; The third step involves obtaining feature maps through parallel convolution. With feature map Subtraction forms a new feature map Feature map With feature map Subtraction forms a new feature map This captures the contrast of the differences between feature maps obtained through different convolutions, thereby enhancing the feature information between different semantic information of the image. The process is as follows: formulas (16) and (17): （16）； （17）； In the formula, Representation of feature map With feature map The feature map obtained by subtraction, Representation of feature map With feature map The feature map obtained by subtraction; The fourth step is feature mapping. With feature map After fusion, the original feature maps of the input improved residual module are concatenated along channels using skip connections to form new feature maps. The feature maps obtained by the difference operation are fused together, which enhances the generalization ability between feature spaces and increases the amount of information describing image features. The skip connection fuses the feature maps obtained by different convolution operations with the original feature map, so that the network has richer semantic information. The process is as follows: Formula (18): （18）； In the formula, This represents the feature map obtained by skip connection fusion. This represents the original feature map of the input residual module; The fifth step is to pass. Convolution operations increase the channel dimension of the feature map; After the improved residual module, new feature maps are extracted. The feature maps output by the improved residual module are then enhanced with a modified SE attention mechanism for local information enhancement. This not only significantly reduces the number of network parameters and computational cost, but also improves image feature extraction and attention on the conveyor belt, increasing the number of channels in the feature maps useful for the current task. The improved SE attention mechanism involves feature extraction in three steps, as detailed below: The first step is to perform global average pooling on the feature map obtained from the improved residual module, compressing the global spatial information into the channel local factors, thereby enhancing the aggregation of spatial information. The process is as follows: (19) （19）； In the formula, This represents the feature map obtained from the improved residual module. express The height of the feature map, express The width of the feature map express The pixel index position of the feature map height. express The pixel index position of the feature map width. This represents the coordinate position of a pixel in the feature map. This indicates that a global average pooling operation is performed on the feature map. Indicates the feature map Execute in spatial dimension Feature map after global average pooling; The second step is to... BN standard normalization is performed, ReLU activation is applied to the linear features obtained by normalization, then a fully connected operation is performed on the feature map, and finally the normalized weights are obtained through the Sigmoid activation function. Its purpose is to fully capture the channel dependency relationship, learn the nonlinear relationship between channels, and establish the connection between channels between the fully connected layer and the nonlinear activation function. The process is as follows: Formula (20): （20）； In the formula, This represents the sigmoid activation function. This represents the ReLU activation function. To obtain normalized weights for the activation function; The third step is to... With the obtained normalized weights Multiplication is performed, and the coefficients of the spatial local feature factors are obtained by multiplication and weighted channel by channel and applied to each space of the original feature map. This completes the recalibration of the original features by channel attention, thereby enhancing the global receptive field of the feature map. The process is as follows: formulas (21) and (22): （21）； （22）； In the formula, These are the coefficients of spatial local characteristic factors. The output feature map is after the improved SE attention mechanism; The sixth layer architecture implements downsampling through average pooling; for each output... The average value of all pixels in the feature map is calculated to obtain the feature information corresponding to the category.

6. The real-time empty conveyor belt detection system based on an improved ResNet network according to claim 5, characterized in that, Before the data acquisition module collects data, a semicircular protractor needs to be placed parallel to the imaging area of ​​the conveyor belt. The CCD industrial camera takes a picture of the calibrated semicircular protractor through the monitoring screen. The number of unit pixels occupied by the calibrated object in the horizontal direction is calculated from the picture. The distance between the camera and the conveyor belt is calculated proportionally, which is the pixel focal length of the camera. The specific details are as follows: A CCD industrial camera is installed above the conveyor belt. The camera power is connected to the relay, and the relay communicates with the central control system via a socket exchange on the same IP network segment. Before the conveyor belt starts operating, the relay power is turned on, the camera is started, and a semicircular protractor is placed parallel to the shooting area of ​​the conveyor belt. After the camera focuses on the calibration object, an image of the calibration object is captured. Based on the captured image of the calibration object, the focal length ratio is calculated for pixel calibration: First, the image is transformed from the RGB color space to the HSV color space. Then, adaptive median filtering is applied to the image to enhance the brightness component of the calibration object in the frequency domain while maintaining the hue and saturation. This enhances the foreground of the calibration object while weakening the low-frequency components of the conveyor belt background. This allows for the counting of pixels in the connected areas of the calibration object. The actual scale information of the calibration object image is converted into pixel size. The number of pixels occupied in the horizontal direction is calculated based on the calibration object. The distance between the camera and the conveyor belt is calculated proportionally to establish the correspondence between the pixel size of the reference object and the actual measured size. Based on the number of pixels per unit length, the calibration ratio coefficient f is defined as: （12）； In the formula, Indicates the pixel size of the calibrated image. This represents the actual size of the reference object; where the calculated scaling factor f is the pixel focal length on the conveyor belt. After determining the camera's pixel focal length, the data acquisition module constructs a custom conveyor belt dataset by capturing images of the conveyor belt during its operation. The dataset includes images of an empty conveyor belt, sand, and ore. The preprocessing steps include: first, scaling the acquired original images using bilinear interpolation to uniformly adjust the size and resolution. The image undergoes brightness and grayscale transformations, followed by median filtering for noise reduction and deblurring. Morphological processing is then applied, using a closing operation to reduce noise interference from the conveyor belt. Finally, pixel values ​​are normalized and scaled to the [0,1] range. Furthermore, rotation, flipping, and cropping operations are performed on sand and ore images in the training set for data augmentation. Rotation is achieved using… , , and The value can be selected, and the flipping includes horizontal flipping, vertical flipping, and horizontal-vertical flipping. The cropping operation uses a random aspect ratio. or Cropping; dividing the preprocessed images into training, validation, and test sets according to a set ratio.

7. The real-time empty detection system for conveyor belts based on an improved ResNet network according to claim 5, characterized in that, The improved ResNet network module is an improvement on the original ResNet network in two parts. The first part is an improvement on the residual module of the original ResNet network: [The original text appears to be incomplete and requires further context for accurate translation.] Convolutional layers were changed to utilize , , Convolutional layers are used to construct parallel feature extraction. Different features are extracted through parallel convolution operations and then difference operations are performed to extract the feature information of the image contour edges. Finally, the two difference features extracted in parallel are fused to obtain the local optimal feature information. The second part involves adding an improved SE attention mechanism after the residual module. The improvement to the SE attention mechanism is to change the channel coefficient adjustment in the excitation operation to channel normalization activation, thereby avoiding parameter loss during channel adjustment. By aggregating the features extracted by the improved SE attention mechanism, attention enhancement is effectively performed on the features extracted by the residual module, enabling the network to optimize the weights of the feature maps extracted at each stage from the perspective of global information.

8. The real-time empty conveyor belt detection system based on an improved ResNet network according to claim 5, characterized in that, The training module specifically performs the following operations: The training set is fed into the improved ResNet network module in batches for training. After the network extracts the features from the images in the training set, the probability value of image classification is obtained through the Softmax function. The loss value is calculated by combining the binary cross-entropy loss with the real values ​​collected and labeled by humans. Finally, the weights are updated based on the loss value calculated by the network. The network parameters are adjusted by the optimizer according to the number of iterations during the training process, and the learning rate is updated. After each preset number of network training iterations, the training effect of the network is verified by the validation set. The iteration continues until the total loss of the validation set reaches the minimum state, and finally the optimal network after training and verification is obtained.

Images