Dry slag machine slag block video identification detection method and system based on deep learning

Abstract

The invention relates to a slag drying machine slag block video identification detection method and system based on deep learning. The method comprises the following steps: carrying out data preprocessing on a slag block video to obtain a slag block picture frame; carrying out slag block detection on the slag block picture frame through a slag block detection model based on a deep learning algorithm to obtain a detection result; the slag block detection model comprises a feature extraction module, a feature fusion module and a target detection module; wherein the feature extraction module is used for inputting a slag block picture frame into a backbone network based on ResNet combined with OfficientNet to obtain a low-level feature and a high-level feature; the feature fusion module is used for inputting the low-level features and the high-level features into a self-adaptive bidirectional feature fusion network based on slag block multi-scale geometric and semantic features to obtain fusion features; the target detection module is used for inputting the fusion features into a target detection network to obtain a detection result. According to the method, based on deep learning, through multi-level feature extraction and feature fusion, the precision and robustness of slag block identification of the dry slag machine can be improved.

Description

Technical Field

[0001] The present invention belongs to the fields of computer and communication technologies, and particularly relates to a method and system for identifying and detecting slag block videos of a dry slag machine based on deep learning. Background Art

[0002] With the continuous development of industrial automation and intelligent technologies, the dry slag machine, as an important industrial device, plays a crucial role in industries such as metallurgy and mining. The function of the dry slag machine is to process the waste slag in the smelting process and remove it from the production line. During the slag conveying process of the dry slag machine, the shape, size, and quantity of the slag blocks directly affect the working efficiency of the equipment and the smooth progress of the subsequent processing process.

[0003] However, traditional slag block identification methods usually adopt rule-based image processing technologies such as thresholding and edge detection. These methods rely on artificially set rules and feature extraction, have poor adaptability to complex slag block morphologies and environmental changes, and have a slow processing speed, making it difficult to meet the requirements of large-scale video data and real-time detection. In the case of complex or overlapping slag block morphologies, it may not be possible to accurately detect all large slag blocks, resulting in missed detections or false detections. Summary of the Invention

[0004] Based on this, in view of the above technical problems, the purpose of the present invention is to provide a method for identifying and detecting slag block videos of a dry slag machine based on deep learning, aiming to achieve precise detection and analysis of the slag blocks of the dry slag machine through deep learning technology, thereby effectively improving the slag conveying efficiency of the dry slag machine, reducing bad operations, and improving the overall performance and quality of the production line.

[0005] In a first aspect, the present application provides a method for identifying and detecting slag block videos of a dry slag machine based on deep learning, the method comprising:

[0006] Performing data preprocessing on the slag block video to obtain slag block picture frames;

[0007] Performing slag block detection on the slag block picture frames through a slag block detection model based on a deep learning algorithm to obtain a detection result, the detection result including at least one of whether there are large slag blocks, the size of the large slag blocks, the quantity of the large slag blocks, and the positions of the large slag blocks, and the slag block detection model including a feature extraction module, a feature fusion module, and an object detection module;

[0008] Wherein, the feature extraction module is configured to input the slag block picture frames into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features, and the backbone network includes a ResNet part and an EfficientNet part;

[0009] The feature fusion module is used to input the low-level features and high-level features into an adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of slag blocks to obtain fused features;

[0010] The object detection module is used to input the fused features into an object detection network to obtain detection results.

[0011] In one embodiment, the feature extraction module includes:

[0012] The low-level feature extraction subunit is used to:

[0013] Input the slag block picture frame into the initial convolutional layer of the ResNet part to obtain an initial feature map;

[0014] Input the initial feature map into the residual unit of the ResNet part for feature extraction to obtain low-level features;

[0015] The high-level feature extraction subunit is used to:

[0016] Input the low-level features into the convolutional layer of the EfficientNet part for feature extraction through depthwise separable convolution to obtain intermediate features;

[0017] Pass the intermediate features through the extended residual unit of the EfficientNet part to obtain high-level features.

[0018] In one embodiment, the feature fusion module includes:

[0019] The scale decomposition subunit is used for the adaptive bidirectional feature fusion network to perform scale decomposition on the low-level features and high-level features based on the multi-scale geometric features of slag blocks by using a multi-scale convolutional kernel group, generating multiple low-level sub-feature maps and high-level sub-feature maps corresponding to different size ranges;

[0020] The adaptive bidirectional fusion subunit is used to:

[0021] In the forward fusion process, weight the slag block semantic feature regions in the low-level sub-feature maps through an attention module based on the slag block semantic features to obtain weighted low-level sub-feature maps;

[0022] Sample and extract features from the weighted low-level sub-feature maps through deformable convolution based on the slag block geometric features to obtain optimized low-level sub-feature maps;

[0023] Gradually fuse the optimized low-level sub-feature maps with the high-level sub-feature maps according to different scales to obtain preliminary fused features;

[0024] In the reverse fusion process, magnify the size of the high-level sub-feature maps through upsampling technology to obtain magnified high-level sub-feature maps;

[0025] The enlarged high-level sub-feature map and the low-level sub-feature map are combined through skip connections to obtain optimized comprehensive sub-feature map features;

[0026] The preliminary fusion features and the optimized high-level sub-feature map features are fused to obtain fused features.

[0027] In one embodiment, the object detection network includes a class prediction sub-module and a location prediction sub-module;

[0028] Among them, the class prediction sub-module is used to input the fused features into a deep neural network for classification prediction to obtain class detection results, and the class detection results are used to characterize whether there are large slag blocks in the slag block picture frame;

[0029] The location prediction sub-module is used for:

[0030] Based on the anchor-free mechanism, the fused features are subjected to size detection to obtain the size of the large slag block;

[0031] By identifying and threshold screening the slag block area of the fused features, and based on the counting mechanism, the number of slag block areas that meet the preset conditions is counted to obtain the number of large slag blocks;

[0032] The fused features are subjected to location prediction through bounding box regression prediction to obtain the location of the large slag block.

[0033] In one embodiment, the loss function calculation formula for predicting the location of the fused features through bounding box regression prediction is:

[0034]

[0035] Among them, represents the loss function value; represents the intersection area of the predicted slag block bounding box and the actual slag block bounding box; represents the union area of the predicted slag block bounding box and the actual slag block bounding box; represents the area of the smallest circumscribed rectangle containing the predicted slag block bounding box and the actual slag block bounding box.

[0036] In one embodiment, preprocessing the slag block video to obtain slag block picture frames includes:

[0037] Based on the motion speed and morphological change characteristics of the slag blocks in the slag block video, a dynamic inter-frame difference model is constructed;

[0038] The slag block video is analyzed through the dynamic inter-frame difference model to obtain an adaptive frame sampling strategy, and the slag block video is sampled according to the adaptive frame sampling strategy to generate target slag block video frames;

[0039] The target slag block video frames are segmented by an adaptive threshold segmentation algorithm based on gray level and texture features to obtain slag block area images;

[0040] Perform image enhancement and denoising processing on the slag block area image to obtain a slag block picture frame.

[0041] In one embodiment, the method further includes:

[0042] When the detection result indicates the presence of large slag blocks, the detection result is abnormal;

[0043] When the detection result is abnormal, generate an alarm message and perform regulation through the DCS system in combination with the device operation status data;

[0044] Among them, the device operation status data includes the opening and closing status of the shut-off door, the extrusion parameters of the shut-off door, and the conveyor belt speed, and the alarm message includes the alarm name, alarm time, event level, and location area.

[0045] In a second aspect, the present application further provides a dry slag machine slag block video recognition and detection system based on deep learning. The system includes:

[0046] A frame acquisition module for performing data preprocessing on the slag block video to obtain a slag block picture frame;

[0047] A slag block detection module for performing slag block detection on the slag block picture frame through a slag block detection model based on a deep learning algorithm to obtain a detection result. The detection result includes at least one of whether there are large slag blocks, the size of the large slag blocks, the number of large slag blocks, and the location of the large slag blocks. The slag block detection model includes a feature extraction module, a feature fusion module, and an object detection module;

[0048] Among them, the feature extraction module is used to input the slag block picture frame into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features. The backbone network includes a ResNet part and an EfficientNet part;

[0049] The feature fusion module is used to input the low-level features and high-level features into an adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of the slag blocks to obtain fused features;

[0050] The object detection module is used to input the fused features into an object detection network to obtain a detection result.

[0051] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method in any one of the first aspects are implemented.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the first aspects are implemented.

[0053] In the above method and system for video recognition and detection of slag blocks based on deep learning, as well as the computer device and storage medium, first, data preprocessing is performed on the slag block video to convert the dynamic information in the video into static frame images, obtaining picture frames of the slag blocks, which provides a basis for subsequent detection processing. And based on the deep learning algorithm, the slag block picture frames are input into the slag block detection model for processing to obtain the detection results. The detection results include information such as whether there are large slag blocks, the size of the large slag blocks, the number of large slag blocks, and the positions of the large slag blocks, providing reliable feedback information for the management personnel so as to take corresponding operation measures. This method constructs a slag block detection model through deep learning technology, can automatically, quickly and accurately detect the slag block features, can realize the effective detection and recognition of the slag blocks in the dry slag machine, and further improve the efficiency of the slag conveying process.

[0054] The slag block detection model consists of a feature extraction module, a feature fusion module and an object detection module. Among them, the feature extraction module inputs the slag block picture frames into the backbone network based on ResNet combined with EfficientNet to extract low-level and high-level features. The ResNet part is responsible for capturing deep image features, while the EfficientNet part can provide richer feature information while maintaining computational efficiency. This process helps to identify the key information of the slag blocks in the multi-level features, further improving the accuracy of slag block detection. The feature fusion module inputs the low-level features and high-level features into the adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of the slag blocks. This module can combine geometric features and semantic information at different scales through an adaptive fusion method to obtain more refined fusion features, providing strong support for subsequent object detection. The object detection module inputs the fusion features into the object detection network. By analyzing the fused features, it can identify and calibrate the position, size, quantity, etc. of the large slag blocks in real time, obtain the detection results of the slag blocks, and provide accurate feedback and control for the operation of the dry slag machine.

[0055] Compared with the traditional slag block recognition methods, this method can automatically learn and extract more comprehensive and robust features through deep neural technology, thereby improving the accuracy of slag block recognition. In addition, through multi-scale feature extraction and fusion, this method can enhance the detection ability of the slag block detection model for large slag blocks, small slag blocks and overlapping slag blocks, can realize accurate and efficient video recognition and detection of the slag blocks in the dry slag machine, effectively improve the automation level of the slag conveying process, reduce unnecessary manual intervention and resource waste, improve the production efficiency and equipment utilization rate of the dry slag machine, and ensure the stability and quality of production. Brief Description of the Drawings

[0056] To more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the related art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0057] Figure 1 Flowchart of the slag block video recognition and detection method based on deep learning provided for an exemplary embodiment of the present invention;

[0058] Figure 2 Schematic diagram of the structure of the slag block video recognition and detection system based on deep learning provided for an exemplary embodiment of the present invention. Detailed implementation manners

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In one embodiment, as Figure 1 shown, a slag block video recognition and detection method based on deep learning is provided. In this embodiment, the application of this method to a terminal is taken as an example for illustration. It can be understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is realized through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] S101: Perform data preprocessing on the slag block video to obtain slag block picture frames.

[0062] Specifically, the slag block video can be collected in real time by a camera installed in the slag conveying area of the dry slag machine, such as the observation window of the dry slag machine shut-off door or the observation window of the slag discharge funnel at the tail of the dry slag machine. Schematically, the slag block video can be used to extract individual slag block picture frames at a certain frame rate, such as extracting a fixed number of frames per second, and the slag block picture frames can be denoised by a filtering algorithm to smooth the image and improve the image quality. For example, median filtering can remove the salt-and-pepper noise in the slag block picture frames. By setting an appropriate filtering window size, the median value of the pixels in the window is used to replace the value of the central pixel, effectively removing isolated noise points; for the Gaussian noise in the slag block picture frames, Gaussian filtering can be used, and an appropriate filtering kernel is designed according to parameters such as the standard deviation of the noise to smooth the image and improve the image quality. Or, according to the temporal characteristics of the slag block video, suitable picture frames can be dynamically selected to ensure that while retaining key information, redundant image data is avoided, thereby improving the efficiency and accuracy of subsequent slag block detection.

[0063] S102: Detect slag chunks in the slag chunk image frames through a slag chunk detection model based on a deep learning algorithm to obtain detection results. The detection results include at least one of whether there are large slag chunks, the size of large slag chunks, the number of large slag chunks, and the positions of large slag chunks. The slag chunk detection model includes a feature extraction module, a feature fusion module, and an object detection module. Among them, the feature extraction module is used to input the slag chunk image frames into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features. The backbone network includes a ResNet part and an EfficientNet part; the feature fusion module is used to input the low-level features and high-level features into an adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of slag chunks to obtain fused features; the object detection module is used to input the fused features into an object detection network to obtain detection results.

[0064] Specifically, the ResNet (Residual Network) part, by introducing a residual structure, focuses on the local details of the image at a relatively shallow backbone network level stage, extracts low-level features such as edges, textures, colors, and corners, which helps the backbone network identify the basic shapes and contours of slag chunks. And due to the residual connections in ResNet, the backbone network can avoid the common gradient disappearance problem during the training process, thus better capturing these basic low-level features. The EfficientNet (Efficient Neural Network) part, by expanding the residual units, enables the backbone network to extract more abstract and complex high-level features, such as semantic information about the shape, size, and position of slag chunks, which helps the backbone network understand the overall structure of slag chunks and their positions in the image. In the feature fusion module, by fusing the low-level features and high-level features based on the multi-scale geometric and semantic features of slag chunks, fused features including low-level local information and high-level global semantic information are obtained, enabling the subsequent object detection module to more accurately identify slag chunks based on these fused features. The object detection network takes the fused features output by the feature fusion module as input, through a series of complex operations such as convolution and pooling, as well as corresponding classification and localization mechanisms, through in-depth analysis and processing of the fused features, distinguishes slag chunks from the background and other irrelevant objects, and gives specific detection results. The detection results include key information such as whether there are large slag chunks, the size of large slag chunks, the number of large slag chunks, and the positions of large slag chunks, which helps the smooth progress of operations such as the operation monitoring of dry slag machines and the subsequent cleaning and processing of slag chunks.

[0065] The above-mentioned dry slag machine slag block video recognition and detection method based on deep learning preprocesses the data of the slag block video, extracts the slag block picture frames, and combines the deep learning algorithm to detect the slag blocks through the slag block detection model, thereby realizing the accurate recognition and monitoring of the slag blocks. In the slag block detection model, first, a backbone network combining ResNet and EfficientNet is used for feature extraction to extract low-level and high-level image features, ensuring the effective capture of details and global information. Secondly, the feature fusion module adopts an adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of the slag blocks to fuse the low-level and high-level features and generate more representative fusion features, further improving the detection accuracy and robustness. Finally, based on the fusion features, the target detection module accurately identifies key information such as the presence, size, quantity, and location of the slag blocks through the target detection network. This process effectively improves the real-time performance and accuracy of slag block detection, can monitor the operating state of the dry slag machine in real time, promptly discover and locate large slag blocks, and help the management personnel quickly take effective measures for processing, thereby ensuring the normal operation of the equipment and improving production efficiency. Compared with the traditional manual detection method, this method greatly improves the efficiency and accuracy of slag block detection through deep learning and reduces the interference of human errors.

[0066] Exemplarily, preprocessing the data of the slag block video to obtain the slag block picture frames includes:

[0067] Constructing a dynamic inter-frame difference model based on the motion speed and morphological change characteristics of the slag blocks in the slag block video;

[0068] Analyzing the slag block video through the dynamic inter-frame difference model to obtain an adaptive frame sampling strategy, and sampling the slag block video according to the adaptive frame sampling strategy to generate target slag block video frames;

[0069] Segmenting the target slag block video frames by an adaptive threshold segmentation algorithm based on gray level and texture features to obtain slag block region images;

[0070] Performing image enhancement and denoising processing on the slag block region images to obtain the slag block picture frames.

[0071] Since the movement speed and shape of the slag block are constantly changing during the slag conveying process of the dry slag machine. For example, the slag block may move slowly at the beginning under the push of the conveyor belt, and then speed up due to the acceleration of the conveyor belt or collision with other slag blocks. It may be originally in block shape, but become irregular shape after collision and extrusion. Therefore, the displacement of the slag block per unit time can be determined by calculating the coordinate change of the slag block in the image coordinate system and combining the frame rate and other information of the video, so as to measure the speed change. The edge contour of the slag block can be extracted by the edge detection algorithm, and the contour changes in the previous and next frames can be compared to calculate the area change of the slag block area and the change in aspect ratio. By combining these parameters related to the movement speed and shape change, a dynamic frame difference model that can reflect the degree of difference between adjacent frames can be constructed. The slag block video is input frame by frame into the constructed dynamic frame difference model. The model will output a difference value for each pair of adjacent frames, which reflects the comprehensive change degree of the slag block between the two frames. The adaptive frame sampling strategy is formulated based on these difference values. For example, if the difference value of adjacent frames exceeds a pre-set larger threshold, it means that the slag block has undergone significant changes between the two frames, which may be a sudden increase in movement speed or a significant change in shape. In this case, both frames will be selected as key frames for subsequent processing; if the difference value is less than a smaller threshold, it means that the slag block has not changed much between the two frames and is relatively stable. In this case, some frames can be skipped to reduce unnecessary data and improve processing efficiency. The target slag block video frame is obtained by sampling the slag block video according to the adaptive frame sampling strategy. This target slag block video frame can more effectively reflect the key changes of the slag block in the entire slag conveying process, and provide a more valuable data basis for subsequent image segmentation and other processing.

[0072] In the target slag block video frame, there are both slag block parts and other information such as the background. An adaptive threshold segmentation algorithm based on gray-scale and texture features can be used to separate the slag block from the complex image background. Among them, the gray-scale feature reflects the reflection and absorption of light by the slag block, making it present a specific black-and-white degree range on the image; the texture feature reflects the unique structural patterns such as particles and blocks on the surface of the slag block. Schematically, by analyzing the gray-scale value distribution of each pixel point in the target slag block video frame, information such as the number and distribution law of pixels in different gray-scale intervals is statistically analyzed. And its texture characteristics can be quantified through a gray-level co-occurrence matrix, etc., to determine the range of characteristic values corresponding to the unique texture pattern of the slag block. According to the comprehensive analysis results of these gray-scale and texture features, one or more thresholds are adaptively calculated, and the pixel points in the target slag block video frame whose gray-scale values and texture features conform to the slag block characteristics are divided into the slag block area, and the remaining pixel points are determined as the background area, so as to realize image segmentation and obtain a regional image containing only the slag block, which is convenient for subsequent further focusing on the slag block itself for processing. Finally, the contrast of the slag block area image can be enhanced through histogram equalization, making the details such as the contour and texture of the slag block clearer, and denoising can be performed through median filtering or Gaussian filtering to improve the image quality and obtain the slag block picture frame.

[0073] In an exemplary embodiment, the feature extraction module includes:

[0074] A low-level feature extraction subunit for:

[0075] Input the slag block picture frame into the initial convolutional layer of the ResNet part to obtain an initial feature map;

[0076] Input the initial feature map into the residual unit of the ResNet part for feature extraction to obtain low-level features;

[0077] A high-level feature extraction subunit for:

[0078] Input the low-level features into the convolutional layer of the EfficientNet part for feature extraction through depthwise separable convolution to obtain intermediate features;

[0079] Pass the intermediate features through the extended residual unit of the EfficientNet part to obtain high-level features.

[0080] Specifically, the initial convolutional layer of ResNet performs preliminary feature extraction on the input slag block image frame by using convolutional kernels of specific sizes and quantities, such as the commonly used 3×3 convolutional kernel, etc., and sliding on the slag block image frame for convolutional operations. As the convolutional kernel slides sequentially on the slag block image frame according to the preset stride, calculations such as weighted summation are performed on the pixels at each corresponding position to obtain the initial feature map. This process transforms the original image information into a feature map form that is more convenient for subsequent network processing. Through the residual units of the ResNet part of the initial feature map, multiple convolutional layers, batch normalization layers, and activation functions are used for step-by-step feature extraction and transformation to obtain low-level features. These low-level features can accurately reflect the slag visual information in the slag block image, such as the specific texture morphology on the surface of the slag block, the precise contour of the edge, and the color change situation in the local area, etc., providing a basis for subsequent high-level feature extraction, etc. And the residual units of ResNet, through their unique shortcut connection and convolutional path structure, effectively solve the problems of gradient disappearance and gradient explosion that occur during the training of the neural network, enabling the network to be stably trained, ensuring that valuable features can be accurately extracted from the initial feature map, and preventing training failure due to the increase in network depth.

[0081] In the high-level feature extraction subunit, depthwise separable convolution splits the traditional convolutional operation into two steps: depth convolution and pointwise convolution, so as to reduce the computational amount while being able to extract features more meticulously. Schematically, for the obtained low-level features, depth convolution can be performed on each channel separately, and then channel fusion and transformation are performed through pointwise convolution. Under the condition of effectively utilizing computing resources, more complex and abstract intermediate features are extracted, enabling the model to operate efficiently even when the hardware resources are limited, and helping to prevent overfitting and improve the generalization ability of the model. The intermediate features are input into the extended residual unit for further processing. By optimizing the network structure, such as adjusting parameters such as the convolutional kernel size and stride, the extended residual unit can expand the receptive field of neurons, and then relevant information about the slag block can be inferred from the semantic level, such as the type of the slag block or its role in the production process, etc., to obtain more comprehensive high-level features. These high-level features help to more accurately determine the attributes such as the position and size of the slag block in the subsequent detection process, and can better adapt to the changes of the slag block in different scenarios, improving the robustness and accuracy of the model.

[0082] In an exemplary embodiment, the feature fusion module includes:

[0083] A scale decomposition subunit for adaptively bidirectional feature fusion network to perform scale decomposition on low-level features and high-level features by using a multi-scale convolutional kernel group based on the multi-scale geometric features of the slag block, generating multiple low-level sub-feature maps and high-level sub-feature maps corresponding to different size ranges;

[0084] An adaptive bidirectional fusion subunit, configured to:

[0085] In the forward fusion process, weight the slag block semantic feature regions in the low-level sub-feature map through an attention module based on the slag block semantic features to obtain a weighted low-level sub-feature map;

[0086] Sample and extract features from the weighted low-level sub-feature map based on the slag block geometric features through deformable convolution to obtain an optimized low-level sub-feature map;

[0087] Gradually fuse the optimized low-level sub-feature map with the high-level sub-feature map at different scales to obtain a preliminary fusion feature;

[0088] In the reverse fusion process, magnify the size of the high-level sub-feature map through an upsampling technique to obtain an enlarged high-level sub-feature map;

[0089] Combine the enlarged high-level sub-feature map with the low-level sub-feature map through a skip connection to obtain an optimized comprehensive sub-feature map feature;

[0090] Fuse the preliminary fusion feature and the optimized high-level sub-feature map feature to obtain a fusion feature.

[0091] Specifically, slag blocks often exhibit different geometric characteristics such as different sizes and shapes in the image, presenting a multi-scale situation. Therefore, a multi-scale convolution kernel group can be used to perform convolution on the low-level features and high-level features respectively, thereby generating multiple low-level sub-feature maps and high-level sub-feature maps corresponding to different size ranges. For example, after a small-size convolution kernel acts on the low-level features, a low-level sub-feature map reflecting the local details and small-range regions of the slag block can be obtained; after a large-size convolution kernel acts, a low-level sub-feature map covering a wider area and reflecting the overall part features of the slag block can be obtained. This process can fully consider the multi-scale characteristics of slag blocks in the image, ensure that the feature information of slag blocks of different sizes is not missed, provide a rich feature basis for subsequent more accurate feature fusion and slag block detection, enable the model to adapt to the actual situation of diverse slag block sizes, and improve the detection ability for slag blocks of different sizes.

[0092] In the adaptive bidirectional fusion sub-unit, in the forward fusion direction, the attention module based on the slag block semantic features can analyze which regions in the low-level sub-feature map can better reflect the semantic features of the slag block. For example, some texture feature regions may be associated with the semantics of specific types of slag blocks. By weighting these regions, the key features of the slag block are highlighted, and a weighted low-level sub-feature map is obtained. This process can help the network focus on the more relevant parts of the image and suppress irrelevant background or other interference information. Specifically, deformable convolution is a flexible convolution operation that can be dynamically adjusted according to the geometric features of the input image, enabling the convolution operation to adapt to the geometric shapes in the image, so as to better process slag blocks of different shapes and angles. Sampling and feature extraction are performed on the weighted low-level feature map through deformable convolution. According to the actual shape, boundary and other geometric information of the slag block, the sampling position of the convolution kernel can be dynamically adjusted, so that the optimized low-level sub-feature map can more accurately reflect the true shape, boundary and other geometric characteristics of the slag block, improving the accuracy of subsequent feature fusion and the judgment of the shape and position of the slag block during slag block detection. Subsequently, the optimized low-level sub-feature map and the corresponding high-level sub-feature map can be fused by appropriate fusion algorithms such as element-wise addition and concatenation in the order of scale from low to high to obtain the preliminary fusion features. Through the fusion at different scales, the network can take into account feature information at different levels, thereby improving the overall expression ability.

[0093] In the reverse fusion direction, upsampling techniques such as common bilinear interpolation and transposed convolution can increase the size of the image. By using upsampling techniques to enlarge the size of the high-level feature map, its size can be matched with that of the low-level feature map to obtain the enlarged high-level sub-feature map, thereby avoiding problems such as fusion difficulties or information loss caused by size differences and ensuring the smooth progress of subsequent feature fusion. The enlarged high-level sub-feature map is connected to the low-level sub-feature map using skip connections, directly transmitting the information in the high-level features to the corresponding positions in the low-level, which can help retain the detailed information of the low-level features while enhancing the overall information of the high-level features, obtaining an optimized comprehensive sub-feature map that combines low-level and high-level features, further improving the quality of feature representation. By fusing the preliminary fusion features obtained from the forward fusion and the optimized comprehensive sub-feature map features obtained from the reverse fusion through appropriate fusion methods such as weighted summation and feature concatenation, the final fusion features are obtained. These fusion features contain a fine combination of low-level and high-level information, can provide a comprehensive feature description, and are suitable for accurate detection and analysis of slag blocks.

[0094] In an exemplary embodiment, the object detection network includes a class prediction sub-module and a location prediction sub-module;

[0095] Among them, the category prediction sub-module is used to input the fused features into a deep neural network for classification prediction to obtain a category detection result, which is used to characterize whether there are large slag blocks in the slag block picture frame;

[0096] The position prediction sub-module is used to:

[0097] Based on the anchor-free mechanism, perform size detection on the fused features to obtain the large slag block size;

[0098] Identify the slag block area and perform threshold screening on the fused features, and based on the counting mechanism, count the number of slag block areas that meet the preset conditions to obtain the number of large slag blocks;

[0099] Perform position prediction on the fused features through bounding box regression prediction to obtain the large slag block position.

[0100] Specifically, after obtaining the fused features from the feature fusion module, the category prediction sub-module will use them as inputs and send them into the deep neural network. The deep neural network can automatically learn based on these rich feature bases what feature combinations correspond to the existence of large slag blocks and what feature combinations indicate the absence of large slag blocks, and output the category detection result. In the position prediction sub-module, the anchor-free mechanism directly predicts the size of the large slag block based on the information in the fused features, without the need to set a fixed anchor box template in advance, and can extract and calculate the corresponding size-related features of the fused features by setting specific branches or modules in the deep neural network to obtain the large slag block size. Compared with the anchor-based mechanism, the anchor-free mechanism avoids problems caused by unreasonable anchor box settings, such as a decrease in detection accuracy due to inaccurate settings of parameters such as anchor box size and ratio, making the detection of the large slag block size more accurate and in line with the actual situation.

[0101] Schematically, after the slag block area can be identified based on the fused features using a deep neural network, certain threshold conditions can be set for screening. These thresholds may involve factors such as the feature intensity and area size of the slag block area. For example, only the slag block areas with feature intensity exceeding a certain threshold and area reaching a certain set value are retained, excluding some areas that may be misjudged or too small to meet the definition of "large slag blocks", and based on a counting mechanism, the number of slag block areas that meet the preset conditions after screening is counted, and finally the number of large slag blocks is obtained. Through this method of region recognition, threshold screening, and counting based on fused features, interference factors can be excluded, and the number of truly compliant large slag blocks in the image can be accurately determined, avoiding errors in quantity statistics caused by misjudgment or including small slag blocks, etc., providing an important quantitative basis for comprehensively grasping the distribution of slag blocks. In addition, bounding box regression prediction refers to directly predicting the bounding box coordinates of each slag block through a regression network, that is, the position and size of the box. Through a deep neural network, based on the fused features, the mapping relationship between the fused features and the actual position of the large slag block is learned, and then the specific position coordinates of the large slag block in the image are predicted. For example, the upper left and lower right coordinates of the rectangular bounding box where the large slag block is located are predicted, or the position of the large slag block is represented in the form of the center point coordinates and width and height, etc., so as to determine the specific position of the large slag block in the slag block picture frame.

[0102] In an exemplary embodiment, the calculation formula of the loss function for predicting the position of the fused features through bounding box regression prediction is:

[0103]

[0104] where L GIoU represents the loss function value; A I represents the intersection area of the predicted slag block bounding box and the actual slag block bounding box; A U represents the union area of the predicted slag block bounding box and the actual slag block bounding box; A C represents the area of the smallest circumscribed rectangle containing the predicted slag block bounding box and the actual slag block bounding box.

[0105] Specifically, the loss function measures the degree of difference between the predicted slag block bounding box and the actual slag block bounding box. The smaller the loss value, the closer the prediction result is to the real situation; the larger the loss value, the lower the accuracy of the prediction. A I The intersection area reflects the overlapping part of the predicted slag block bounding box and the actual slag block bounding box. The larger the overlapping area, the closer the prediction of the position is to the real situation. A U The union area considers the total area covered by the predicted slag block bounding box and the actual slag block bounding box, and is used together with the intersection area to calculate a relative relationship between the two, in order to more comprehensively measure the accuracy of the position prediction. A CThe minimum bounding rectangle provides a larger reference range. By calculating the intersection area and union area, it can better reflect the differences in position and size between the predicted slag block bounding box and the actual slag block bounding box, enabling the loss function to more accurately evaluate the quality of the prediction results. This loss function formula can effectively evaluate the position accuracy of the bounding box regression prediction by comprehensively considering factors such as the intersection, union, and minimum bounding rectangle of the predicted slag block bounding box and the actual slag block bounding box.

[0106] In an exemplary embodiment, the method further includes:

[0107] When the detection result indicates the presence of large slag blocks, the detection result is abnormal;

[0108] When the detection result is abnormal, an alarm message is generated and regulation is performed through the DCS system in combination with the device operation status data;

[0109] Among them, the device operation status data includes the opening and closing status of the shut-off door, the extrusion parameters of the shut-off door, and the conveyor belt speed, and the alarm message includes the alarm name, alarm time, event level, and location area.

[0110] Illustratively, when the contour size of the detected slag block exceeds the preset slag block size threshold, it is determined that there are large slag blocks, and at this time the detection result is abnormal; if the sizes of all slag blocks are within the normal range, the detection result is normal. When the detection result is abnormal, through the alarm and regulation mechanism, an alarm message is first generated according to the detection result and sent to the staff. Among them, the alarm name can clearly point out the key to the problem, such as "Alarm for risk of large slag block blockage", etc., so that the staff can quickly understand the core of the problem. The alarm time accurately records the moment when the abnormality occurs, facilitating subsequent tracing and analysis of the time node when the problem appears. And the event level is divided according to the severity of the problem, such as minor, serious, urgent, etc., so that the staff can intuitively know the urgency of response. The location area accurately indicates the specific location where the abnormality occurs, such as near the shut-off door, a certain section of the conveyor belt, etc., facilitating the staff to quickly locate and troubleshoot the problem.

[0111] In addition, this method combines the device operation status data obtained in real time, such as the opening and closing status of the shut-off door, the extrusion parameters of the shut-off door, and the conveyor belt speed, etc., and conducts regulation through DCS (Distributed Control System). For example, starting the slag squeezing operation, adjusting the conveyor belt speed, or changing the device operation status, etc. This process can improve the automatic control level of the slag conveying process, reduce human intervention, and minimize the failure risk and the probability of equipment damage. Schematically, the DCS system is an automatic control system widely used in the field of industrial process control, such as power plants, chemical industries, oil and gas, water treatment, etc. It can disperse multiple control function modules to different on-site areas through a distributed architecture and centrally manage them based on a communication network, thereby realizing the monitoring and control of the entire production process.

[0112] Based on the same inventive concept, as Figure 2 shown, the embodiment of the present application also provides a dry slag machine slag block video recognition and detection system 200 based on deep learning. The system includes:

[0113] A frame acquisition module 201, configured to perform data preprocessing on the slag block video to obtain a slag block picture frame;

[0114] A slag block detection module 202, configured to perform slag block detection on the slag block picture frame through a slag block detection model based on a deep learning algorithm to obtain a detection result. The detection result includes at least one of whether there is a large slag block, the size of the large slag block, the number of large slag blocks, and the position of the large slag block. The slag block detection model includes a feature extraction module 2021, a feature fusion module 2022, and a target detection module 2023;

[0115] Among them, the feature extraction module 2021 is configured to input the slag block picture frame into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features. The backbone network includes a ResNet part and an EfficientNet part;

[0116] The feature fusion module 2022 is configured to input the low-level features and high-level features into an adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of the slag block to obtain fusion features;

[0117] The target detection module 2023 is configured to input the fusion features into a target detection network to obtain a detection result.

[0118] The system is divided into two main modules. The frame acquisition module 201 acquires the video data of slag blocks in real time, preprocesses the video data, and obtains the slag block picture frames. These picture frames provide real-time image data for slag block detection, ensuring the timeliness and accuracy of data acquisition. Secondly, through the slag block detection module 202, the processed slag block picture frames are input into the slag block detection model based on deep learning for detection, so as to obtain the slag block detection results. The detection results include at least one of whether there are large slag blocks, the size of the large slag blocks, the number of large slag blocks, and the specific positions of the large slag blocks, which helps the management personnel understand the slag conveying process, make judgments on the operating status of the dry slag conveyor in a timely manner, take corresponding measures in advance, and avoid equipment failures or affecting the normal production process caused by slag block problems.

[0119] In the slag block detection model, the feature extraction module 2021 extracts low-level features and high-level features by inputting the slag block picture frames into the backbone network combined with ResNet and EfficientNet. The backbone network consists of two parts, ResNet and EfficientNet, which can effectively extract different levels of features of the image and improve the quality of feature extraction of the slag block image. And the feature fusion module 2022 inputs the low-level and high-level features into the adaptive bidirectional feature fusion network based on the multi-scale geometric and semantic features of the slag blocks, which can not only retain the unique advantages of each layer of features, but also eliminate redundant information, so as to obtain more optimized and comprehensive fusion features. Finally, the object detection module 2023 inputs the fusion features into the object detection network to obtain the final detection results. Through the collaborative work of different modules, the system can detect the large slag block information in the slag block video in real time through multi-scale feature extraction based on deep learning, effectively improving the accuracy, precision and robustness of slag block detection, and ensuring the real-time monitoring and accurate identification of slag blocks.

[0120] Furthermore, the feature extraction module 2021 includes:

[0121] The low-level feature extraction subunit is used to:

[0122] Input the slag block picture frames into the initial convolutional layer of the ResNet part to obtain the initial feature map;

[0123] Input the initial feature map into the residual unit of the ResNet part for feature extraction to obtain low-level features;

[0124] The high-level feature extraction subunit is used to:

[0125] Input the low-level features into the convolutional layer of the EfficientNet part for feature extraction through depthwise separable convolution to obtain intermediate features;

[0126] The intermediate features are passed through the extended residual units of the EfficientNet part to obtain high-level features.

[0127] Furthermore, the feature fusion module 2022 includes:

[0128] A scale decomposition subunit, which is used for the adaptive bidirectional feature fusion network to perform scale decomposition on the low-level features and high-level features by using a multi-scale convolution kernel group based on the multi-scale geometric features of the slag blocks, and generate multiple low-level sub-feature maps and high-level sub-feature maps corresponding to different size ranges;

[0129] An adaptive bidirectional fusion subunit, which is used for:

[0130] In the forward fusion process, the slag block semantic feature regions in the low-level sub-feature maps are weighted by an attention module based on the slag block semantic features to obtain weighted low-level sub-feature maps;

[0131] The weighted low-level sub-feature maps are sampled and feature-extracted by deformable convolution based on the slag block geometric features to obtain optimized low-level sub-feature maps;

[0132] The optimized low-level sub-feature maps are gradually fused with the high-level sub-feature maps at different scales to obtain preliminary fusion features;

[0133] In the reverse fusion process, the high-level sub-feature maps are enlarged in size by upsampling technology to obtain enlarged high-level sub-feature maps;

[0134] The enlarged high-level sub-feature maps are combined with the low-level sub-feature maps through skip connections to obtain optimized comprehensive sub-feature map features;

[0135] The preliminary fusion features and the optimized high-level sub-feature map features are feature-fused to obtain fusion features.

[0136] Furthermore, the object detection network includes a class prediction sub-module and a location prediction sub-module;

[0137] Among them, the class prediction sub-module is used to input the fusion features into a deep neural network for classification prediction to obtain class detection results, and the class detection results are used to characterize whether there are large slag blocks in the slag block picture frame;

[0138] The location prediction sub-module is used for:

[0139] Performing size detection on the fusion features based on the anchor-free mechanism to obtain the large slag block size;

[0140] Identifying and threshold-screening the slag block regions of the fusion features, and counting the number of slag block regions that meet the preset conditions based on the counting mechanism to obtain the large slag block number;

[0141] The fused features are used for position prediction through bounding box regression prediction to obtain the positions of large slag blocks.

[0142] Furthermore, the calculation formula of the loss function for position prediction of the fused features through bounding box regression prediction is as follows:

[0143]

[0144] where L GIoU represents the loss function value; A I represents the intersection area between the predicted slag block bounding box and the actual slag block bounding box; A U represents the union area between the predicted slag block bounding box and the actual slag block bounding box; A C represents the area of the smallest circumscribed rectangle containing the predicted slag block bounding box and the actual slag block bounding box.

[0145] Furthermore, the system also includes a data preprocessing module for:

[0146] Constructing a dynamic inter-frame difference model based on the motion speed and morphological change characteristics of slag blocks in the slag block video;

[0147] Analyzing the slag block video through the dynamic inter-frame difference model to obtain an adaptive frame sampling strategy, and sampling the slag block video according to the adaptive frame sampling strategy to generate target slag block video frames;

[0148] Segmenting the target slag block video frames through an adaptive threshold segmentation algorithm based on gray scale and texture features to obtain slag block region images;

[0149] Performing image enhancement and denoising processing on the slag block region images to obtain slag block picture frames.

[0150] Furthermore, the system also includes a regulation module for:

[0151] When the detection result is that there are large slag blocks, the detection result is abnormal;

[0152] When the detection result is abnormal, generating an alarm message and performing regulation through the DCS system in combination with the device operation status data;

[0153] where the device operation status data includes the opening and closing status of the shut-off door, the shut-off door extrusion parameter, and the conveyor belt speed, and the alarm message includes the alarm name, alarm time, event level, and location area.

[0154] In an exemplary embodiment, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the dry slag machine slag block video recognition and detection method based on deep learning are implemented. A multi-core processor is preferred to improve the parallel processing ability of the system. Memory: Provide sufficient temporary storage space to support the operation of the program and the processing of data. The memory capacity should be large enough to accommodate a large amount of supply information and computing tasks.

[0155] In an exemplary embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the dry slag machine slag block video recognition and detection method based on deep learning are implemented. The computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), solid state drive (SSD, Solid State Drives), or optical disc, etc. Among them, the random access memory may include resistive random access memory (ReRAM, Resistance RandomAccess Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).

[0156] The above-described embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the application embodiments. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A video recognition and detection method for slag blocks of a dry slag machine based on deep learning, characterized in that: The method comprises: Performing data preprocessing on the slag block video to obtain a slag block picture frame; Based on the deep learning algorithm, the slag block image frame is subjected to slag block detection by a slag block detection model to obtain a detection result, wherein the detection result includes at least one of the following items: whether there is a large slag block, the size of the large slag block, the number of large slag blocks, and the position of the large slag block. The slag block detection model includes a feature extraction module, a feature fusion module, and a target detection module; The feature extraction module is used to input the slag block picture frame into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features, and the backbone network includes a ResNet part and an EfficientNet part; The feature fusion module is used to input the low-level features and the high-level features into an adaptive bidirectional feature fusion network based on multi-scale geometric and semantic features of slag blocks to obtain fused features; The target detection module is used to input the fusion feature into the target detection network to obtain the detection result.

2. The method according to claim 1, characterized in that The feature extraction module comprises: Low-level feature extraction subunit, used to: Inputting the slag block image frame into the initial convolutional layer of the ResNet part to obtain an initial feature map; Inputting the initial feature map into the residual unit of the ResNet part to perform feature extraction to obtain the low-level features; High-level feature extraction subunit for: Input the low-level features into the convolution layer of the EfficientNet part to extract features through depth-wise separable convolution to obtain intermediate features; The intermediate features are passed through the extended residual unit of the EfficientNet part to obtain the high-level features.

3. The method according to claim 2, characterized in that The feature fusion module comprises: A scale decomposition subunit, used for the adaptive bidirectional feature fusion network to perform scale decomposition on the low-level features and the high-level features based on the multi-scale geometric features of the slag block using a multi-scale convolution kernel group, and generate a plurality of low-level sub-feature maps and high-level sub-feature maps corresponding to different size ranges; Adaptive bidirectional fusion subunit, used for: In the forward fusion process, the slag semantic feature region in the low-level sub-feature map is weighted by an attention module based on the slag semantic feature to obtain a weighted low-level sub-feature map; The weighted low-level sub-feature map is sampled and feature extracted based on the geometric features of the slag block by deformable convolution to obtain an optimized low-level sub-feature map; The optimized low-level sub-feature map is gradually fused with the high-level sub-feature map according to different scales to obtain a preliminary fused feature; In the reverse fusion process, the high-level sub-feature map is enlarged by upsampling technology to obtain an enlarged high-level sub-feature map; Combining the enlarged high-level sub-feature map with the low-level sub-feature map through a skip connection to obtain optimized comprehensive sub-feature map features; The preliminary fusion feature and the optimized high-level sub-feature map feature are fused to obtain the fusion feature.

4. The method according to claim 1, characterized in that The target detection network includes a category prediction submodule and a position prediction submodule; The category prediction submodule is used to input the fusion features into a deep neural network for classification prediction to obtain a category detection result, and the category detection result is used to characterize whether there is a large slag block in the slag block picture frame; The position prediction submodule is used for: Performing size detection on the fused features based on the anchor-free frame mechanism to obtain the size of the large slag block; The number of large slag blocks is obtained by performing slag block area recognition and threshold screening on the fusion features, and counting the number of slag block areas that meet preset conditions based on a counting mechanism; The fused features are used for position prediction through bounding box regression prediction to obtain the position of the large slag block.

5. The method according to claim 4, characterized in that The loss function calculation formula for performing position prediction by using the fusion feature through bounding box regression prediction is: Among them, L GIoU Represents the loss function value; A I A represents the intersection area of ​​the predicted slag block bounding box and the actual slag block bounding box; U A represents the union area of ​​the predicted slag block boundary box and the actual slag block boundary box; C It represents the area of ​​the minimum circumscribed rectangle containing the predicted slag block boundary box and the actual slag block boundary box.

6. The method according to claim 1, characterized in that The step of performing data preprocessing on the slag block video to obtain the slag block picture frame includes: Based on the movement speed and shape change characteristics of the slag block in the slag block video, a dynamic frame difference model is constructed; Analyzing the slag block video through the dynamic inter-frame difference model to obtain an adaptive frame sampling strategy, and sampling the slag block video according to the adaptive frame sampling strategy to generate a target slag block video frame; The target slag block video frame is segmented using an adaptive threshold segmentation algorithm based on grayscale and texture features to obtain a slag block area image; The slag block area image is subjected to image enhancement and denoising processing to obtain the slag block picture frame.

7. The method according to claim 1, characterized in that The method further comprises: When the detection result shows that there is a large slag block, the detection result is abnormal; When the detection result is abnormal, an alarm message is generated, and the DCS system is used to control the device in combination with the device operation status data; The equipment operation status data includes the opening and closing status of the shut-off door, the shut-off door squeezing parameters and the conveyor belt speed, and the alarm information includes the alarm name, alarm time, event level and location.

8. The deep learning-based dry slag machine slag block video recognition and detection system is characterized by: The system comprises: A frame acquisition module is used to perform data preprocessing on the slag block video to obtain a slag block picture frame; a slag block detection module, configured to perform slag block detection on the slag block picture frame through a slag block detection model based on a deep learning algorithm to obtain a detection result, wherein the detection result includes at least one of the following items: whether there is a large slag block, the size of the large slag block, the number of large slag blocks, and the position of the large slag block; the slag block detection model includes a feature extraction module, a feature fusion module, and a target detection module; The feature extraction module is used to input the slag block picture frame into a backbone network based on ResNet combined with EfficientNet to obtain low-level features and high-level features, and the backbone network includes a ResNet part and an EfficientNet part; The feature fusion module is used to input the low-level features and the high-level features into an adaptive bidirectional feature fusion network based on multi-scale geometric and semantic features of slag blocks to obtain fused features; The target detection module is used to input the fusion feature into the target detection network to obtain the detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images