Conveyor belt deviation detection method and device, storage medium and equipment

Abstract

The invention relates to a conveyor belt deviation detection method and device, a storage medium and equipment, and the method comprises the steps: obtaining monitoring video frame images of a conveyor belt, carrying out the feature extraction of the monitoring video frame images, obtaining the image features of at least two scales, and carrying out the detection of the deviation of the conveyor belt based on the multi-scale image features of a plurality of continuous monitoring video frame images. A multi-layer Transform encoder is used to acquire a time sequence enhancement characteristic; and carrying out difference on the multi-scale image features of the adjacent monitoring video frame images to obtain dynamic change features, inputting the time sequence enhancement features and the dynamic change features into a conveyor belt deviation detection model, and obtaining a conveyor belt deviation detection result. The deviation detection of the conveyor belt is carried out by combining the time sequence enhancement characteristics of the time sequence dependency relationship between the continuous frames and the dynamic change characteristics between the frames, so that the modeling capability of the dynamic change of the deviation is enhanced, the deviation trend of the conveyor belt can be accurately captured, and the accuracy of the deviation detection of the conveyor belt is improved.

Description

Technical Field

[0001] This invention relates to the field of conveyor belts, and in particular to a method, apparatus, storage medium, and device for detecting conveyor belt misalignment. Background Technology

[0002] Currently, conveyor belt misalignment detection mainly relies on contact sensors and traditional vision algorithms. Contact sensors detect the belt position through physical contact, but they are complex to install and maintain, have a high failure rate, and cannot provide specific information on the amount of misalignment. Traditional vision algorithms are based on edge detection or template matching techniques. In industrial settings, due to interference factors such as dust, changes in lighting, and vibration, their detection accuracy decreases significantly, resulting in high false alarm and false negative rates. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and device for detecting conveyor belt misalignment, which can improve the accuracy of conveyor belt misalignment detection.

[0004] In a first aspect, embodiments of this application provide a method for detecting conveyor belt misalignment, including:

[0005] Acquire monitoring video frame images of the conveyor belt;

[0006] Feature extraction is performed on the monitored video frame images to obtain image features at least two scales;

[0007] Based on the cross-attention mechanism, feature fusion is performed on image features at least two scales to obtain multi-scale image features;

[0008] Based on the multi-scale image features of multiple consecutive monitoring video frames, temporal enhancement features are obtained using a multi-layer Transformer encoder.

[0009] Multi-scale image features of adjacent monitoring video frames are differentially analyzed to obtain dynamic change features;

[0010] The temporal enhancement features and the dynamic change features are input into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

[0011] Secondly, embodiments of this application provide a conveyor belt misalignment detection device, the device comprising:

[0012] The video frame image acquisition module is used to acquire monitoring video frame images of the conveyor belt;

[0013] The image feature acquisition module is used to extract features from the monitored video frame image and acquire image features at least two scales.

[0014] The image feature fusion module is used to perform feature fusion on the image features of at least two scales based on the cross-attention mechanism to obtain multi-scale image features;

[0015] The temporal enhancement feature acquisition module is used to acquire temporal enhancement features based on multi-scale image features of multiple consecutive monitoring video frames using a multi-layer Transformer encoder.

[0016] The dynamic change feature acquisition module is used to perform multi-scale image feature difference on adjacent monitoring video frame images to obtain dynamic change features;

[0017] The conveyor belt deviation detection module is used to input the temporal enhancement features and the dynamic change features into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the conveyor belt deviation detection method as described in any of the above claims.

[0019] Fourthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0020] When the processor executes the computer program, it implements the steps of the conveyor belt deviation detection method as described in any of the above.

[0021] In this embodiment, monitoring video frame images of the conveyor belt are acquired, and feature extraction is performed on these images to obtain image features at least two scales. Based on the multi-scale image features of multiple consecutive monitoring video frame images, a multi-layer Transformer encoder is used to obtain temporal enhancement features. The multi-scale image features of adjacent monitoring video frame images are differentially analyzed to obtain dynamic change features. The temporal enhancement features and dynamic change features are input into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results. By combining the temporal enhancement features of the temporal dependency between consecutive frames and the dynamic change features between each frame, conveyor belt deviation detection is performed, enhancing the modeling ability of dynamic changes in deviation, accurately capturing the deviation trend of the conveyor belt, and improving the accuracy of conveyor belt deviation detection.

[0022] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0023] Figure 1 This is a flowchart of a conveyor belt misalignment detection method in one embodiment of the present invention;

[0024] Figure 2 This is a flowchart of step S110 in one embodiment of the present invention;

[0025] Figure 3 This is a flowchart of step S110 in another embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a conveyor belt misalignment detection device in one embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a computer device according to one embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0030] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0032] Furthermore, in the description of this application, unless otherwise stated, "several" refers to two or more. "And / or" describes the correspondence between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Please see Figure 1 Therefore, this application provides a method for detecting conveyor belt misalignment, including:

[0034] S110: Acquire monitoring video frame images of the conveyor belt;

[0035] Monitoring video frame images can be video frame images collected by camera devices at the conveyor belt working site.

[0036] S120: Perform feature extraction on the monitored video frame image to obtain image features at least two scales;

[0037] Optionally, pyramid feature extraction modules such as Gaussian Pyramid or Image Pyramid can be used to extract image features at different scales. Large-scale (low-resolution) image features ignore details and highlight structure, while small-scale (high-resolution) image features preserve details.

[0038] S130: Based on the cross-attention mechanism, feature fusion is performed on the image features of at least two scales to obtain multi-scale image features;

[0039] For image features at different scales, they can be upsampled or downsampled to achieve the same resolution, and then feature fusion can be performed based on a cross-attention mechanism.

[0040] In this embodiment, the image features include low-resolution (480×270×128) global deviation trend features. and high-resolution (1920×1080×128) edge detail features Global deviation trend characteristics Upsampling is performed using bilinear interpolation to achieve a resolution that matches edge detail features. The resolution (1920×1080×128) is used to obtain the upsampled feature U1. Then, U1 is used as the query vector for edge detail features. As key and value vectors, the fused multi-scale features are obtained through cross-attention computation. (1920×1080×128).

[0041] Repeat the above operation on all monitored video frames (e.g., in this embodiment, perform the above operation sequentially on frames 2 to 250) to obtain a multi-scale feature sequence.

[0042] S140: Based on the multi-scale image features of multiple consecutive monitoring video frames, temporal enhancement features are obtained using a multi-layer Transformer encoder;

[0043] The multi-layer Transformer encoder computes inter-frame feature dependencies through a multi-layer self-attention mechanism. Specifically, the multi-scale features (each frame size H×W×C) of N consecutive frames are flattened into a temporal sequence, and positional encoding is added to preserve frame order information. The self-attention mechanism dynamically aggregates the features of each frame with contextual information from other frames to capture the temporal changes in belt position. Furthermore, block attention or spatial downsampling is employed to reduce the computational overhead of high-resolution images, outputting enhanced features that fuse the temporal information of N frames for subsequent deviation prediction.

[0044] In this embodiment, the frame rate of the monitoring video is 25fps. Based on engineering experience such as the dynamic characteristics of the belt and technical constraints such as computing power and real-time performance, N can be set to 5. 5 frames covering 0.2 seconds can effectively reflect the displacement changes in the early stages of belt deviation. Too short (e.g., 3 frames) may lead to misjudgment of noise or instantaneous jitter; too long (e.g., 10 frames) will introduce redundant information and increase computational latency. Multi-scale features from 5 consecutive frames are input. The inter-frame temporal dependencies are captured by a multi-layer Transformer encoder, and temporal enhancement features are output.

[0045] In other embodiments, N can be set to other natural numbers according to the actual application, and no specific limitation is made here.

[0046] S150: Differentiate the multi-scale image features of adjacent monitoring video frames to obtain dynamic change features;

[0047] Specifically, it is assumed that there are multi-scale image features of two adjacent monitoring video frames. and Calculate the inter-frame difference as follows: Output dynamic change characteristics

[0048] In this embodiment, a sliding window can be set according to actual application requirements, and the sliding window can be used to realize the differential calculation of multi-scale image features of adjacent monitoring video frame images.

[0049] The sliding window in this embodiment can have a sliding step size of 1 frame. The dynamic change features correspond to the temporal enhancement features. In the above embodiment, the temporal enhancement features... The first dynamically changing feature is generated based on features from the first to the sixth frames. Multi-scale image features can be obtained from the fourth monitored video frame image. Multi-scale image features of the 5th monitored video frame image Start the calculation. The calculated inter-frame difference is divided into... Dynamically changing features are directly output by calculating the feature difference between adjacent frames. Based on the same principle, if the temporal enhancement feature F6 is generated based on frames 2 to 6, the corresponding dynamic change feature F6 processes frames 5 and 6; if the temporal enhancement feature F7 is generated based on frames 3 to 7, the corresponding dynamic change feature F7 processes frames 6 and 7, and so on. This process is repeated on subsequent monitoring video frames to obtain the dynamic change feature sequence from frame 6 to frame 250.

[0050] S160: Input the time-series enhancement features and the dynamic change features into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

[0051] The conveyor belt deviation detection model can be a pre-trained model for the user.

[0052] In this embodiment, monitoring video frame images of the conveyor belt are acquired, and feature extraction is performed on these images to obtain image features at least two scales. Based on the multi-scale image features of multiple consecutive monitoring video frame images, a multi-layer Transformer encoder is used to obtain temporal enhancement features. The multi-scale image features of adjacent monitoring video frame images are differentially analyzed to obtain dynamic change features. The temporal enhancement features and dynamic change features are input into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results. By combining the temporal enhancement features of the temporal dependency between consecutive frames and the dynamic change features between each frame, conveyor belt deviation detection is performed, enhancing the modeling ability of dynamic changes in deviation, accurately capturing the deviation trend of the conveyor belt, and improving the accuracy of conveyor belt deviation detection.

[0053] like Figure 2 As shown, in one embodiment, step S110 acquires monitoring video frame images of the conveyor belt, including:

[0054] S111: Acquire the monitoring video stream collected by the camera device;

[0055] S112: Use the ffmpeg tool to parse the monitoring video stream and extract multiple monitoring video frame images.

[0056] Ffmpeg is an open-source, cross-platform audio and video processing framework that covers a range of audio and video processing functions, including recording, conversion, and streaming. It supports a wide range of audio and video formats such as AVI, MP4, MOV, FLV, WMV, MPEG, and MKV, and can run stably on various operating systems such as Windows, Mac OS, and Linux.

[0057] The monitoring video stream is a video of the conveyor belt running at the work site. Assuming the resolution of the monitoring video stream is 1080P, the frame rate is 25fps, and the duration is 10 seconds (250 frames in total), the monitoring video stream contains the following content: the conveyor belt runs normally for the first 5 seconds; starting from the 6th second, the conveyor belt gradually deviates to the left, and the deviation increases from 0cm to 10cm; the deviation reaches 5cm in the 8th second; and the deviation reaches 10cm in the 10th second.

[0058] The ffmpeg tool can be pre-installed on a designated video analysis device. This device can establish a communication connection with the camera via wireless technologies such as Wi-Fi, Bluetooth, or wired connections. The video analysis device can be the computer equipment in an AI video inspection center or other devices with the aforementioned capabilities.

[0059] After the video parsing device receives the monitoring video stream collected by the camera device, it uses the ffmpeg tool to parse the monitoring video stream into monitoring video frame images, extracts each monitoring video frame image I(x) (in this embodiment, the resolution of the monitoring video frame image is 1920×1080, RGB format), and divides the video stream into a continuous image frame sequence I1, I2, ..., IN, where N is the total number of frames.

[0060] In the above embodiments, the conveyor belt can be a conveyor belt used in industrial scenarios such as non-ferrous metal ore beneficiation and coal transportation. In such industrial scenarios, the monitored images are easily affected by interference factors such as dust and changes in lighting, resulting in poor detection accuracy.

[0061] Therefore, as Figure 3 As shown, in one embodiment, after acquiring the monitoring video frame image of the conveyor belt, the method further includes:

[0062] S113: Based on the Dehaze-Net algorithm, perform dust removal processing on the monitored video frame images;

[0063] The Dehaze-Net algorithm is a deep learning-based image dehazing algorithm. In this embodiment, the monitoring video frame image is specifically processed to remove dust interference and obtain a clear image after dust removal, in the following manner:

[0064]

[0065] Where J1(x) is the monitoring video frame image after removing dust interference, I1(x) is the xth monitoring video frame image, A is the atmospheric light value, and t(x) is the transmittance map.

[0066] S114: Based on the Retinex algorithm, perform illumination normalization processing on the monitored video frame image.

[0067] The Retinex algorithm treats an image I(x,y) as the product of scene "illumination" or "brightness" (L) and scene "reflectance" (R). By transforming the multiplication into addition through logarithmic transformation, and then estimating and removing the scene "illumination" or "brightness" (L), the enhanced scene "reflectance" (R) can be obtained, achieving effects such as dehazing, backlight removal, enhancement of dark details, and compression of dynamic range.

[0068] Specifically, the monitoring video frame images are subjected to illumination normalization processing in the following manner to remove the influence of illumination changes in the monitoring video frame images:

[0069] R1(x)=log(J1(x))-log(G(x)*J1(x))

[0070] Where R1(x) is the monitoring video frame image obtained after removing the influence of illumination changes, G(x) is the Gaussian kernel function, and * indicates the convolution operation.

[0071] Following the steps described above, all the monitored video frames (250 frames in this embodiment) are processed to remove dust interference and illumination variation effects, resulting in a preprocessed image sequence R1, R2, ..., R 250 .

[0072] In this embodiment, the Dehaze-Net algorithm is used to remove dust interference from the monitoring video frame image, and the Retinex algorithm is used to remove the influence of illumination changes from the monitoring video frame image, thereby improving the accuracy of conveyor belt deviation detection and its robustness in complex environments such as dust and illumination changes.

[0073] In one embodiment, feature extraction of the monitored video frame image includes:

[0074] The monitored video frame image is input into a pre-trained feature extraction module. The first feature extraction unit of the feature extraction module extracts the first image features, and the second feature extraction unit of the feature extraction module extracts the second image features. The first feature extraction unit includes an input layer, a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, and an output layer connected in sequence. The second feature extraction unit includes an input layer, multiple dilated convolutional layers with different dilation rates, and an output layer. The resolution of the first image features is smaller than the resolution of the second image features.

[0075] Specifically, the feature extraction module can be a pyramid feature extraction module, which includes a first feature extraction unit and a second feature extraction unit. The first feature extraction unit is used to extract low-resolution first image features, and the second feature extraction unit is used to extract high-resolution second image features.

[0076] The first feature extraction unit and the second feature extraction unit share the input data and extract features at different scales through different operations.

[0077] In this embodiment of the application, the first image feature is a global feature, and the second image feature is a local feature.

[0078] In this embodiment, the resolution of the monitored video frame image is 1920×1080, and the first image feature is the global deviation trend feature. The first feature extraction unit includes a first convolutional layer for performing 3×3 convolution calculations, a first max pooling layer for performing 2×2 max pooling, a second convolutional layer for performing 3×3 convolution calculations, and a second max pooling layer for performing 2×2 max pooling.

[0079] The preprocessed monitoring video frame image R(x)∈R 1920×1080×3 The input is fed into the input layer, where it undergoes a 3×3 convolution in the first convolutional layer to obtain feature map C1 (1920×1080×64). It then undergoes a 2×2 max pooling in the first max pooling layer to obtain feature map P1 (960×540×64). Next, it undergoes a 3×3 convolution in the second convolutional layer to obtain feature map C2 (960×540×128). Finally, it undergoes a 2×2 max pooling in the second max pooling layer to reduce the resolution to 480×270×128. The output layer then outputs the global deviation trend feature.

[0080] The first convolutional layer performs a 3×3 convolution operation as follows:

[0081] F1=ReLU(W1*R(x)+b1)∈R 1920×1080×64

[0082] Wherein, the convolution kernel W1∈R of the first convolutional layer 3×3×3×64 Bias b1∈R 64 ReLU(·) is the activation function.

[0083] The first max pooling layer performs 2×2 max pooling as follows:

[0084] F2 = MaxPool 2×2 (F1)∈R 960×540×64

[0085] Among them, MaxPool 2×2 (·) is the 2×2 max pooling function.

[0086] The second convolutional layer performs a 3×3 convolution operation as follows:

[0087] F3=ReLU(W2*F2+b2)∈R 960×540×128

[0088] Wherein, the convolution kernel W2∈R 3×3×64×128 Bias b2∈R 128 .

[0089] The second max pooling layer performs 2×2 max pooling as follows:

[0090] F global =MaxPool 2×2 (F3)∈R 480×270×128

[0091] The second image feature is the local edge detail feature. The second feature extraction unit includes a first dilated convolutional layer with a dilation rate of 1 and a 3×3, a second dilated convolutional layer with a dilation rate of 2 and a 3×3, and a third dilated convolutional layer with a dilation rate of 4 and a 3×3. The activation function for the first, second, and third dilated convolutional layers is the ReLU function.

[0092] The preprocessed monitoring video frame image R(x)∈R 1920×1080×3 The input layer passes through a first dilated convolutional layer with a dilation rate of 1 and a 3×3 to extract local detail features, resulting in feature map D1 (1920×1080×64). Feature map D1 is then fed into a second dilated convolutional layer with a dilation rate of 2 and a 3×3 to further extract multi-scale local information, resulting in feature map D2 (1920×1080×128). Finally, feature map D2 is fed into a third dilated convolutional layer with a dilation rate of 4 and a 3×3 to capture a wider range of local information, resulting in local edge detail features. (1920×1080×128).

[0093] The first dilated convolutional layer is calculated using a 3×3 dilated convolution with a dilation rate of 1 in the following manner:

[0094] F d1 =ReLU(DilatedConv(R(x),W d1 ,r=1)+b d1 )∈R 1920×1080×64

[0095] Where DilatedConv(·) is the dilated convolution function, W d1 ∈R 3×3×3×64 For hollow convolution kernels, b d1 ∈R 64 This is the bias term, and r = 1 represents the void ratio.

[0096] The second dilated convolutional layer is calculated using a 3×3 dilated convolution with a dilation rate of 2 as follows:

[0097] F d2 =ReLU(DilatedConv(F d1 W d2 ,r=2)+b d2 )∈R 1920×1080×128

[0098] Among them, W d2 ∈R 3×3×64×128 For hollow convolution kernels, b d2 ∈R 128 This is the bias term, and r = 2 represents the void ratio.

[0099] The third dilated convolutional layer is calculated as follows: a 3×3 dilated convolution with a dilation rate of 4 is performed:

[0100] F local =ReLU(DilatedConv(F d2 W d4 (r = 4) + b d4 )∈R 1920×1080×128

[0101] Among them, W d4 ∈R 3×3×128×128 For hollow convolution kernels, b d4 ∈R 128 For the bias term, r = 4 represents the void ratio.

[0102] In one embodiment, feature fusion is performed on the image features at at least two scales based on a cross-attention mechanism, including:

[0103] Feature fusion is performed on the image features at at least two scales in the following manner:

[0104]

[0105] Where Q is the first image feature F at the first resolution. global K and V are the second image features F at the second resolution. local K and V are the same, d k For feature dimensions.

[0106] Preferably, before feature fusion, the first image feature F can be... global Upsampled to the second image feature F local The same resolution (in this embodiment, the first image feature F) global Upsampled to 1920×1080×128). To reduce computational load, image features can be further segmented to improve processing efficiency.

[0107] In one embodiment, the conveyor belt deviation detection model includes an input layer, a global average pooling layer, a fully connected layer, and an output layer. The model inputs the temporal enhancement features and the dynamic change features, including:

[0108] The temporal enhancement features and the dynamic change features are input into the input layer, and the temporal enhancement features and the dynamic change features are concatenated in the following manner:

[0109]

[0110] Where t is the t-th monitoring video frame image, For splicing features, For temporal enhancement features, For dynamically changing features, Concat is the concatenation function; concatenation features The dimensions are (H, W, C); splicing features in the embodiments of this application. The specific dimensions are 1920×1080×256, H=1920, W=1080, C=256.

[0111] The concatenated features are input into the global average pooling layer, and global average pooling and compression are performed on the concatenated features in the following manner:

[0112]

[0113] in, For pooling features, (H,W) represents the spatial dimension of the splicing features. k = 1, 2, ..., C can be determined based on the resolution of the splicing features. In this embodiment, k = 1, 2, ..., 256. Specifically, the pooling features are compressed to 1×1.

[0114] The pooled features are output to the fully connected layer to obtain a preliminary prediction of the deviation:

[0115]

[0116] Where z is the initial prediction value, W is the weight matrix of the fully connected layer (256×2), and b is the bias vector (2×1); the initial prediction value z includes the deviation score z1 and the non-deviation score z2.

[0117] The preliminary predicted values ​​are output to the output layer, and the probability distribution of the deviation is obtained based on the Softmax function:

[0118]

[0119] Where p(deviation) is the probability distribution of deviation, and p(non-deviation) is the probability distribution of non-deviation.

[0120] Obtain the predicted deviation value as follows:

[0121] Δ t = p(deviation)·Scale

[0122] Where Scale is a scaling factor used to map the probability value to the actual deviation range. In this embodiment, Scale is from 0cm to 10cm, and Δ t This is the predicted value for deviation.

[0123] In one embodiment, the method further includes:

[0124] The number of model channels in the conveyor belt deviation detection model is dynamically reduced based on the operating conditions.

[0125] Specifically, the number of model channels in the conveyor belt misalignment detection model is dynamically adjusted according to the working conditions to reduce the computational load of the conveyor belt misalignment detection model and improve the inference speed.

[0126] A teacher model is constructed based on the multi-scale fusion network and the multi-layer Transformer encoder, and a student model is constructed based on the MobileNetV3 model and the GRU model. The teacher model is then subjected to a distillation operation.

[0127] The teacher model learns a lightweight MobileNetV3+GRU student model through a distillation operation, further compressing the model size and reducing model complexity through lightweight design, enabling it to run efficiently on resource-constrained edge devices. At the same time, this application can achieve low-latency inference, fully meeting the real-time requirements of industrial scenarios and overcoming the shortcomings of poor real-time performance in existing technologies.

[0128] The conveyor belt deviation detection method of this application acquires multi-scale image features and performs multi-scale feature fusion, while also performing dust removal and illumination normalization processing on the monitoring video frame images. This enables higher detection accuracy in complex environments such as dust and varying lighting conditions, effectively solving the problem of low detection accuracy in existing technologies. Secondly, by introducing a Transformer temporal encoder and a temporal difference module, the method enhances the modeling capability for dynamic changes in deviation, accurately capturing deviation trends and compensating for the shortcomings of insufficient temporal modeling capabilities in existing technologies. Finally, thanks to its lightweight design, the model complexity is significantly reduced, allowing it to run efficiently on resource-constrained edge devices. Furthermore, this application achieves low-latency inference, fully meeting the real-time requirements of industrial scenarios and overcoming the poor real-time performance of existing technologies. In summary, this application achieves comprehensive optimization in detection accuracy, temporal modeling, lightweight model, and low-latency inference, providing reliable technical support for safe industrial production.

[0129] like Figure 4 As shown in the figure, this application embodiment also provides a conveyor belt misalignment detection device, the device comprising:

[0130] The video frame image acquisition module 110 is used to acquire monitoring video frame images of the conveyor belt;

[0131] The image feature acquisition module 120 is used to extract features from the monitoring video frame image and acquire image features at least two scales.

[0132] The image feature fusion module 130 is used to perform feature fusion on the image features of at least two scales based on the cross-attention mechanism to obtain multi-scale image features;

[0133] The temporal enhancement feature acquisition module 140 is used to acquire temporal enhancement features based on multi-scale image features of multiple consecutive monitoring video frame images using a multi-layer Transformer encoder.

[0134] The dynamic change feature acquisition module 150 is used to perform differential analysis on the multi-scale image features of adjacent monitoring video frame images to acquire dynamic change features.

[0135] The conveyor belt deviation detection module 160 is used to input the time-series enhancement features and the dynamic change features into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

[0136] It should be noted that the conveyor belt misalignment detection device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the conveyor belt misalignment detection method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the equipment can be divided into different functional modules to complete all or part of the functions described above. In addition, the conveyor belt misalignment detection device provided in the above embodiments and the conveyor belt misalignment detection method in the above embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0137] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the conveyor belt misalignment detection method as described in any of the above claims.

[0138] The embodiments of this application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0139] like Figure 5 As shown, this application embodiment also provides a computer device 200, including a memory 210, a processor 220, and a computer program stored in the memory 210 and executable by the processor 220;

[0140] When the processor 220 executes the computer program, it implements the steps of the conveyor belt deviation detection method as described in any of the above.

[0141] The memory 210 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0142] The processor 220 is the control unit of the computer device 200. It connects to various components of the computer device 200 via various interfaces and lines. By running or executing programs or modules stored in the memory 210, and by calling data stored in the memory 210, it performs various functions of the computer device 200 and processes data. For example, when the processor 220 executes the computer program stored in the memory 210, it implements all or part of the steps of the conveyor belt misalignment detection method described in this application embodiment; or it implements all or part of the functions of the conveyor belt misalignment detection device. The processor 220 can be composed of integrated circuits, such as a single packaged integrated circuit, or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0144] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting conveyor belt misalignment, characterized in that, include: Acquire monitoring video frame images of the conveyor belt; Feature extraction is performed on the monitored video frame images to obtain image features at least two scales; Based on the cross-attention mechanism, feature fusion is performed on image features at least two scales to obtain multi-scale image features; Based on the multi-scale image features of multiple consecutive monitoring video frames, temporal enhancement features are obtained using a multi-layer Transformer encoder. Multi-scale image features of adjacent monitoring video frames are differentially analyzed to obtain dynamic change features; The temporal enhancement features and the dynamic change features are input into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

2. The conveyor belt misalignment detection method according to claim 1, characterized in that, Feature extraction of the monitored video frame images includes: The monitored video frame image is input into a pre-trained feature extraction module. The first feature extraction unit of the feature extraction module extracts the first image features, and the second feature extraction unit of the feature extraction module extracts the second image features. The first feature extraction unit includes an input layer, a first convolutional layer, a first max-pooling layer, a second convolutional layer, a second max-pooling layer, and an output layer. The second feature extraction unit includes an input layer, multiple dilated convolutional layers with different dilation rates, and an output layer. The resolution of the first image features is smaller than the resolution of the second image features.

3. The conveyor belt misalignment detection method according to claim 1, characterized in that, Based on the cross-attention mechanism, feature fusion is performed on the image features at least two scales, including: Feature fusion is performed on the image features at at least two scales in the following manner: Where Attention(Q,K,V) represents multi-scale image features, Q is the first image feature at the first resolution, K and V are the second image features at the second resolution, and d k For feature dimensions.

4. The conveyor belt misalignment detection method according to claim 1, characterized in that, The conveyor belt deviation detection model includes an input layer, a global average pooling layer, a fully connected layer, and an output layer. The temporal enhancement features and the dynamic change features are input into the conveyor belt deviation detection model, including: The temporal enhancement features and the dynamic change features are input into the input layer, and the temporal enhancement features and the dynamic change features are concatenated in the following manner: Where t is the t-th monitoring video frame image, For splicing features, For temporal enhancement features, For dynamically changing features, Concat is a concatenation function that concatenates features. The dimensions are (H, W, C); The concatenated features are input into the global average pooling layer, and global average pooling and compression are performed on the concatenated features in the following manner: in, The pooling feature is (H,W), where (H,W) is the spatial dimension of the concatenated feature, and k = 1, 2, ..., C; The pooled features are output to the fully connected layer to obtain a preliminary prediction of the deviation: Where z is the initial predicted value, W is the weight matrix of the fully connected layer, b is the bias vector, and the initial predicted value z includes the deviation score z1 and the non-deviation score z2. The preliminary predicted values ​​are output to the output layer, and the probability distribution of the deviation is obtained based on the Softmax function: Where p(deviation) is the probability distribution of the deviation; Obtain the predicted deviation value as follows: Δ t = p(deviation)·Scale Where Scale is the scaling factor, Δ t This is the predicted value for deviation.

5. The conveyor belt misalignment detection method according to claim 1, characterized in that, Acquire monitoring video frame images of the conveyor belt, including: Acquire the monitoring video stream captured by the camera device; The monitoring video stream was parsed using the ffmpeg tool to extract multiple monitoring video frame images.

6. The conveyor belt misalignment detection method according to claim 1, characterized in that, After acquiring the monitoring video frame images of the conveyor belt, the following is also included: The monitoring video frame images are processed to remove dust based on the Dehaze-Net algorithm; The monitoring video frame image is subjected to illumination normalization processing based on the Retinex algorithm.

7. The conveyor belt misalignment detection method according to claim 1, characterized in that, The method further includes fusing image features at at least two scales using a multi-scale feature fusion network. The number of model channels in the conveyor belt deviation detection model is dynamically reduced based on the operating conditions. A teacher model is constructed based on the multi-scale fusion network and the multi-layer Transformer encoder, and a student model is constructed based on the MobileNetV3 model and the GRU model. The teacher model is then subjected to a distillation operation.

8. A conveyor belt misalignment detection device, characterized in that, The device includes: The video frame image acquisition module is used to acquire monitoring video frame images of the conveyor belt; The image feature acquisition module is used to extract features from the monitored video frame image and acquire image features at least two scales. The image feature fusion module is used to perform feature fusion on the image features of at least two scales based on the cross-attention mechanism to obtain multi-scale image features; The temporal enhancement feature acquisition module is used to acquire temporal enhancement features based on multi-scale image features of multiple consecutive monitoring video frames using a multi-layer Transformer encoder. The dynamic change feature acquisition module is used to perform multi-scale image feature difference on adjacent monitoring video frame images to obtain dynamic change features; The conveyor belt deviation detection module is used to input the temporal enhancement features and the dynamic change features into the conveyor belt deviation detection model to obtain the conveyor belt deviation detection results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the conveyor belt deviation detection method as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, it implements the steps of the conveyor belt deviation detection method as described in any one of claims 1-7.

Images