Belt crack detection model construction method and detection method based on three-domain feature learning

Abstract

The application discloses a kind of based on three-domain feature learning's belt crack detection model construction method and detection method, construction method includes: the collection of original image sequence of belt crack, and the artificial marking of crack is carried out to obtain corresponding box label image;Based on the set of original image sequence, the set of feature image sequence of belt crack is constructed;Respectively through time, space, frequency domain three-domain feature learning algorithm respectively processes primary feature, obtains time feature, spatial feature and frequency domain feature;Space feature, time sequence feature and frequency domain feature are deeply fused, and uniform feature representation is generated F fst ;Detection head is positioned to uniform feature representation by defining loss function F fst According to the error of positioning result determined according to corresponding box label image, the network parameters of belt crack detection model are iteratively optimized by error back propagation mode, and the detection accuracy and robustness of crack detection model are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of AI visual recognition, and in particular to a belt crack detection model construction method and detection method based on three-domain feature learning. BACKGROUND

[0002] In the modern industrial production system, the belt conveyor is an indispensable core equipment in the industries of mine exploitation, port logistics, thermal power generation and building material processing, etc. due to its advantages of high efficiency, continuous and large capacity of material transmission. For example, in coal mine exploitation, large-scale belt conveyors can realize long-distance transportation of thousands of tons of coal per hour; in the port, the belt conveying system undertakes a large amount of loading and unloading of bulk goods such as ore and grain. However, due to long-term operation under high-intensity and harsh conditions, cracks are easily generated on the surface of the belt, which seriously threatens the safety and continuity of industrial production.

[0003] The generation of belt cracks is caused by multiple complex factors. From the perspective of material properties, the gangue in the mine and the gravel in the building material plant often have sharp edges, which continuously scratch and press the surface of the belt during the belt bearing and running process, causing local damage to the rubber material and further forming cracks; if metal impurities are mixed in the coal transported by the power plant, it will also accelerate the wear of the belt. In terms of mechanical operation, there is continuous friction and tension between the belt and the driving drum and the roller, and long-term high-load operation will cause the surface temperature of the belt to rise, accelerating the aging of the rubber material; the joint of the belt is more prone to stress concentration under alternating stress due to the difference in material and structure, which can cause crack initiation and propagation. In addition, environmental factors cannot be ignored, a humid environment can cause the rubber material of the belt to hydrolyze, reducing its physical strength; a high-temperature environment accelerates the oxidation of rubber, making it brittle and hard, losing its elasticity, which all create conditions for the generation of cracks.

[0004] If the belt cracks are not discovered and treated in time, it will cause serious consequences, and therefore, efficient and accurate detection of belt cracks has become a key technical problem to be solved in the industrial field.

[0005] Currently, the belt crack detection technology mainly covers traditional detection methods and computer vision-based detection methods. In the traditional detection method, the mechanical detection method usually reflects the belt crack condition by means of the physical deformation of the mechanical structure, for example, a contact detection device is arranged below the belt, and an alarm is triggered when the belt cracks and sags, but this method is easily disturbed by the normal fluctuation of the belt, and the false alarm rate is high; the embedded detection method monitors the stress change of the belt by embedding sensors such as strain gauges and optical fiber sensors in the belt, but the sensor installation process is complex, and signal attenuation problems are prone to occur after long-term operation; the acoustic wave detection method detects the vibration sound wave characteristics generated when the crack occurs, but the complex sound wave interference such as mechanical noise and material impact sound in the industrial field makes it difficult to ensure the detection accuracy of the method; the electromagnetic induction detection method relies on the change of electromagnetic signals generated by the internal metal components of the belt at the crack to determine the crack state, but its detection effect on non-metallic material belts or small cracks is not good.

[0006] The computer vision-based detection method has developed rapidly in recent years, and has gradually become a research hotspot due to its non-contact and strong real-time advantages. This kind of method is divided into model-driven and data-driven types. The model-driven method mainly relies on manually designed algorithms to extract crack features, and common ones include edge detection algorithms and threshold segmentation algorithms. These algorithms identify cracks by analyzing the gray level and texture information of the belt image, but in complex industrial environments, factors such as uneven lighting, dust pollution, and belt surface stains can severely interfere with feature extraction, resulting in poor algorithm robustness and low detection accuracy. At the same time, in order to ensure detection accuracy, a large amount of image preprocessing and complex calculations are often required, and the processing efficiency is difficult to meet the real-time detection demand.

[0007] In the data-driven method, the detection model based on deep learning uses the powerful feature learning ability of neural networks to overcome some of the shortcomings of traditional models to some extent. However, existing deep learning-based methods mostly train and learn models based on single-frame images, only focusing on spatial features of images, and failing to fully exploit the dynamic evolution rules of belt cracks in time series and the implicit information in frequency domain. In addition, due to the lack of large-scale, high-quality, and finely labeled special data sets for belt crack detection, model training is not sufficient, and the generalization ability is limited; and the belt crack detection is simply modeled as an object detection task, which is difficult to accurately describe the subtle morphology, trend and distribution characteristics of the crack, and cannot provide comprehensive and accurate crack information for enterprises, which is not conducive to the formulation of subsequent maintenance decisions.

[0008] The disclosure of the above background art content is only used to assist in understanding the concept and technical solutions of the present application, and it does not necessarily belong to the prior art of the present application, nor does it necessarily provide technical teaching; in the absence of explicit evidence that the above content was disclosed before the filing date of the present application, the above background art should not be used to evaluate the novelty and inventiveness of the present application. SUMMARY

[0009] The application aims to provide a belt crack detection model construction method, which is based on spatial, temporal and frequency domain feature learning, and significantly improves the learning ability and detection accuracy of the model for belt crack features.

[0010] To achieve the above-mentioned purpose, the technical scheme adopted by the application is as follows:

[0011] A belt crack detection model construction method based on three-domain feature learning, comprising the following steps:

[0012] Collecting a learning sample set: collecting a set of original image sequences of belt cracks, and performing artificial labeling of the cracks to obtain corresponding box label images;

[0013] Based on the set of original image sequences, a set of feature image sequences of belt cracks is constructed, and each feature image sequence in the set includes a feature image corresponding to a plurality of consecutive original images;

[0014] A hierarchical spatial perception module is constructed, which is configured to extract spatial features of belt cracks in the spatial dimension from a feature image sequence, denoted as F S ;

[0015] A time aggregation module is constructed, which is configured to mine the time correlation information of the context of the feature image sequence to capture the dynamic evolution law of the belt crack in the time dimension, extract the time sequence features of the belt crack, denoted as F T ;

[0016] A wavelet enhanced frequency domain perception module is constructed, which is configured to map the feature image sequence into the wavelet frequency domain for feature decoupling and enhancement, and obtain the frequency domain features of the belt crack, denoted as F F ;

[0017] A cross-domain feature compensation module is constructed, which is configured to dynamically align and fuse the spatial features F S , the time sequence features F T and the frequency domain features F F , bridge the differences between multi-domain features through residual compensation mechanism, generate unified feature representation, denoted as F fst ;

[0018] A detection head is constructed, which is configured to define a loss function to locate the unified feature representation F fst , and determine the error of the positioning result according to the corresponding box label image, and iteratively optimize the network parameters of the belt crack detection model through error back propagation.

[0019] Further, any one of the above technical solutions or a combination of the above technical solutions is used to determine a key frame in each feature image sequence according to a preset rule, and the remaining frame images are determined as reference frames;

[0020] The hierarchical space perception module is configured with a short-distance space feature branch and a long-distance space feature branch, wherein the short-distance space feature branch adopts a multi-scale feature fusion network structure to obtain local region space features of multiple reference frames through different scales of receptive fields;

[0021] The long-distance space feature branch uses an attention mechanism to capture space features of long-distance dependencies in the key frame;

[0022] In combination with the local region space features of the reference frames and the space features of the key frame, the space features F of the belt crack in the spatial dimension are determined S .

[0023] Further, any one of the above technical solutions or a combination of the above technical solutions is used, the short-distance space feature branch is configured with a local relationship extraction module and a non-local attention module, the long-distance space feature branch is configured with a memory enhancement module, and the space features F are determined in the following manner S :

[0024] The local relationship extraction module extracts features of the reference frames, and splices and fuses the features with the key frame to obtain inter-frame local space relationships F l ;

[0025] The non-local attention module focuses on long-distance dependencies of the inter-frame local space relationships F l of the reference frames, mines correlations between adjacent frames at different pixel point pairs, and is used to realize long-distance dependency modeling to obtain local global space relationships F g of the reference frames;

[0026] The memory enhancement module simulates a persistence attention ability of a human eye visual tracking system, stores the local global space relationships F g of the reference frames by using the key frame for query, and realizes continuous updating of a spatial context to obtain the space features F S of the belt crack in the spatial dimension.

[0027] Further, any one of the above technical solutions or a combination of the above technical solutions is used to obtain the inter-frame local space relationships F l in the following manner:

[0028]

[0029] Among them, [F1,…,F t ] represents the feature images of frame t in the feature image sequence, where t≥3, where [F1,…,F t-1 ] indicates the reference frame, F t This represents a keyframe, Concat() represents the concatenation function, and Conv() represents the convolution function. This represents element-wise multiplication, where σ represents the activation function. This represents the updated feature map after the reference frame and key frame have been stitched and fused together.

[0030] Furthermore, following any one or a combination of the aforementioned technical solutions, the local-global spatial relationship F of the reference frame is obtained in the following manner. g :

[0031]

[0032] Where Q represents from F l The resulting query matrix is ​​used to calculate the correlation between the current location and other locations; K represents the correlation between F and other locations. l The resulting key matrix is ​​used as an index for queries and for matching relevance; V represents the key matrix from F. l The resulting value matrix is ​​used to aggregate relevant feature information through semantic content; W Q W represents the weight matrix of the query matrix Q. K W represents the weight matrix of the key matrix K. V Let V be the weight matrix of the value matrix V, MatMul() represents matrix multiplication, and Softmax() represents normalization of the correlation scores. This represents the scaling factor used for standardization adjustments.

[0033] Furthermore, based on any one or a combination of the aforementioned technical solutions, the feature query mechanism and feature storage mechanism of the memory enhancement module are implemented in the following ways:

[0034]

[0035] in, This represents the convolution function of the key values ​​under the query mechanism. This represents the convolution function for semantic content under the query mechanism. This represents the convolution function for key values ​​under the storage mechanism. A convolution function representing semantic content under a storage mechanism;

[0036] The keyframe F t The local-global spatial relationship F with the reference frame gInput the memory enhancement module together, get the spatial characteristics F of the belt crack in the spatial dimension S .

[0037] Further, any of the preceding technical solutions or combinations thereof, set the feature image sequence as [F1,…,F t ] represents the t-frame feature image, the time aggregation module is configured with affinity calculation module and difference calculation module, wherein:

[0038] The affinity calculation module calculates the information C(i,j) between any two different frames in the feature image sequence by the following formula: C(i,j) = F i ·F j , wherein F i represents the i-th feature image in the feature image sequence, and F j represents the j-th feature image in the feature image sequence.

[0039] The difference calculation module calculates the pixel-level difference feature map D between the selected frame image and its adjacent frame image by the following formula:

[0040]

[0041] Wherein, D1 represents the t-frame feature image F t in the Euclidean space, C (1,1) represents the affinity weight of the t-frame feature image F t with itself; D j represents the (t-j+1)-frame feature image F t-j+1 in the Euclidean space, d( ) represents difference operation, C (1, j) represents the affinity weight between the t-frame feature image F t and the (t-j+1)-frame feature image F t-j+1 .

[0042] Further, any of the preceding technical solutions or combinations thereof, the time sequence characteristics F T of the belt crack are obtained by the following method:

[0043] Define the fusion feature of the time dimension :

[0044]

[0045] Wherein, t represents the number of image frames in the feature image sequence, represents element-level multiplication, F jF1 represents a difference feature map between the jth feature map and the 1st feature map in the feature image sequence, F1 represents a difference feature map between the jth feature map and the 1st feature map in the feature image sequence,

[0046] The fusion feature in the time dimension is obtained by fusing the feature images in the time dimension, After mutual splicing and fusion of a plurality of sequence frames, the time sequence feature F of the belt crack is obtained T .

[0047] Further, any one of the technical solutions or a combination of the technical solutions, the wavelet enhanced frequency domain perception module is configured with a frequency domain decoupling module, a frequency domain reconstruction module and a frequency domain feature fusion module, wherein:

[0048] The frequency domain decoupling module is configured to perform frequency domain decoupling on each feature image in the feature image sequence, decompose it into a low-frequency component and a high-frequency component, and recursively decouple the low-frequency component to obtain the lowest layer of low-frequency feature information;

[0049] The frequency domain reconstruction module restores the features upward by recursive step-by-step manner to obtain the features with the same feature dimension as the original features before the frequency domain decoupling by the frequency domain decoupling module;

[0050] The frequency domain feature fusion module fuses the feature set after the frequency domain decoupling by the frequency domain decoupling module and the reconstruction by the frequency domain reconstruction module to obtain high-level frequency domain fusion features Z (0) .

[0051] Further, any one of the technical solutions or a combination of the technical solutions, the frequency domain decoupling module is a cascaded wavelet decomposition feature WT(X):

[0052]

[0053] wherein, represents the current decomposition level of multi-level decoupling, represents the low-frequency component from the frequency domain component of the upper level, represents the horizontal high-frequency component from the frequency domain component of the upper level, represents the vertical high-frequency component from the frequency domain component of the upper level, represents the diagonal high-frequency component from the frequency domain component of the upper level;

[0054] The frequency domain reconstruction module processes different sub-band frequency domain components in the cascaded wavelet decomposition feature WT(X) through a small convolution kernel, and then obtains an output feature Y through inverse wavelet transform: Y = IWT(Conv(W, WT(X))), wherein IWT() is inverse wavelet transform, X represents an input feature of a current level, W represents a deep convolution kernel with a size of k x k, and each sub-band frequency domain of the current level obtained through WT() is subjected to convolution calculation with W;

[0055] The formula for convolution calculation of the output feature Y in the jth level is as follows:

[0056]

[0057] wherein W (j) represents a weight kernel in the jth level, represents a high-frequency component in the jth level , and , Conv() represents convolution operation, represents a low-frequency component of the output feature Y in the jth level, represents a horizontal high-frequency component of the output feature Y in the jth level, represents a vertical high-frequency component of the output feature Y in the jth level, represents a diagonal high-frequency component of the output feature Y in the jth level.

[0058] Further, in any of the technical solutions or the combination of multiple technical solutions described above, the frequency domain feature fusion module restores the output feature Y through inverse wavelet transform to obtain a frequency domain fusion feature Z (j) by the following method:

[0059]

[0060] wherein Z (j+1) = 0 represents an initial reconstruction frequency domain feature of the bottom level, Z (0) represents a high-level frequency domain fusion feature obtained through recursive reconstruction, represents that the (j+1)th level frequency domain fusion feature Z is subjected to bottom-up superposition fusion with .

[0061] The high-level frequency domain fusion feature Z (0) is subjected to splicing fusion of multiple feature images in a feature image sequence to obtain the frequency domain feature F F of the belt crack.

[0062] Further, in any of the technical solutions or the combination of multiple technical solutions described above, the cross-domain feature compensation module is configured to:

[0063] The spatial feature F S , the time sequence feature F T is convoluted and fused to obtain a time-space fused feature F st : F st = Conv(Concat[F S , F T ]), wherein Concat() represents a concatenation function, and Conv() represents a convolution function.

[0064] The local frequency domain time feature F T and the global frequency domain time feature F F are calculated based on the time sequence feature F t and the frequency domain feature F st :

[0065]

[0066] wherein WFM() represents a wavelet enhanced frequency domain perception algorithm, F ft represents a set of t frame feature images in the feature image sequence, and Swin() represents a Swin Transformer attention mechanism.

[0067] The local frequency domain time feature F ft and the time-space fused feature F fst are cross-domain feature fused to obtain a local frequency domain space-time fused feature F ft ; the global frequency domain time feature F fst and the local frequency domain time feature F reg are cross-domain feature fused to obtain a local-global fused frequency domain time feature F cls ; and the local frequency domain space-time fused feature F obj and the local-global fused frequency domain time feature F p are cross-domain feature fused to obtain a unified feature representation F ture .

[0068] Further, any one of the technical solutions or a combination of the technical solutions described above, the local frequency domain space-time fused feature F is fused by the following formula:

[0069]

[0070] wherein F p represents a primary local frequency domain fused feature, X represents an input feature map of a channel space attention channel module, CAB() represents a channel attention module function, SAB() represents a spatial channel attention module function, CSAB() represents a channel-spatial attention module function, and n is the number of times the CSAB module is used.

[0071] the local-global fusion frequency domain time feature F ft is fused by the following formula:

[0072]

[0073] wherein, is a primary local-global fusion feature;

[0074] the unified feature representation F fst is fused by the following formula:

[0075]

[0076] wherein, denotes a primary spatio-temporal frequency fusion feature.

[0077] Further, any one of the technical solutions or a combination of the technical solutions described above, the detection head adopts the following loss function to calculate the loss:

[0078] ;

[0079] wherein, denotes the intersection over union loss, which is used to measure the degree of overlap between the predicted region and the label region, denotes the hyperparameter of the intersection over union loss function; denotes the normalized wasserstein distance loss, which is used to measure the degree of difference between the predicted boundary and the real boundary, denotes the hyperparameter of the normalized wasserstein distance loss function; denotes the improved bounding box regression loss, λ reg denotes the hyperparameter weight of the bounding box regression loss; denotes the target classification loss, λ cls denotes the hyperparameter weight of the target classification loss; denotes the target prediction loss, λ obj denotes the hyperparameter weight of the target prediction loss; is calculated using the S-shaped focal loss function, the loss adopts the normalized wasserstein distance loss function to fit the feature distribution of the crack; wherein, C is a constant related to the data set, B p and B ture denote the predicted frame and the real frame in feature learning, respectively, N p and N tureGaussian distribution representing the predicted box and the real box.

[0080] Further, any of the technical solutions or combinations of the technical solutions described above, the feature image sequence set of the belt crack is constructed by the following method:

[0081] Collecting the original image sequence of the belt, each original image sequence has a plurality of continuous frame original images;

[0082] Converting each frame of the original image into a uniform preset size, and normalizing the image pixel value to obtain a preprocessed image sequence;

[0083] Initializing the time domain sampling window, selecting a continuous t-frame window image set I t ={I1,I2,…,I t} from each preprocessed image sequence;

[0084] Inputting each frame in the window image set I t to the shared weight backbone feature extraction module to extract primary features, to obtain a feature image sequence F t ={F1,F2,…,F t}∈R t×c×h×w , wherein t represents the number of image frames in the feature image sequence, c represents the channel number of the feature image, h represents the height of the feature image, and w represents the width of the feature image;

[0085] A plurality of feature image sequences F t comprise a feature image sequence set.

[0086] Another aspect of the application, the application provides a belt crack detection method, comprising the following steps:

[0087] Using the method described above to construct a belt crack detection model;

[0088] Collecting the image sequence of the belt to be detected, and inputting it into the belt crack detection model;

[0089] The belt crack detection model predicts the area of the belt to be detected with cracks, and outputs the predicted result.

[0090] The technical solutions provided by the application have the following beneficial effects:

[0091] a. Design a time, space, and frequency domain feature learning algorithm, focusing on learning the features of cracks in the three key domains of time, space, and frequency. By capturing the spatial information of cracks in the spatial domain, such as shape, texture, and location, analyzing the trend and dynamic characteristics of cracks over time in the time domain, and extracting the frequency distribution and periodic characteristics of cracks in the frequency domain, a comprehensive and accurate learning of crack features is achieved, enhancing the model's ability to recognize complex crack patterns and improving the accuracy and robustness of detection;

[0092] b. Design a feature level deep fusion algorithm to more efficiently and accurately fuse the time, space, and frequency domain features, providing more complete, high-quality, and representative feature representations for subsequent crack detection and recognition, significantly improving the model's learning ability and detection accuracy of belt crack features, enhancing the model's robustness and generalization ability in complex industrial scenarios, and achieving efficient and accurate detection of belt cracks, providing reliable technical support for belt maintenance in industrial production. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0094] Figure 1 The flowchart of the belt crack detection model construction method based on three-domain feature learning provided by an exemplary embodiment of the present application is shown in the figure;

[0095] Figure 2 The schematic block diagram of the belt crack detection model based on three-domain feature learning provided by an exemplary embodiment of the present application is shown in the figure;

[0096] Figure 3 The basic flowchart of spatial domain feature learning provided by an exemplary embodiment of the present application is shown in the figure;

[0097] Figure 4 The schematic block diagram of the hierarchical spatial perception module provided by an exemplary embodiment of the present application is shown in the figure;

[0098] Figure 5 The basic flowchart of time domain feature learning provided by an exemplary embodiment of the present application is shown in the figure;

[0099] Figure 6 The schematic block diagram of the time aggregation module provided by an exemplary embodiment of the present application is shown in the figure;

[0100] Figure 7 The basic flow diagram of frequency domain feature learning provided for an exemplary embodiment of the present application;

[0101] Figure 8 The schematic block diagram of the wavelet enhanced frequency domain perception module provided for an exemplary embodiment of the present application;

[0102] Figure 9 The first single-frame original image provided for an embodiment of the present application;

[0103] Figure 10 The box label image after artificial labeling corresponding to Figure 9 ;

[0104] Figure 11 The detection result of the belt crack detection model constructed by using the embodiment of the present application on Figure 9 ;

[0105] Figure 12 The second single-frame original image provided for an embodiment of the present application;

[0106] Figure 13 The box label image after artificial labeling corresponding to Figure 12 ;

[0107] Figure 14 The detection result of the belt crack detection model constructed by using the embodiment of the present application on Figure 12 ;

[0108] Figure 15 The third single-frame original image provided for an embodiment of the present application;

[0109] Figure 16 The box label image after artificial labeling corresponding to Figure 15 ;

[0110] Figure 17 The detection result of the belt crack detection model constructed by using the embodiment of the present application on Figure 15 ;

[0111] Figure 18 The flow diagram of the crack detection method using the belt crack detection model based on three-domain feature learning provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0112] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0113] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0114] In one embodiment of the present application, a three-domain feature learning based belt crack detection model construction method is provided, as shown in Figure 1 and Figure 2 The construction method comprises the following steps:

[0115] S100: Collecting a learning sample set: collecting a set of original image sequences of belt cracks, and performing artificial labeling of cracks to obtain corresponding box label images;

[0116] Specifically, each original image sequence has N original images of consecutive frames, Figure 9 、 Figure 12 、 Figure 15 Three original images in different sequences are represented as shown in Figure 10 、 Figure 13 、 Figure 16 , that is, the cracks in the original images are identified and marked by artificial labeling, and the non-crack regions are not framed.

[0117] S200: Constructing a set of feature image sequences of belt cracks based on the set of original image sequences, each feature image sequence in the set comprising a feature image corresponding to a plurality of consecutive original images;

[0118] The set of feature image sequences of the belt crack is specifically constructed in the following manner: S210, converting each frame of the original image sequence into a uniform preset size, and normalizing the image pixel values to obtain a preprocessed image sequence;

[0119] S220, initializing a time domain sampling window, and selecting a continuous t-frame window image set I t ={I1,I2,…,I t} from each preprocessed image sequence, where H and W represent the height and width of the preprocessed image respectively; in this way, multiple window image sets I t can be obtained from one original image sequence, for example, if the original image sequence has 100 continuous original images and the capacity of the sampling window is 5 images, at most 96 window image sets I t can be obtained from the original image sequence.

[0120] S230, inputting each frame in the window image set I t to a backbone feature extraction module with shared weights to extract primary features and obtain a corresponding t-frame feature image sequence F t ={F1,F2,…,F t}∈R t×c×h×w , where c represents the channel number of the feature image, h represents the height of the feature image, and w represents the width of the feature image.

[0121] S240, multiple feature image sequences F t form a set of feature image sequences.

[0122] S300: constructing a hierarchical spatial perception module configured to extract spatial features of the belt crack in the spatial dimension from a feature image sequence, denoted as F S .

[0123] S400: constructing a time aggregation module configured to mine context-related information in time for the feature image sequence to capture the dynamic evolution law of the belt crack in the time dimension and extract the time sequence features of the belt crack, denoted as F T .

[0124] S500: constructing a wavelet enhanced frequency domain perception module configured to map the feature image sequence to the wavelet frequency domain for feature decoupling and enhancement to obtain the frequency domain features of the belt crack, denoted as F F .

[0125] S600: constructing a cross-domain feature compensation module configured to compensate the spatial features F S , the time sequence features F T and the frequency domain features FF Dynamic alignment and fusion are performed, and a unified feature representation F is generated by a residual compensation mechanism to bridge the differences between multi-domain features fst ;

[0126] S700: A detection head is constructed, which is configured to define a loss function for the unified feature representation F fst , and determine the error of the positioning result according to the corresponding box label image, and iteratively optimize the network parameters of the belt crack detection model through error back propagation to improve the crack recognition and positioning ability of the model. After feature fusion and network optimization, a belt crack detection model based on time-space-frequency three-domain feature learning and hierarchical fusion is finally constructed. The model has strong crack detection performance and can efficiently and accurately identify belt cracks to ensure the safe operation of the belt conveying system.

[0127] The construction methods of the layered space perception module, the time aggregation module, and the wavelet enhanced frequency domain perception module will be described one by one as follows:

[0128] As shown in Figure 3 and Figure 4 , the layered space perception module is configured with a short-distance space feature branch and a long-distance space feature branch. In this embodiment, the key frames in each feature image sequence F t are determined according to a preset rule, and the remaining frame images are determined as reference frames. The reference frames are determined as [F1,…, F t-1 ], and the key frames are F t . The key frames and other reference frames constitute a dynamic space-time context. The layered space perception module extracts shallow features and multi-level space features of the key frames from the input sample image sequence through a multi-scale feature pyramid structure. The shallow features include edge, texture, and other bottom-level visual elements, which are used for accurate positioning of the crack region boundary. The deep features include encoded semantic information (such as crack type and severity), which improves the recognition ability of complex crack patterns.

[0129] The short-distance space feature branch adopts a multi-scale feature fusion network structure to obtain local region space features of multiple reference frames through different scales of receptive fields.

[0130] The long-distance space feature branch uses an attention mechanism to capture the space features of the long-distance dependencies in the key frame F t , thereby achieving comprehensive space feature extraction of the key frame in the current sampling window.

[0131] The local region space features of the reference frames and the space features of the key frame F t are combined to determine the space features F S of the belt crack in the spatial dimension.

[0132] Specifically, the short-distance spatial feature branch is configured with a local relationship extraction module and a non-local attention module, and the long-distance spatial feature branch is configured with a memory enhancement module, and the spatial feature F is determined by the following manner S :

[0133] S310, the local relationship extraction module extracts features from the reference frame, and fuses the features with the key frame F t to obtain the inter-frame local spatial relationship F l , specifically as follows:

[0134]

[0135] Where [F1,…,F t ] represents the t-th frame feature image in the feature image sequence, t≥3, wherein [F1,…,F t-1 ] represents the reference frame, F t represents the key frame, Concat( ) represents the concatenation function, Conv( ) represents the convolution function, represents element-level multiplication, and σ represents the activation function, represents the updated feature map after the reference frame and the key frame are fused.

[0136] S320, the non-local attention module focuses on the long-distance dependence of the inter-frame local spatial relationship F l of the reference frame, and mines the correlation between adjacent frames at different pixel point pairs, so as to realize long-distance dependence modeling, and obtain the local-global spatial relationship F g of the reference frame, specifically as follows:

[0137]

[0138] Where Q represents a query matrix obtained by mapping F l , which is used to calculate the correlation between the current position and other positions; K represents a key matrix obtained by mapping F l , which is used as an index for the query and is used to match the correlation; V represents a value matrix obtained by mapping F l , which is used to aggregate relevant feature information through semantic content; W Q represents the weight matrix of the query matrix Q, W K represents the weight matrix of the key matrix K, W V represents the weight matrix of the value matrix V, MatMul( ) represents matrix multiplication, and Softmax( ) represents normalization of the correlation score, represents a scaling factor for standardization adjustment, which avoids gradient disappearance in the calculation process.

[0139] S330, the memory enhancement module simulates the persistence of attention of the human eye visual tracking system, and stores and queries the key frame F t to query the local-global spatial relationship F g of the reference frame to achieve the spatial context of the belt crack in the spatial dimension F S , and the feature storage and query mechanism of the memory enhancement module is implemented as follows:

[0140]

[0141] wherein, and are convolution functions based on different embedding layers, represents the convolution function of the key key value under the query mechanism, represents the convolution function of the semantic content under the query mechanism, represents the convolution function of the key key value under the storage mechanism, represents the convolution function of the semantic content under the storage mechanism;

[0142] The key frame F t and the local-global spatial relationship F g of the reference frame are input into the memory enhancement module to obtain the spatial feature F S of the belt crack in the spatial dimension.

[0143] Regarding the time aggregation module: the spatiotemporal attention mechanism is used to capture the time sequence dependence of the shallow features extracted by the hierarchical spatial perception module, and the perception ability of the crack development key node (such as crack initiation and expansion mutation) is strengthened to obtain the time sequence dependent feature. The feature image sequence is set as [F1,…,F t ] represents the t-frame feature image, as shown in Figure 5 The time aggregation module is configured with an affinity calculation module and a difference calculation module, and the architecture is as shown in Figure 6 , wherein:

[0144] S410, the affinity calculation module calculates the information C(i,j) between any two different frames in the feature image sequence by the following formula: C(i,j) = F i ·F j , wherein F i represents the i-th feature image in the feature image sequence, and F j represents the j-th feature image in the feature image sequence.

[0145] S420, the difference calculation module calculates the pixel-level difference feature map D between the selected frame image and its adjacent frame image through the following formula:

[0146]

[0147] wherein D1 represents the t-th frame feature map F t in the Euclidean space, C (1,1) represents the t-th frame feature map F t and its own affinity weight; D j represents the (t-j+1)-th frame feature map F t-j+1 in the Euclidean space, d( ) represents difference operation, C (1, j) represents the t-th frame feature map F t and the (t-j+1)-th frame feature map F t-j+1 between them.

[0148] S430, define the fusion feature in time dimension :

[0149]

[0150] wherein t represents the number of image frames in the feature image sequence, represents element-level multiplication, F j -F1 represents the difference feature map between the j-th feature map and the 1-st feature map in the feature image sequence, represents the correlation feature map between the 1-st feature map and the j-th feature map in the feature image sequence;

[0151] S440, the fusion feature in time dimension is obtained by splicing and fusing multiple sequence frames with each other to obtain the time sequence feature F T of the belt crack.

[0152] Regarding the wavelet enhanced frequency domain perception module: the deep layer space feature extracted by the hierarchical space perception module is mapped to the frequency domain through two-dimensional discrete wavelet transform, and the deep layer space feature is decomposed into a low-frequency approximation component (capturing the overall morphology of the crack area) and a high-frequency detail component (highlighting the edge mutation feature), and the time sequence dependent feature output by the time aggregation module is complemented in feature to obtain a time-frequency feature. As shown in Figure 7 , the wavelet enhanced frequency domain perception module is configured with a frequency domain decoupling module, a frequency domain reconstruction module and a frequency domain feature fusion module, wherein:

[0153] As shown in Figure 8As shown, the frequency domain decoupling module is configured to perform frequency domain decoupling on each frame of feature image in the sequence of feature images, decompose it into a low-frequency component and a high-frequency component, and recursively decouple the low-frequency component to obtain the lowest layer of low-frequency feature information;

[0154] The frequency domain reconstruction module restores the features upward in a recursive and step-by-step manner to obtain features with the same original feature dimension before the frequency domain decoupling by the frequency domain decoupling module;

[0155] The frequency domain feature fusion module fuses the feature set after the frequency domain decoupling by the frequency domain decoupling module and the reconstruction by the frequency domain reconstruction module to obtain high-layer frequency domain fusion features Z (0) , and further fuses multiple frames of feature images in the sequence of feature images to obtain frequency domain features F F , specifically as follows:

[0156] S510, the frequency domain decoupling module is a cascaded wavelet decomposition feature WT(X):

[0157]

[0158] The cascaded wavelet decomposition feature WT(X) obtains lower-layer frequency domain features by recursively decomposing the low-frequency component X LL , and obtains 4 frequency domain components (1 low-frequency component and 3 high-frequency components) each time; in the above formula, represents the current decomposition level of multi-level decoupling, represents the low-frequency component from the frequency domain component of the previous level, represents the horizontal high-frequency component from the frequency domain component of the previous level, represents the vertical high-frequency component from the frequency domain component of the previous level, represents the diagonal high-frequency component from the frequency domain component of the previous level;

[0159] S520, the frequency domain reconstruction module processes different sub-band frequency domain components in the cascaded wavelet decomposition feature WT(X) through a small convolution kernel, and then obtains output features Y through inverse wavelet transform: Y=IWT(Conv(W,WT(X))), where IWT() is inverse wavelet transform, X represents the input feature of the current level, W represents a deep convolution kernel with a size of k×k, and each sub-band frequency domain of the current level obtained through WT() transform is convolved with W;

[0160] S530, the formula for convolution calculation of the output feature Y in the jth level is as follows:

[0161]

[0162] wherein W (j) represents the weight kernel in the jth level, represents the high frequency component in the jth level , and , Conv() represents the convolution operation, represents the low frequency component of the output feature Y in the jth level, represents the horizontal high frequency component of the output feature Y in the jth level, represents the vertical high frequency component of the output feature Y in the jth level, represents the diagonal high frequency component of the output feature Y in the jth level.

[0163] S540, the frequency domain feature fusion module restores the output feature Y by step-by-step fusion through inverse wavelet transform to obtain the frequency domain fusion feature Z (j) :

[0164]

[0165] wherein Z (j+1) = 0 represents the initial reconstructed frequency domain feature of the bottom level, Z (0) represents the high-level frequency domain fusion feature obtained by recursive reconstruction, represents the bottom-up superposition fusion of the (j+1)th level frequency domain fusion feature Z and ;

[0166] S550, the high-level frequency domain fusion feature Z (0) is fused by splicing the t frame feature maps in the feature map sequence to obtain the frequency domain feature F of the belt crack F .

[0167] The extraction methods of the spatial feature F S , the time sequence feature F T and the frequency domain feature F F are described in detail in the above embodiments, based on the three-domain features being obtained, the cross-domain feature compensation module adopts residual connection and channel attention mechanism to splice and fuse the shallow spatial feature extracted by the hierarchical spatial perception module and the spatio-temporal feature output by the wavelet enhanced frequency domain perception module, to obtain the feature to be detected, which is specifically configured to perform the following steps:

[0168] S610, the spatial feature F S , the time sequence feature F T are convoluted and fused to obtain the time-space fusion feature F st : F st = Conv(Concat[F S , FT ]), wherein Concat( ) represents a concatenation function, Conv( ) represents a convolution function, the convolution function uses a 3x3 size convolution kernel and uses BN normalization and a SiLU( ) activation function;

[0169] S620, fusing the time sequence feature F T , the frequency domain feature F F to calculate the local frequency domain time feature and the global frequency domain time feature :

[0170]

[0171] wherein WFM( ) represents a wavelet enhanced frequency domain perception algorithm, F t represents a set of t frame feature images in the feature image sequence, and Swin( ) represents a Swin Transformer attention mechanism;

[0172] In order to fuse the time space fusion feature F , the local frequency domain time feature and the global frequency domain time feature , a multi-dimensional feature fusion strategy and a residual compensation mechanism are designed to realize deep fusion of the time space frequency three-domain features, as follows:

[0173] S630, fusing the local frequency domain time feature F and the time space fusion feature F st to obtain a local frequency domain time space fusion feature F , that is, wherein RCU1( ) represents a cross-domain feature compensation unit, and the specific fusion formula is as follows:

[0174]

[0175] wherein F represents a primary local frequency domain fusion feature, X represents an input feature map of a channel space attention channel module, CAB( ) represents a channel attention module function, SAB( ) represents a spatial channel attention module function, CSAB( ) represents a channel-spatial attention module function, and n is the number of times the CSAB module is used.

[0176] S640, fusing the global frequency domain time feature F and the local frequency domain time feature F to obtain a local global fusion frequency domain time feature F ft , that is, wherein RCU2( ) represents a cross-domain feature compensation unit, and the specific fusion formula is as follows:

[0177]

[0178] wherein, is a primary local-global fusion feature.

[0179] S650, the local frequency domain spatio-temporal fusion feature and the local-global fusion frequency domain temporal feature F ft is obtained by cross-domain feature fusion to obtain a unified feature representation F fst that is, wherein RCU3() represents a cross-domain feature compensation unit, and the specific fusion formula is as follows:

[0180]

[0181] wherein, represents a primary spatio-temporal frequency fusion feature.

[0182] Regarding the detection head: for pixel-level classification and boundary regression of the to-be-detected feature output by the cross-domain feature compensation unit by defining a loss function, completing crack region positioning, calculating error loss, and iteratively optimizing the network parameters of the belt crack detection model. The detection head uses the following loss function to calculate the loss:

[0183] ;

[0184] wherein, represents the intersection over union loss, which is used to measure the overlap degree of the predicted region and the label region, represents the hyperparameter of the intersection over union loss function; represents the normalized Wosters distance loss loss (i.e. Normalized Wasserstein Distance loss), which is used to measure the difference between the predicted boundary and the true boundary, represents the hyperparameter of the normalized Wosters distance loss function; represents an improved boundary box regression loss, λ reg represents the hyperparameter weight of the boundary box regression loss; represents a target classification loss, λ cls represents the hyperparameter weight of the target classification loss; represents a target prediction loss, λ obj represents the hyperparameter weight of the target prediction loss; uses the S-shaped focal loss function (i.e. sigmoid focal loss function) to calculate, the loss uses the normalized Wosters distance loss function (i.e. Normalized Wasserstein Distance loss ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​fitting the characteristic distribution of the crack, wherein, C is a constant related to the data set, B p and B ture pred and gt represent the prediction frame and the real frame in feature learning, N p and N ture pred and gt represent the Gaussian distribution of the prediction frame and the real frame.

[0185] The embodiment constructs a hierarchical spatial perception module in the detection model, which is used to extract multi-level spatial features related to key frames from the input sample image sequence, so as to comprehensively and meticulously capture the spatial information of the key frames in the image sequence; a convolutional long short-term memory network module is constructed, which is used to capture the time sequence dependence of the shallow features extracted by the spatio-temporal feature extractor module, so as to obtain the time sequence dependent features; a time aggregation module is constructed, which is used to aggregate the features in the time dimension of the key frames in the input image sequence, so as to capture the dynamic change trend and time sequence relationship of the key frames in the time sequence; a spatial multi-scale feature fusion module is constructed, which is used to splice and fuse the shallow spatial features in the multi-level spatial features of the key frames, so as to obtain the features to be segmented and detected; a wavelet enhanced frequency domain perception module is constructed, which is used to perform frequency domain analysis on the input image sequence, enhance the frequency features of the image through wavelet transform technology, and capture the subtle changes and periodic characteristics of the crack in the frequency domain; a cross-domain feature compensation module is constructed, which is specially used for deeply fusing the time aggregation features, hierarchical spatial perception features and wavelet enhanced frequency domain perception features extracted above; a detection head is constructed, which is used to locate the crack region of the belt to be detected by defining a loss function, calculate the error loss and iteratively optimize the network parameters of the belt crack detection model. Through the synergistic effect of the above modules, a model with good training parameters is finally obtained, so as to effectively locate and detect the crack region target of the conveying belt. The model constructed by the construction method makes the detection and positioning of the crack region target of the conveying belt more accurate and robust, can effectively detect the crack region target of the conveying belt, and can segment and detect the crack in the target region, and can be applied in important fields such as maintenance and supervision of enterprise material conveying belts.

[0186] In an embodiment of the present application, a belt crack detection method is provided, as shown in Figure 18 The detection method comprises the following steps:

[0187] The belt crack detection model is constructed by using the method described above;

[0188] An image sequence of the belt to be detected is collected and input into the belt crack detection model;

[0189] The belt crack detection model predicts a region where a belt to be detected has a crack, and outputs a result of the prediction.

[0190] It should be noted that the belt crack detection method provided in this embodiment and the belt crack detection model construction method based on time, space and frequency domain feature learning provided in the above embodiment belong to the same inventive concept, and all the contents in the above model construction method embodiment are incorporated into this crack detection method embodiment by reference.

[0191] The present application provides the following specific embodiments: a belt crack detection dataset BeltCrack14ks is constructed, which includes 14087 pictures of 29 belt conveying sequences, and the 29 belt conveying sequences are divided into a training set, a validation set and a test set according to a ratio of 6:2:2, wherein the training set includes 8773 pictures of 18 sequences; wherein the validation set includes 2596 pictures of 5 sequences; wherein the test set includes 7932 pictures of 6 sequences;

[0192] (1) Initialization of the experiment:

[0193] Before inputting the belt crack detection dataset, the size of all images in the training set and the test set is converted to 512*512 pixels, the sampling time window size is set to 5, and then the image pixel value is normalized and input into the model for belt crack region detection. In this embodiment, SGD is used as the optimizer, the batch size is set to 4, the initial learning rate is set to 0.01, the momentum term is set to 0.937, and the BeltCrack14ks is trained for 100 epochs on the dataset. All evaluation experiments of the model in this embodiment are performed on two NVIDIA RTX 4090 graphics cards. In addition, in the belt crack region segmentation detection experiment of this embodiment, the evaluation indexes mainly include mAP50, Precision, Recall and F1.

[0194] (2) Backbone feature extraction stage:

[0195] Before inputting the preprocessed image data into the model, the time domain sampling window is initialized, and then t consecutive images (I t ={I1,I2,…,I t} in I are randomly selected as key frames, and each frame is input into the shared weight backbone feature extraction module to extract primary features, obtaining a multi-frame feature set F t ={F1,F2,…,F t}∈R t×c×h×w .

[0196] (3) Hierarchical spatial perception stage:

[0197] obtaining a plurality of frame feature sets F generated through the primary feature extraction stage t ={F1,F2,…,F t} , and inputting key frames F t in the plurality of frame feature sets F into a long-distance spatial feature branch to obtain long-distance spatial features F SG , and inputting reference frames [F1,…,F t-1 ] into a short-distance spatial feature branch to obtain short-distance spatial features .

[0198] (4) Time aggregation stage:

[0199] obtaining a plurality of frame feature image sequences F generated through the primary feature extraction stage t =[F1,F2,…,F t ] , inputting each frame into an affinity calculation module and a difference calculation module, obtaining affinity weights between any two different image frames through the affinity calculation module, and further obtaining inter-frame and intra-frame information C={C (1,1) ,C (1,2) ,…,C (1,t)}; wherein C (1,1) represents an affinity weight of the t-th frame feature image F t in the plurality of frame feature image sequences with itself, C (1,2) represents an affinity weight between the t-th frame feature image F t and the (t-1)-th frame feature image F t-1 in the plurality of frame feature image sequences, and C (1,t) represents an affinity weight between the t-th frame feature image F t and the first frame feature image F1 in the plurality of frame feature image sequences.

[0200] calculating pixel-level difference feature maps D={D1,D2,…,D t} between the selected frame and its adjacent frames through the difference calculation module, wherein D1 represents a pixel-level difference value of the t-th frame feature image F t in the plurality of frame feature image sequences in the Euclidean space, D2 represents a pixel-level difference value of the (t-1)-th frame feature image F t-1 in the plurality of frame feature image sequences in the Euclidean space, and D t represents a pixel-level difference value of the first frame feature image F1 in the plurality of frame feature image sequences in the Euclidean space.

[0201] (5) Wavelet enhanced frequency domain perception stage:

[0202] obtaining a plurality of frame feature image sequences F generated through the primary feature extraction stage t ={F1,F2,…,F tAfter that, each frame is input into the frequency domain decoupling module for step-by-step decoupling, and 4 frequency domain subbands are obtained at each decoupling level The obtained frequency domain subbands are input into the frequency domain reconstruction module and the frequency domain fusion module for restoration to obtain the frequency domain features of the image frame .

[0203] (6) Cross-domain feature fusion stage:

[0204] The time feature, the space feature and the frequency domain feature obtained above are fused to obtain the time-space-frequency three-domain fusion feature, i.e., the unified feature representation F, which is finally input into the detection head fst .

[0205] (7) Detection and loss function calculation stage:

[0206] The loss value is calculated through the predefined loss function, and the model is effectively learned by using the back propagation gradient descent algorithm to obtain the model parameters with high training performance, so as to effectively detect the crack region of the conveying belt.

[0207] The trained model is used to detect the crack of the conveying belt image sequence shown in Figure 9 , Figure 12 , Figure 15 The detection results are shown in Figure 11 , Figure 14 , Figure 17 Through comparative analysis, Figure 10 the box label in Figure 9 , Figure 13 the box label in Figure 14 , Figure 16 the box label in Figure 17 , it can be clearly seen that the belt crack detection model based on time, space and frequency domain three-domain feature learning constructed in the embodiment has high detection accuracy and strong robustness. The model can accurately detect the conveying belt sequence image collected by the ordinary RGB camera imaging device, accurately determine whether the target conveying belt region has cracks, and can be widely used in key fields such as maintenance and supervision of enterprise belt conveyors.

[0208] It is to be noted that, as used in this document, the term "indicia" is intended to encompass any type of data, information, or other content, whether in the form of text, graphics, images, video, audio, or otherwise. It is to be further noted that, as used in this document, the terms "coupled" and "connected," along with derivatives thereof, can be used to mean one or more of the following: in electrical communication with; physically contacting with; in long-distance communication with; and / or not in direct contact with. It is to be further noted that, as used in this document, the terms "include" and "comprise," along with derivatives thereof, can be used to indicate inclusion of one or more elements or steps; these terms are not intended to, nor do they, imply that any or all functionality can be included with any or all elements or steps; and / or any such elements or steps are each independently repeatable both logically and temporally. In addition, the term "exemplary" is intended to refer to a non-limiting example, embodiment, or aspect. Moreover, the term "in response to" is intended to mean that a particular action is performed in response to one or more events or conditions, but not necessarily directly or immediately in response to the one or more events or conditions.

[0209] The foregoing is merely illustrative of the principles of this application and various modifications can be made by those skilled in the art, without departing from the spirit and scope of the application. Accordingly, the above description is not intended to limit the scope of the application.

Claims

1. A method for constructing a belt crack detection model based on three-domain feature learning, characterized in that, Includes the following steps: Collect a learning sample set: Collect a set of original image sequences of belt cracks and manually annotate the cracks to obtain corresponding box label images; A set of feature image sequences of belt cracks is constructed based on the set of original image sequences. Each feature image sequence in the set includes feature images that correspond one-to-one with multiple consecutive original images. A hierarchical spatial awareness module is constructed, which is configured to extract the spatial features of belt cracks in the spatial dimension from a sequence of feature images, denoted as F. S ; A time aggregation module is constructed and configured to mine the temporal correlation information of the context of the feature image sequence to capture the dynamic evolution of belt cracks in the time dimension and extract the temporal features of belt cracks, denoted as F. T ; A wavelet-enhanced frequency domain sensing module is constructed, which is configured to map the feature image sequence to the wavelet frequency domain for feature decoupling and enhancement, thereby obtaining the frequency domain features of the belt crack, denoted as F. F ; A cross-domain feature compensation module is constructed, which is configured to compensate the spatial feature F S Temporal characteristics F T and frequency domain features F F Dynamic alignment and fusion are performed, and the differences between multi-domain features are bridged through a residual compensation mechanism to generate a unified feature representation, denoted as F. fst The cross-domain feature compensation module is configured to: compensate the spatial feature F S Temporal characteristics F T Convolutional fusion is performed to obtain the temporal-spatial fusion feature F. st :F st =Conv(Concat[F S ,F T ]), where Concat() represents the concatenation function and Conv() represents the convolution function; through the temporal feature F T Frequency domain characteristics F F Calculate local frequency domain time characteristics and global frequency domain time features : Where WFM() represents the wavelet-enhanced frequency domain sensing algorithm, F t This represents the set of t-frame feature images in the feature image sequence, and Swin() represents the Swin Transformer attention mechanism; for local frequency domain temporal features Features of temporal and spatial integration F st Cross-domain feature fusion yields local frequency domain spatiotemporal fusion features. ; global frequency domain time characteristics With local frequency domain time characteristics Cross-domain feature fusion yields local-global fused frequency domain time features F ft ; for the local frequency domain spatiotemporal fusion features Combined with local and global frequency domain time features F ft Cross-domain feature fusion yields a unified feature representation F fst ; wherein, the local frequency domain spatiotemporal fusion feature The following formula is used for fusion: ,in, Let X represent the primary local frequency domain fusion feature, X represent the input feature map of the channel spatial attention channel module, CAB() represent the channel attention module function, SAB() represent the spatial channel attention module function, CSAB() represent the channel-spatial attention module function, and n represent the number of times the CSAB module is used; the local-global fusion frequency domain temporal feature F ft The following formula is used for fusion: ,in, This represents the initial local-to-global fusion feature; the unified feature is represented by F. fst The following formula is used for fusion: ,in, This indicates the primary spatiotemporal frequency fusion characteristics; A detection head is constructed, which is configured to apply a loss function to the unified feature representation F. fst The model is located and the error of the location result is determined based on the corresponding box label image. The network parameters of the belt crack detection model are iteratively optimized through error backpropagation.

2. The method for constructing a belt crack detection model according to claim 1, characterized in that, Keyframes in each feature image sequence are determined according to preset rules, and the remaining frame images are determined as reference frames; The hierarchical spatial perception module is configured with a short-range spatial feature branch and a long-range spatial feature branch. The short-range spatial feature branch adopts a multi-scale feature fusion network structure to acquire local regional spatial features of multiple reference frames through receptive fields of different scales. The long-range spatial feature branch uses an attention mechanism to capture the spatial features of long-range dependencies in the keyframes; By combining the local regional spatial features of the reference frame and the spatial features of the key frame, the spatial features F of the belt crack in the spatial dimension are determined. S .

3. The method for constructing a belt crack detection model according to claim 2, characterized in that, The short-range spatial feature branch is equipped with a local relation extraction module and a non-local attention module, while the long-range spatial feature branch is equipped with a memory enhancement module. The spatial feature F is determined through the following methods. S : The local relation extraction module extracts features from the reference frame and concatenates and fuses them with the key frame to obtain the inter-frame local spatial relation F. l ; The nonlocal attention module focuses on the inter-frame local spatial relationship F of the reference frame. l The long-range dependency relationship is obtained by mining the correlation between different pixel pairs in adjacent frames to achieve long-range dependency modeling and obtain the local-global spatial relationship F of the reference frame. g ; The memory enhancement module, simulating the continuous attention capability of a human visual tracking system, uses a feature storage and query mechanism to query the keyframes and establish the local-to-global spatial relationship F of the reference frames. g Used for storage, enabling continuous updates of the spatial context, and obtaining the spatial features F of the belt crack in the spatial dimension. S .

4. The method for constructing a belt crack detection model according to claim 3, characterized in that, The inter-frame local spatial relationship F is obtained in the following manner. l : ； Among them, [F1,…,F t ] represents the feature images of frame t in the feature image sequence, where t≥3, where [F1,…,F t-1 ] indicates the reference frame, F t The keyframe is represented by `Concat()`, the concatenation function by `Conv()`, the convolution function by `⨀`, and the activation function by `σ`. This represents the updated feature map after the reference frame and key frame have been stitched and fused together.

5. The method for constructing a belt crack detection model according to claim 3, characterized in that, The local-global spatial relationship F of the reference frame is obtained in the following manner. g : ； Where Q represents from F l The resulting query matrix is ​​used to calculate the correlation between the current location and other locations; K represents the correlation between F and other locations. l The resulting key matrix is ​​used as an index for queries and for matching relevance; V represents the key matrix from F. l The resulting value matrix is ​​used to aggregate relevant feature information through semantic content; W Q W represents the weight matrix of the query matrix Q. K W represents the weight matrix of the key matrix K. V Let V be the weight matrix of the value matrix V, MatMul() represents matrix multiplication, and Softmax() represents normalization of the correlation scores. This represents the scaling factor used for standardization adjustments.

6. The method for constructing a belt crack detection model according to claim 5, characterized in that, The feature query mechanism and feature storage mechanism of the memory enhancement module are implemented in the following ways: ； in, This represents the convolution function of key values ​​under the query mechanism. This represents the convolution function for semantic content under the query mechanism. This represents the convolution function for key values ​​under the storage mechanism. A convolution function representing semantic content under a storage mechanism; The keyframe F t The local-global spatial relationship F with the reference frame g The data is input into the memory enhancement module to obtain the spatial features F of the belt crack in the spatial dimension. S .

7. The method for constructing a belt crack detection model according to claim 1, characterized in that, The feature image sequence is set to [F1,…,F1]. t The t-frame feature image represents the time aggregation module, which is configured with an affinity calculation module and a difference calculation module, wherein: The affinity calculation module calculates the inter-frame and intra-frame information C(i,j) between any two different frames in the feature image sequence using the following formula: C(i,j) = F i ·F j , of which F i F represents the i-th feature map in the feature image sequence. j This represents the j-th feature map in the feature image sequence; The difference calculation module calculates the pixel-level difference feature map D between the selected frame image and its neighboring frame images using the following formula: ； Wherein, D1 represents the feature map F of the t-th frame in the feature image sequence. t The pixel-level difference value in Euclidean space, C (1,1) F represents the feature map of frame t in the feature image sequence. t Weight of affinity with itself; D j The feature image F represents the feature map of the (t-j+1)th frame in the feature image sequence. t-j+1 In Euclidean space, the pixel-level difference value is represented by d(), which indicates the difference operation. (1, j) F represents the feature map of frame t in the feature image sequence. t With the feature map F of the (t-j+1)th frame t-j+1 The affinity weight between them.

8. The method for constructing a belt crack detection model according to claim 7, characterized in that, The temporal characteristics F of the belt crack were obtained in the following manner. T : Define the fusion features in the time dimension : ； Where t represents the number of image frames in the feature image sequence, ⨀ represents element-wise multiplication, and F j - F1 represents the difference feature map between the j-th feature map and the 1st feature map in the feature image sequence. The feature map representing the correlation between the first feature map and the j-th feature map in the feature image sequence; The fusion features of the time dimension The temporal features F of the belt crack are obtained by splicing and fusing multiple sequence frames. T .

9. The method for constructing a belt crack detection model according to claim 1, characterized in that, The wavelet-enhanced frequency domain sensing module is configured with a frequency domain decoupling module, a frequency domain reconstruction module, and a frequency domain feature fusion module, wherein: The frequency domain decoupling module is configured to perform frequency domain decoupling on each frame of the feature image sequence, decompose it into low-frequency components and high-frequency components, and recursively decouple the low-frequency components to obtain the lowest-level low-frequency feature information. The frequency domain reconstruction module reconstructs features upwards in a recursive manner to obtain features with the same dimensions as the original features before frequency domain decoupling by the frequency domain decoupling module. The frequency domain feature fusion module fuses the feature sets that have been decoupled by the frequency domain decoupling module and reconstructed by the frequency domain reconstruction module to obtain the high-level frequency domain fused feature Z. (0) .

10. The method for constructing a belt crack detection model according to claim 9, characterized in that, The frequency domain decoupling module is a cascaded wavelet decomposition feature WT(X): ； in, This indicates the current decomposition level of the multi-level decoupling. Indicates components from the previous frequency domain. The low-frequency components, Indicates components from the previous frequency domain. Horizontal high-frequency components, Indicates components from the previous frequency domain. The vertical high-frequency components, Indicates components from the previous frequency domain. The high-frequency components on the diagonal; The frequency domain reconstruction module processes the frequency domain components of different sub-bands in the cascaded wavelet decomposition feature WT(X) through small convolution kernels, and then performs inverse wavelet transform to obtain the output feature Y: Y=IWT(Conv(W,WT(X))), where IWT() is the inverse wavelet transform, X represents the input feature of the current level, W represents a depth convolution kernel of size k×k, and each sub-band frequency domain obtained by WT() transformation is convolved with W; The formula for performing convolution calculation on the output feature Y at the j-th level is as follows: ； Among them, W (j) This represents the weight kernel in the j-th level. Represents the high-frequency components in the j-th level. , and Conv() represents the convolution operation. This represents the low-frequency component of the output feature Y at the j-th level. This represents the horizontal high-frequency component of the output feature Y at the j-th level. This represents the vertical high-frequency component of the output feature Y at the j-th level. This represents the diagonal high-frequency component of the output feature Y at the j-th level.

11. The method for constructing a belt crack detection model according to claim 10, characterized in that, The frequency domain feature fusion module obtains the frequency domain fused feature Z by fusing the output feature Y step by step and performing inverse wavelet transform to restore it. (j) : ； Among them, Z (j+1) =0 indicates the lowest level of initial reconstructed frequency domain features, Z (0) This represents the high-level frequency domain fusion features obtained through recursive reconstruction. This indicates that the frequency domain fusion feature Z after the (j+1)th level is... Perform bottom-up superposition and fusion; The high-level frequency domain fusion feature Z (0) By splicing and fusing multiple feature images from the feature image sequence, the frequency domain feature F of the belt crack is obtained. F .

12. The method for constructing a belt crack detection model according to any one of claims 1 to 11, characterized in that, The detection head uses the following loss function to calculate the loss: ； in, This represents the intersection-union loss, which measures the degree of overlap between the predicted region and the labeled region. The hyperparameters representing the intersection-union ratio (IU) loss function; This represents the normalized Wasserstein distance loss, which measures the degree of difference between the predicted boundary and the true boundary. The hyperparameters representing the normalized Wasserstein distance loss function; λ represents the improved bounding box regression loss. reg The hyperparameter weights represent the bounding box regression loss; Let λ represent the target classification loss. cls The hyperparameter weights represent the target classification loss; λ represents the target prediction loss. obj The hyperparameter weights represent the target prediction loss; and Calculated using the S-shaped focus loss function The loss function uses a normalized Wasserstein distance loss function to fit the characteristic distribution of the cracks; where C is a constant related to the dataset, and B... p and B ture N represents the predicted bounding box and the ground truth bounding box in feature learning, respectively. p and N ture This represents the Gaussian distribution of the predicted bounding box and the ground truth bounding box.

13. The method for constructing a belt crack detection model according to any one of claims 1 to 11, characterized in that, A set of feature image sequences of belt cracks is constructed using the following method: Acquire raw image sequences of the belt, with each raw image sequence containing multiple consecutive frames of raw images; Each original image frame is converted to a uniform preset size, and the image pixel values ​​are normalized to obtain a preprocessed image sequence. Initialize the temporal sampling window, and select a set of t consecutive frames I from each preprocessed image sequence. t ={I1,I2,…,I t }; The window image set I t Each frame in the algorithm is input into a shared-weight backbone feature extraction module to extract primary features, resulting in a feature image sequence F composed of the corresponding t-frame feature images. t ={F1,F2,…,F t }∈R t×c×h×w Where t represents the number of image frames in the feature image sequence, c represents the number of channels in the feature image, h represents the height of the feature image, and w represents the width of the feature image; Multiple feature image sequences F t A set of feature image sequences.

14. A method for detecting belt cracks, characterized in that, Includes the following steps: A belt crack detection model is constructed using the method described in any one of claims 1 to 13; The image sequence of the belt to be detected is acquired and input into the belt crack detection model; The belt crack detection model predicts the areas where cracks exist in the belt to be detected and outputs the prediction results.

Images