Belt tear detection method and system based on discrete state selectable space model

Abstract

The present invention provides a belt tear detection method and system based on a discrete state optional space model, comprising: collecting multiple images of belt running sites; marking tear targets on each image to obtain an image-label pair, and dividing all image-label pairs into a training set and a verification set in proportion; preprocessing the image data in the training set to expand the training set to obtain an input image I and a label L; using the input image I and the input language guidance text to train a belt tear open domain detection backbone network to obtain a trained belt tear open domain detection backbone network, the belt tear open domain detection backbone network consisting of multiple modules based on a discrete state optional space model M; using the trained belt tear open domain detection backbone network to detect images of the belt running site to be detected, and outputting tear detection results of the images of the belt running site to be detected. The present invention can efficiently detect belt tear.

Description

Technical Field

[0001] The present invention relates to the technical field of belt tear detection, and in particular to a belt tear detection method and system based on a discrete state selectable space model. Background Art

[0002] In modern industrial production, belt conveyors play a vital role in mining, logistics, manufacturing and other fields with their efficient and continuous material transportation capabilities. As the core component of the conveyor, the performance of the belt directly affects the efficiency and safety of production. However, during use, the belt is easily affected by factors such as material impact, improper loading, and mechanical failure, resulting in frequent tearing accidents. Once the belt is torn, it will not only cause downtime for maintenance and affect production progress, but also bring huge economic losses and safety hazards.

[0003] In response to this problem, technical R&D personnel and engineers have invested a lot of research and development work to find a timely and reliable belt tear detection technology. Traditional sensor-based detection technology has low accuracy and high installation and maintenance costs. Therefore, end-to-end image target detection methods using artificial intelligence have been widely used. High-resolution cameras are used to capture belt images in real time, and deep learning algorithms are used to identify tears and mark target locations and categories, which improves the response speed and detection accuracy of the detection system. However, in order to further improve the recognition accuracy of end-to-end image detection algorithms, larger and deeper models have to be designed, which means higher computing requirements, making it difficult for the algorithm to be implemented in actual factory application scenarios with limited computing resources. In addition, as an image-based single-modal detection system, the end-to-end image detection method is usually highly complex and dynamic in real-world application scenarios. However, the existing multimodal fusion methods are single, and the commonly used methods are to directly splice multimodal features or use cross self-attention, which cannot achieve fine-grained feature fusion, and the performance improvement for intensive prediction tasks such as belt tear target detection is very limited. Finally, in actual applications, new tear types often appear due to factors such as belt materials, working environment, shooting angles, and lighting conditions. When faced with the need for open domain detection, end-to-end image detection methods trained based on pre-defined categories cannot identify these unseen categories. Therefore, it is necessary to manually collect and annotate new category data and retrain or fine-tune the model, which requires significant manpower, time, and resource investment. Summary of the invention

[0004] The present invention provides a belt tear detection method and system based on a discrete state selectable space model to solve the problems existing in the prior art. The technical solution provided by the present invention is as follows:

[0005] On the one hand, a belt tear detection method based on a discrete state selectable space model is provided, the method comprising:

[0006] S1. Collect multiple images of the belt running site;

[0007] S2, label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into training set and validation set in proportion;

[0008] S3, preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L;

[0009] S4, using the input image I and the input language guidance text to train the belt tear open domain detection backbone network, to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D;

[0010] The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method;

[0011] The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary;

[0012] The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ;

[0013] The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ;

[0014] The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modality feature reconstruction prediction result p l , and output multimodal fusion features x mix ;

[0015] The multimodal decoder D is used to decode the multimodal fusion feature x mixDecode and output text consisting of tearing categories and detection box coordinates;

[0016] S5. Use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected.

[0017] On the other hand, a belt tear detection system based on a discrete state selectable space model is provided, the system comprising:

[0018] A collection unit, used for collecting a plurality of images of the belt running scene;

[0019] A labeling unit is used to label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into a training set and a validation set in proportion;

[0020] A preprocessing unit, used for preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L;

[0021] The training unit is used to train the belt tear open domain detection backbone network using the input image I and the input language guidance text to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D;

[0022] The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method;

[0023] The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary;

[0024] The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ;

[0025] The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ;

[0026] The multimodal interactive fusion module T based on mask reconstruction vl, used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modality feature reconstruction prediction result p l , and output multimodal fusion features x mix ;

[0027] The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates;

[0028] The detection unit is used to use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected.

[0029] On the other hand, an electronic device is provided, which includes a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the above-mentioned belt tear detection method based on discrete state optional space model.

[0030] On the other hand, a computer-readable storage medium is provided, in which instructions are stored, and the instructions are loaded and executed by a processor to implement the above-mentioned belt tear detection method based on discrete state selectable space model.

[0031] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0032] This paper proposes a new belt tear detection method, which has linear computational complexity, multimodal feature extraction and reasoning capabilities, and end-to-end open domain detection capabilities without manual adjustment. This method uses a new feature extraction and decoding method to achieve linear computational complexity, and by introducing text information, the model can infer new, reasonable, and physical-world tear categories, even if these categories have not appeared directly in the training data before, thereby achieving adaptive open domain detection without manually collecting and annotating data on new tear types or manually adjusting model parameters. Specifically:

[0033] 1) The backbone network for belt tear open domain detection based on the discrete state optional space model can model different elements in the input sequence with different weights, reducing the information redundancy of long sequence data. Its internal parallel computing method can also improve the reasoning speed of the model. The discrete state optional space model has linear computational complexity. In the case of high resolution and long sequences, the computational complexity of the model is much smaller than that of previous neural network models.

[0034] 2) The multimodal interactive fusion module based on mask reconstruction uses strict training tasks to force the module to understand the subtle correlation between visual features and text features, thereby achieving a deep fusion of element-level visual features and text features and enhancing the complementarity between multimodal features.

[0035] 3) Based on the language-guided text template, a consistent interface with the language model was customized for the object detection task, and an end-to-end open domain detection framework was designed. This eliminates the need to manually annotate image data and training models for new categories when new categories appear, improves the generalization of the model to new tearing categories, and reduces the cost of model maintenance.

[0036] 4) The image serialization method based on cross scanning retains more image details and continuity compared to the basic image serialization method based on single-direction scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flow chart of a belt tear detection method based on a discrete state selectable space model provided by an embodiment of the present invention;

[0039] Figure 2 A flow chart of another belt tear detection method based on a discrete state selectable space model provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of a belt tear open domain detection backbone network provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of the structure of a cross-scan image serialization module provided in an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of constructing a vocabulary provided by an embodiment of the present invention;

[0043] Figure 6 A schematic diagram of the structure of a language-to-text conversion module provided in an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of the structure of a visual encoder provided by an embodiment of the present invention;

[0045] Figure 8 A schematic diagram of the structure of a text encoder provided by an embodiment of the present invention;

[0046] Fig. 9 A schematic diagram of the structure of a multimodal interactive fusion module based on mask reconstruction provided by an embodiment of the present invention;

[0047] Fig.10 A schematic diagram of the structure of a multimodal decoder provided by an embodiment of the present invention;

[0048] Fig.11 A schematic diagram of a discrete state selectable space model structure provided by an embodiment of the present invention;

[0049] Fig.12 A block diagram of a belt tear detection system based on a discrete state selectable space model provided by an embodiment of the present invention;

[0050] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a belt tear detection method based on a discrete state selectable space model, the method comprising:

[0053] S1. Collect multiple images of the belt running site;

[0054] S2, label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into training set and validation set in proportion;

[0055] S3, preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L;

[0056] S4, using the input image I and the input language guidance text to train the belt tear open domain detection backbone network, to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l, Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D;

[0057] The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method;

[0058] The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary;

[0059] The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ;

[0060] The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ;

[0061] The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modal feature reconstruction prediction result p l , and output multimodal fusion features x mix ;

[0062] The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates;

[0063] S5. Use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected.

[0064] In the scenario of belt transportation, a high-resolution linear array camera is used to collect multiple images of the belt running site. The images are used as the input of the visual modality. The images are converted into an image sequence by serializing the images. At the same time, a language guidance text with a specified input format is designed as the input of the text modality and encoded into a text sequence. A visual encoder and a text encoder based on a discrete state optional space model are built to extract features from the input long image sequence and text sequence, in which different elements in the sequence are given different weights through a learnable hyperparameter matrix. A multimodal interactive fusion module based on mask reconstruction is designed. Through element-level mask operations and the constraints of the loss function, text features are reconstructed according to the masked visual features, and visual features are reconstructed according to the masked text features, forcing the model to learn and understand the connection between visual features and text features with semantic relevance, thereby achieving more fine-grained multimodal feature fusion. Finally, the multimodal features are decoded into text output in the form of tuples through a multimodal decoder. In the training phase, the label prediction error is measured using the cross entropy (CE) loss function, and the multimodal interaction fusion is measured using the mean square error (MSE) loss function. The model with the best performance on the validation set is retained as the final belt tear open domain detection model. In the inference phase, the output target detection result text is converted into a target detection box and the visualization result is provided to the user, such as Figure 2 As shown, a belt tear detection method based on a discrete state optional space model provided by an embodiment of the present invention is described in detail below. The method includes:

[0065] S1. Collect multiple images of the belt running site;

[0066] The embodiment of the present invention uses a high-resolution linear array camera to capture images of the belt running site.

[0067] S2, label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into training set and validation set in proportion;

[0068] The embodiment of the present invention manually labels the tearing targets on the image and classifies all data into different tearing target categories. <c>, including crack opening, belt hole, belt stacking, foreign body penetration, wire leakage, belt surface scratches, and marking the upper left corner coordinates (x1, y1) and lower right corner coordinates (x2, y2) of the target detection frame, and associate each image with the corresponding category <c>, the upper left point P1 (x1, y1) and the lower right point P2 (x2, y2) are regarded as an image-label pair;

[0069] All image-label pairs are divided into training set and validation set in a ratio of 4:1. The model will be trained on the training set and the model effect will be verified on the validation set. The model with the best performance on the validation set will be selected as the final model.

[0070] S3, preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L;

[0071] In the embodiment of the present invention, all image-label pairs are subjected to data enhancement by random horizontal mirror flipping, random scale scaling, and random size cropping, so as to expand the training data set and obtain input images I and labels L.

[0072] S4, using the input image I and the input language guidance text to train the belt tear open domain detection backbone network, to obtain the trained belt tear open domain detection backbone network, such as Figure 3 As shown in FIG. 1 , the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D;

[0073] The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method;

[0074] The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary;

[0075] The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ;

[0076] The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ;

[0077] The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modal feature reconstruction prediction result p l , and output multimodal fusion features x mix ;

[0078] The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates;

[0079] Alternatively, if Figure 4 As shown, the cross-scanning image serialization module is specifically used for:

[0080] Use convolution operation to cut the input image into a series of small image patches (size 16×16 resolution);

[0081] Considering the semantic continuity between image blocks, in order to improve the long-distance modeling capability of the model, all small image blocks are rearranged using cross scanning methods in four different directions to form a long sequence of four image blocks. The four long sequences are added and averaged to obtain a new long sequence of image blocks.

[0082] The new long sequence of image blocks is flattened into a one-dimensional vector to obtain the serialized image vector.

[0083] The embodiment of the present invention designs an effective language guidance text and specifies the required output format to achieve the description of the visual task of belt tear target detection.

[0084] Optionally, the template of the language guidance text is:

[0085] Identify the objects belonging to the belt tear category in the corresponding input image and draw a detection box around the object. The output format should be a list of tuples in the format of (C,x1,y1,x2,y2), where C is the tear category name, x1, y1, x2, y2 are the coordinates of the upper left corner and lower right corner of the detection box, the x coordinate should be in the range of 2048, and the y coordinate should be in the range of 2000.

[0086] Since deep learning models can only perform various calculations mathematically, it is required to digitize the text represented by the string. The vocabulary can vectorize the string and is the basis for the model to understand and generate text.

[0087] Alternatively, if Figure 5 As shown, the construction of the vocabulary specifically includes:

[0088] A large amount of belt tearing related text data is collected from the Internet and preprocessed, including UTF-8 encoding standardization, word segmentation into character units, and character units with similar semantics (such as "find" and "appear") are regarded as unique character units;

[0089] Count the occurrence frequencies of unique character units, remove unique character units with low occurrence frequencies, assign a unique integer index number to each of the remaining character units, and correspond the character units to the index numbers one by one;

[0090] Introducing classification tags <c0> , <c1> ,...， <c5>} to replace the category names {"rip open", "face hole", ..., "face scratch"}, and construct a mapping between category names and classification tags in the category set of the vocabulary, including {"rip open": <c0>,"With face hole": <c1>,"Belt Stacking": <c2>,"Foreign body penetration": <c3>,"Wire Leakage": <c4>,"Surface scratch": <c5>}.

[0091] This design overcomes the inefficiency of needing to predict multiple characters to achieve classification, and can directly predict shorter classification tokens, promoting efficient and accurate belt tearing object detection. The vocabulary is used in the subsequent language-to-text conversion module and the multimodal decoder.

[0092] Alternatively, if Figure 6 As shown, the language text conversion module is specifically used for:

[0093] After word segmentation, the language guidance text is searched and matched with the vocabulary, and the language guidance text is converted into a vector composed of index numbers;

[0094] The vector composed of index numbers is mapped to a denser serialized text vector through a word embedding model, wherein the word embedding model is a pre-trained Word2Vec model, which is composed of a one-hot encoding input layer, a fully connected hidden layer and a fully connected output layer. The input of the word embedding model is the language guidance text, and the output is the serialized text vector.

[0095] Alternatively, if Figure 7 As shown, the visual encoder E v , which is composed of the discrete state optional space model M stacked with N1 layers, extracts features from the serialized image vector, and the visual encoder E v The input is the serialized image vector, and the output is the visual modality feature x with the same size as the input v ;

[0096] like Figure 8 As shown, the text encoder E l , which is composed of the discrete state optional space model M stacked with N2 layers, performs feature extraction on the serialized text vector, and the text encoder E l The input is the serialized text vector, and the output is the text modality feature x with the same size as the input l .

[0097] Alternatively, if Fig. 9 As shown, the multimodal interactive fusion module T based on mask reconstruction vl , through the element-level masking operation, using the constraints of the loss function, the text features are reconstructed according to the masked visual features, and the visual features are reconstructed according to the masked text features, forcing the model to learn and understand the connection between the visual features and text features with semantic relevance, and realize more fine-grained multimodal feature fusion. The input is the visual modality feature x v and the text modality feature x l , the output is the multimodal fusion feature x mix , specifically used for:

[0098] Use the random seed generated with x v 、x l Visual mask vector m of the same shape v , text mask vector m l , m v and m l The vector elements of are all 0 or 1, 1 means no mask, 0 means mask;

[0099] The masked visual feature x is obtained by vector multiplication v =x v ×m v , masked text feature x l masked =x l ×m l T ;

[0100] The masked visual feature x v masked Through the weight-sharing feature reconstruction module R, the text modality feature reconstruction prediction result p is obtained l , the masked text feature x l masked Through the weight-sharing feature reconstruction module R, the visual modality feature reconstruction prediction result p is obtained v The feature reconstruction module R is composed of the discrete state optional space model M stacked with N3 layers and followed by a fully connected layer. The feature reconstruction module R is called twice, but the weights are shared between the two calls. One is to convert the x v masked After stacking N3 layers of the discrete state optional space model M and the fully connected layer, the p is obtained. l , once for the x l masked After stacking N3 layers of the discrete state optional space model M and the fully connected layer, the p is obtained. v , concatenate the features before passing through the fully connected layer twice to obtain the multimodal fusion feature x mix ;

[0101] In the training phase, two MSE loss functions are used to constrain the similarity between the reconstruction result and the original feature, and the gradient is back-propagated to update the weight of the feature reconstruction module R;

[0102] In the inference stage, the fully connected layer is deleted (since the fully connected layer in the feature reconstruction module R is only used for data dimensionality reduction to reduce the amount of calculation when calculating the MSE loss function, the fully connected layer can be deleted in the inference stage);

[0103] The two MSE loss function formulas are as follows, where n is the sequence length and x is vi 、p vi 、x li 、p li They are the features x v 、p v 、x l 、p l Elements in:

[0104]

[0105]

[0106] In the multimodal interactive fusion module based on mask reconstruction, the feature reconstruction module R is allowed to perform cross-modal reconstruction at a fine-grained level by masking the element level in the sequence features. During the training process, the reconstruction module will gradually improve its ability to understand the fine-grained and local information of visual modal features and textual modal features, thereby achieving fine-grained multimodal feature fusion and ultimately improving the performance of intensive prediction tasks such as target detection.

[0107] Alternatively, if Fig.10 As shown, the multimodal decoder D is composed of the discrete state optional space model M stacked with N4 layers, and the multimodal fusion feature x mix Decoding is performed, and the input of the multimodal decoder D is the x mix , the output is a text consisting of tear categories and detection box coordinates, which is specifically used for:

[0108] Initialize the process of generating text and set a start symbol <start>, indicating the start of text generation;

[0109] According to the x mix , predict the character units in the output text. When generating each character unit, it will first output an output vector consisting of multiple index numbers, then calculate the probability of each index number in the output vector through a Softmax layer to obtain a probability vector, and select the index number corresponding to the position with the highest probability in the probability vector to obtain the prediction result in the form of the index number of the current time step;

[0110] Repeat the above steps to gradually generate the prediction results in the form of index numbers for each time step until the end of the sentence is generated. <end>, get an index number vector consisting of index numbers;

[0111] Searching and matching corresponding character units in the vocabulary according to the index numbers in the index number vector, and splicing all character units together in order to obtain a character unit sequence Pred, that is, a complete text output;

[0112] During the training process, the output of the multimodal decoder D is constrained by the CE loss function, and the loss function formula is as follows:

[0113]

[0114] Among them, G is a character sequence consisting of the label L text corresponding to the input image I, and Pred i and G i Represent the elements in Pred and G respectively, C is the total number of predicted character units, and is also the length of the sequences Pred and G.

[0115] Optionally, the discrete state optional space model M can only accept serialized vector inputs, including but not limited to: serialized image vectors, serialized text vectors, visual modal features, text modal features, and multimodal fusion features, and can realize modeling of the mapping relationship between the input sequence and the output sequence. In the process of learning the input-output mapping, useful features in the input sequence (similar to convolution, self-attention and other mechanisms) are automatically learned to better represent the input sequence, and ultimately understand the patterns, structures and laws in the input data, and convert this information into correct outputs, thereby realizing many functions, including but not limited to: feature extraction, decoding and multimodal feature fusion, and is designed according to discrete linear state space modeling. The discrete linear state space modeling is first improved to discrete state optional space modeling, and then pre-processing, post-processing, and gated jump connection branches are added to the discrete state optional space modeling to obtain the discrete state optional space model M, such as Fig.11 As shown;

[0116] The discrete linear state space modeling comprises:

[0117] For a one-dimensional input sequence Through potential mapping N is the dimension of the corresponding hidden layer, and the sequence t is the sequence index or time step subscript, then the discrete linear state space modeling is expressed as:

[0118]

[0119] y t =Ch t

[0120]

[0121]

[0122] in, are all constant matrices independent of the input sequence, E is the unit matrix, Δ represents the change of the corresponding parameter, according to the above formula, any kth element y in the output sequence k It is expressed as:

[0123] where k = 1, 2, 3, ..., t

[0124]

[0125] It can be seen that the computational complexity of discrete linear state space modeling is In the case of long sequences, the computational complexity is much smaller than The self-attention calculation method.

[0126] In order to achieve efficient parallel computing, the output sequence y is converted into the form of convolution calculation. The formula is as follows:

[0127]

[0128]

[0129] in, is the tensor form of the input sequence, is the tensor form of the output sequence, is the convolution kernel;

[0130] The discrete linear state space modeling is improved to discrete state selectable space modeling, including:

[0131] Based on the output sequence y and the input sequence x, a discrete state optional space model is constructed by associating a learnable hyperparameter matrix with the input sequence index or time step C t Replace the constant matrix in the original discrete linear state space modeling C, during the training process, different weights are adaptively assigned to different elements in the input sequence, and irrelevant noise that may appear in the input sequence is assigned a lower weight, thereby achieving an optional effect on the input sequence, removing information redundancy in the input image, and modeling the truly effective semantic information in the input sequence. C t The specific implementation process is as follows:

[0132] The input sequence x is passed through three different linear mapping layers Linear to obtain the learnable intermediate variable hyperparameter matrix s B 、s C 、s Δ , the formula is as follows:

[0133] s B =Linear1 N (x)

[0134] s C =Linear2 N (x)

[0135] s Δ =Linear3 D (x)

[0136] Where N and D are the dimensions of the linear mapping layer, and the matrix B corresponding to the input sequence index or time step t is denoted as B t , there is B t =B t-1 +s B , the matrix C corresponding to the input sequence index or time step t is denoted as C t , there is C t =C t-1 +sc;

[0137] Calculate the change matrix Δ for the input sequence index or time step t t =μ+sΔ=μ+Linear(x)=Linear(x), where μ is a constant with an extremely small absolute value;

[0138] According to the matrix corresponding to the input sequence index or time step t and matrix The following formulas are derived respectively:

[0139]

[0140]

[0141] in is the Sigmoid Linear Unit (SiLU) activation function, which obtains a learnable hyperparameter matrix associated with the input sequence index or time step C t Then, substitute it into the discrete linear state space modeling to obtain the following formula:

[0142] y t =C t h t =Linear N (x t )h t

[0143]

[0144] g t =σ(Linear N (x t ))

[0145] where x t is the element in the input sequence x, y t is the element in the output sequence y, h t , g t is an intermediate variable;

[0146] During the training process, a batch learning strategy is (usually) used. Multiple single sequences of a batch size of b are input at the same time. Grouped convolution is used to achieve parallel computation of multiple sequence inputs in the newly added dimension. The input sequence and output sequence are converted from and becomes and Extending the discrete state optional space modeling from a single-input single-output system to a multi-input multi-output system, where b is the number of sequences of multiple single sequences input;

[0147] The pre-processing, the post-processing, and the gated jump connection branch are added to the discrete state selectable space modeling to obtain the discrete state selectable space model M, so as to effectively process complex sequence modeling tasks, including:

[0148] The pre-processing first reduces the dimension of the input sequence through a fully connected layer to reduce the computational complexity of the subsequent discrete state optional space modeling, and then connects a convolution and activation function for preliminary feature extraction and the introduction of nonlinearity to increase the learning ability of the model, and then uses the discrete state optional space modeling to model the input sequence after the pre-processing;

[0149] The post-processing includes a fully connected layer to perform sequence dimension upscaling on the output sequence of the discrete state optional space modeling back to the same dimension as the input sequence;

[0150] The gated jump connection branch consists of a fully connected layer for sequence dimensionality reduction and a SiLU activation function for controlling the weight of the jump connection, and the output range is between 0 and 1. The contribution of the gated jump connection is controlled by multiplying the output of the gated jump connection branch with the output of the discrete state optional space modeling. The gated jump connection can better control the flow and learning of information, enhance the expression and learning capabilities of the network, and improve the robustness and generalization ability of the model by introducing a gating function realized by the activation function in the jump connection.

[0151] In the inference stage, in order to facilitate users to observe the prediction results, the category and coordinates of the output text "(C,x1,y1,x2,y2)" are extracted, and the target detection box is visualized on the input image according to the upper left point P1(x1,y1) and the lower right point P2(x2,y2), and the category name is marked in the detection box.

[0152] S5. Use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected.

[0153] A belt tear detection method based on a discrete state optional space model is proposed in an embodiment of the present invention. In view of the fact that existing feature extraction algorithms are difficult to simultaneously meet the requirements of high-resolution long sequence processing, low computational complexity, and fast reasoning, a discrete state optional space model is used to achieve linear computational complexity, and a learnable selective hyperparameter is introduced to further extract truly effective semantic information in the input data, and provide features of two modal data for subsequent multimodal detection tasks. In view of the problem that the interactive fusion of two representation information under bimodal data is single and insufficient in the existing detection technology, an embodiment of the present invention proposes a multimodal interactive fusion technology based on mask reconstruction, introduces a mask reconstruction fusion method, and uses strict training tasks to force the model to understand the relationship between multimodal information, thereby achieving effective information fusion. In view of the requirements of open domain detection of belt tear, an effective language guidance text is designed for the detection task, and the prior knowledge contained in the language text is used to assist in the construction of a multimodal open domain detection framework, and the prediction of tear categories and detection target coordinates is achieved by specifying a standard text output format.

[0154] like Fig.12 As shown, an embodiment of the present invention further provides a belt tear detection system based on a discrete state selectable space model, the system comprising:

[0155] A collecting unit 1210 is used to collect a plurality of images of the belt running scene;

[0156] A labeling unit 1220 is used to label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into a training set and a validation set in proportion;

[0157] A preprocessing unit 1230 is used to preprocess the image data in the training set to expand the training set and obtain an input image I and a label L;

[0158] The training unit 1240 is used to train the belt tear open domain detection backbone network using the input image I and the input language guidance text to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D;

[0159] The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method;

[0160] The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary;

[0161] The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ;

[0162] The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ;

[0163] The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modality feature reconstruction prediction result p l , and output multimodal fusion features x mix ;

[0164] The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates;

[0165] The detection unit 1250 is used to use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected.

[0166] An embodiment of the present invention provides a belt tear detection system based on a discrete state selectable space model, and its functional structure corresponds to a belt tear detection method based on a discrete state selectable space model provided by an embodiment of the present invention, which will not be described in detail here.

[0167] Fig.13 It is a structural diagram of an electronic device 1300 provided in an embodiment of the present invention. The electronic device 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1301 and one or more memories 1302, wherein the memories 1302 store instructions, and the instructions are loaded and executed by the processor 1301 to implement the steps of the above-mentioned belt tear detection method based on the discrete state optional space model.

[0168] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned belt tear detection method based on a discrete state selectable space model. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / end> < / start> < / c1> < / c0> < / c> < / c>

Claims

1. A belt tear detection method based on a discrete state selectable space model, characterized in that: The method comprises: S1. Collect multiple images of the belt running site; S2, label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into training set and validation set in proportion; S3, preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L; S4, using the input image I and the input language guidance text to train the belt tear open domain detection backbone network, to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D; The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method; The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary; The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ; The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ; The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modal feature reconstruction prediction result p l , and output multimodal fusion features x mix ; The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates; S5. Use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected, wherein the belt tear open domain detection backbone network models different elements in the input sequence with different weights.

2. The method according to claim 1, characterized in that The cross-scanning image serialization module is specifically used for: Use convolution operation to cut the input image into a series of small image blocks; All small image blocks are rearranged by cross scanning in four different directions to form a long sequence of four image blocks. The four long sequences are added and averaged to obtain a new long sequence of image blocks. The new long sequence of image blocks is flattened into a one-dimensional vector to obtain the serialized image vector.

3. The method according to claim 1, characterized in that The template of the language guidance text is: Identify the objects belonging to the belt tear category in the corresponding input image and draw a detection box around the object. The output format should be a list of tuples in the format of (C,x1,y1,x2,y2), where C is the tear category name, x1, y1, x2, y2 are the coordinates of the upper left corner and lower right corner of the detection box, the x coordinate should be in the range of 2048, and the y coordinate should be in the range of 2000.

4. The method according to claim 1, characterized in that: The construction of the vocabulary specifically includes: A large amount of belt tearing related text data is collected from the Internet and preprocessed, including UTF-8 encoding standardization, word segmentation into character units, and treating character units with similar semantics as unique character units; Count the occurrence frequencies of unique character units, remove unique character units with low occurrence frequencies, assign a unique integer index number to each of the remaining character units, and correspond the character units to the index numbers one by one; Introducing classification tags <c0> , <c1> ,...， <c5>} to replace the category names {"rip open", "face hole", ..., "face scratch"}, and construct a mapping between category names and classification tags in the category set of the vocabulary, including {"rip open": <c0>,"With face hole": <c1>,"Belt Stacking": <c2>,"Foreign body penetration": <c3>,"Wire exposed": <c4>,"Surface scratch": <c5>}。< / c5> < / c1> < / c0> 5. The method according to claim 1, characterized in that The language text conversion module is specifically used for: After word segmentation, the language guidance text is searched and matched with the vocabulary, and the language guidance text is converted into a vector composed of index numbers; The vector composed of index numbers is mapped to a denser serialized text vector through a word embedding model, wherein the word embedding model is a pre-trained Word2Vec model, which is composed of a one-hot encoding input layer, a fully connected hidden layer and a fully connected output layer. The input of the word embedding model is the language guidance text, and the output is the serialized text vector.

6. The method according to claim 1, characterized in that The visual encoder E v , which is composed of the discrete state optional space model M stacked with N1 layers, extracts features from the serialized image vector, and the visual encoder E v The input is the serialized image vector, and the output is the visual modality feature x with the same size as the input v ; The text encoder E l , which is composed of the discrete state optional space model M stacked with N2 layers, performs feature extraction on the serialized text vector, and the text encoder E l The input is the serialized text vector, and the output is the text modality feature x with the same size as the input l .

7. The method according to claim 1, characterized in that The multimodal interactive fusion module T based on mask reconstruction vl , through the element-level masking operation, using the constraints of the loss function, the text features are reconstructed according to the masked visual features, and the visual features are reconstructed according to the masked text features, forcing the model to learn and understand the connection between the visual features and text features with semantic relevance, and realize more fine-grained multimodal feature fusion. The input is the visual modality feature x v and the text modality feature x l , the output is the multimodal fusion feature x mix , specifically used for: Use the random seed generated with x v 、x l Visual mask vector m of the same shape v , text mask vector m l , m v and m l The vector elements of are all 0 or 1, 1 means no mask, 0 means mask; The masked visual feature x is obtained by vector multiplication v masked =x v ×m v T , masked text feature x l masked =x l ×m l T ; The masked visual feature x v masked Through the weight-sharing feature reconstruction module R, the text modality feature reconstruction prediction result p is obtained l , the masked text feature x l masked Through the weight-sharing feature reconstruction module R, the visual modality feature reconstruction prediction result p is obtained v The feature reconstruction module R is composed of the discrete state optional space model M stacked with N3 layers and followed by a fully connected layer. The feature reconstruction module R is called twice, but the weights are shared between the two calls. One is to convert the x v masked After stacking N3 layers of the discrete state optional space model M and the fully connected layer, the p is obtained. l , once for the x l masked After stacking N3 layers of the discrete state optional space model M and the fully connected layer, the p is obtained. v , concatenate the features before passing through the fully connected layer twice to obtain the multimodal fusion feature x mix ; In the training phase, two MSE loss functions are used to constrain the similarity between the reconstruction result and the original feature, and the gradient is back-propagated to update the weight of the feature reconstruction module R; In the inference phase, the fully connected layer is deleted; The two MSE loss function formulas are as follows, where n is the sequence length and x is vi 、p vi 、x li 、p li They are the features x v 、p v 、x l 、p l Elements in:

8. The method according to claim 1, characterized in that The multimodal decoder D is composed of N4 layers of the discrete state optional space model M, and performs multimodal fusion feature x mix Decoding is performed, and the input of the multimodal decoder D is the x mix , the output is a text consisting of tear categories and detection box coordinates, which is specifically used for: Initialize the process of generating text and set a start symbol <start> , indicating the start of text generation;< / start> According to the x mix , predict the character units in the output text. When generating each character unit, it will first output an output vector consisting of multiple index numbers, then calculate the probability of each index number in the output vector through a Softmax layer to obtain a probability vector, and select the index number corresponding to the position with the highest probability in the probability vector to obtain the prediction result in the form of the index number of the current time step; Repeat the above steps to gradually generate the prediction results in the form of index numbers for each time step until the end of the sentence is generated. <end> , get an index number vector consisting of index numbers;< / end> Searching for the corresponding character unit in the vocabulary according to the index number in the index number vector, and splicing all the character units together in order to obtain a character unit sequence Pred, that is, a complete text output; During the training process, the output of the multimodal decoder D is constrained by the CE loss function, and the loss function formula is as follows: Among them, G is a character sequence consisting of the label L text corresponding to the input image I, and Pred i and G i Represent the elements in Pred and G respectively, C is the total number of predicted character units, and is also the length of the sequences Pred and G.

9. A belt tear detection system based on a discrete state selectable space model, characterized in that: The system comprises: a collection unit, a labeling unit, a preprocessing unit, a training unit, and a detection unit; Wherein, the collecting unit is used to collect multiple images of the belt running scene; A labeling unit is used to label the torn objects on each image to obtain an image-label pair, and divide all image-label pairs into a training set and a validation set in proportion; A preprocessing unit, used for preprocessing the image data in the training set to expand the training set, and obtaining an input image I and a label L; The training unit is used to train the belt tear open domain detection backbone network using the input image I and the input language guidance text to obtain a trained belt tear open domain detection backbone network, wherein the belt tear open domain detection backbone network is composed of multiple modules based on a discrete state optional space model M, including: a cross-scanning image serialization module, a language text conversion module, a visual encoder E v , Text Encoder E l , Multimodal interactive fusion module based on mask reconstruction T vl and a multimodal decoder D; The cross-scanning image serialization module is used to convert the input image into a serialized image vector by using a cross-scanning method; The language text conversion module is used to convert the input language guidance text into a serialized text vector using the constructed vocabulary; The visual encoder E v , used to extract features from the serialized image vector and output visual modality features x v ; The text encoder E l , used to extract features from the serialized text vector and output text modality features x l ; The multimodal interactive fusion module T based on mask reconstruction vl , used to obtain the visual modality feature reconstruction prediction result p using a mask-based reconstruction method v , text modal feature reconstruction prediction result p l , and output multimodal fusion features x mix ; The multimodal decoder D is used to decode the multimodal fusion feature x mix Decode and output text consisting of tearing categories and detection box coordinates; A detection unit is used to use the trained belt tear open domain detection backbone network to detect the image of the belt running scene to be detected, and output the tear detection result of the image of the belt running scene to be detected, wherein the belt tear open domain detection backbone network models different elements in the input sequence with different weights.

Images