A conveyor belt crack detection method and system based on multi-modality

Abstract

The present invention proposes a conveyor belt crack detection method and system based on multimodality, which belongs to the field of target detection. The method comprises: using a camera, a first detection model and manual marking to collect a conveyor belt abnormal data set; when the conveyor belt abnormal data set reaches a preset scale threshold, based on the data set, a second detection model is established and trained after a multimodal model is used as a base and a language model is introduced; the second detection model is used to replace the first detection model, and the above process is repeated. The present invention realizes the effective collection of conveyor belt abnormality data, and at the same time, utilizes multimodal technology to connect image data and fine-grained labels through text semantics to realize the prediction and detection tasks of conveyor belt cracks, so that timely repairs can be made to avoid material leakage or pollution and protect the environment.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular, relates to a conveyor belt crack detection method and system based on multi-modality. Background Art

[0002] As a key component connecting different processes, the stability and reliability of the conveyor belt are crucial to ensure the efficient operation of the production line. Long-term continuous operation, heavy loads and harsh working environments can cause wear and tear on the conveyor belt, especially cracks. Once cracks appear on the conveyor belt, if they cannot be detected and repaired in time, it will not only cause equipment downtime and affect production efficiency, but may also cause serious safety accidents. For example, in harsh environments such as mines and tunnels, cracks in the conveyor belt may cause leakage or contamination of materials such as cement, affecting the surrounding environment. Therefore, it is of great practical significance to develop an efficient, accurate and more refined conveyor belt crack detection technology.

[0003] Traditional manual inspections are difficult to ensure timely detection, especially in harsh environments such as mines and tunnels. This method can no longer meet actual needs. Some existing intelligent detection methods based on artificial intelligence methods can only determine whether there are cracks but cannot give the extent of the cracks. When such methods detect cracks, the conveyor belt often has very obvious cracks, which will seriously threaten production safety.

[0004] In recent years, the Transformer model has achieved great success in the fields of natural language processing and computer vision, which has promoted the development of multimodal models. Transformer-based multimodal models such as CLIP (Contrastive Language-Image Pre-Training) and VLM (Vision-and-Language Model) have significantly improved the performance of multimodal tasks by pre-training a large number of image-text combinations, and have demonstrated strong generalization and zero-sample capabilities in the field of vision.

[0005] In this context, the present invention proposes a conveyor belt crack detection method and system based on multi-modality. Summary of the invention

[0006] The present invention aims to overcome the shortcomings of the prior art and proposes a conveyor belt crack detection method and system based on multimodality to achieve the following objectives: to achieve effective collection of conveyor belt abnormal data, and at the same time to utilize multimodal technology to connect image data and fine-grained labels through text semantics to achieve prediction and detection tasks.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A conveyor belt crack detection method based on multi-modality, the method comprising:

[0009] Step S1, using a camera, a first detection model and manual marking to collect a conveyor belt abnormality data set;

[0010] Step S2: When the conveyor belt abnormality data set reaches a preset scale threshold, based on the data set, a second detection model is established and trained after a multimodal model is used as a base and a language model is introduced;

[0011] Step S3, use the second detection model to replace the first detection model, and repeat steps S1, S2, and S3.

[0012] Preferably, the step S1 comprises:

[0013] Step S11, arranging n cameras at multiple nodes of the conveyor belt to take pictures, setting the shooting time interval to t, and uploading all the taken pictures and corresponding information to the server, wherein the corresponding information includes a timestamp and a location;

[0014] Step S12: The server uses the first detection model to roughly detect whether there are cracks in the image;

[0015] Step S13: When an obvious crack is found, the server gives a signal and sends all the images stored at the location and the corresponding timestamps to manual processing. The manual process determines whether there are cracks and the degree of cracks in all the images stored at the location. The degree of cracks and timestamps in each image are used as two text labels, which together with the corresponding images constitute a conveyor belt anomaly data set.

[0016] Step S14: Check and repair the cracks at the location, reset the pictures stored at the location, and continue monitoring and shooting.

[0017] Preferably, the crack degree labels include complete, microscopic defects, hairline cracks, initial cracks, small cracks, medium cracks, severe cracks, and breaks.

[0018] Preferably, the first detection model adopts a YOLO detection model and is pre-trained using a public crack dataset.

[0019] Preferably, step S11 includes: for image data at a fixed position on a conveyor belt, the server is configured to only retain image data collected from the position from P+α time before the current moment, where P is the advance warning time preset by the server, and α is the preset reserved time.

[0020] Preferably, the second detection model uses CLIP as a multimodal model and introduces a language model for fine-tuning training.

[0021] Preferably, the step S2 comprises:

[0022] Step S21, introducing an adapter structure into CLIP to fine-tune model parameters, and freezing the original parameters of CLIP;

[0023] Step S22: introducing a language model based on CLIP to form a CLIP-LM model as a second detection model, wherein CLIP is used to receive image data and the language model is used to output text results;

[0024] Step S23: using the Lora structure for fine-tuning training of the language model;

[0025] Step S24: performing joint fine-tuning training on the CLIP-LM model on the conveyor belt anomaly dataset.

[0026] Preferably, in step S24, the joint fine-tuning training method is as follows:

[0027] Description of the degree of cracks in the conveyor belt , first through the word segmentation Vectorize it and get ,Right now;

[0028] ;

[0029] in, , is the vector length. CLIP uses ; Then use CLIP's text encoder ,right To encode:

[0030] ;

[0031] in, , is the feature dimension; The description of the degree of cracks in the conveyor belts of different types is combined into a classification matrix. Behavior, namely:

[0032] ;

[0033] Among them, the classification matrix ;

[0034] For 1 picture data in the dataset , C represents dimension, H represents height, W represents width, after data enhancement:

[0035] ;

[0036] Convert it to , represents the height after data enhancement, Indicates the width after data enhancement;

[0037] Encode using the CLIP image encoder:

[0038] ;

[0039] in, , is the feature dimension; the obtained image features and the classification matrix Multiply the product to get the judgment score of the crack category , used to calculate the crack classification loss;

[0040] Will Input to the vision-language Adapter module and get:

[0041] ;

[0042] Set prompt text for the language model and pass it through the word segmenter in turn After CLIP text encoder, we get ;

[0043] Will , and The data is input into the language model, and the crack judgment results are recalibrated through the language model to predict whether cracks will appear in the future. The loss of the language model results and labels is calculated, and the loss function is combined to optimize the model training.

[0044] Preferably, the loss function adopts a prediction score cross entropy loss function and a language model loss function, wherein:

[0045] The prediction score cross entropy loss function is:

[0046] ;

[0047] in, It is for The predicted scores of the cracks; It is The labels of the samples;

[0048] The language model loss function consists of two parts: the multi-classification loss of the crack degree and the binary classification loss of predicting whether cracks will occur in the future, namely:

[0049] ;

[0050] in, is a one-hot tag. is the prediction score of the language model for the i-th label; A prediction score for the language model to predict whether a crack will occur in the future;

[0051] The final loss function is:

[0052] ;

[0053] in, , which is used to control the proportion of the prediction score cross entropy loss function and the language model loss function.

[0054] At the same time, the present application also proposes a multimodal conveyor belt crack detection system, the system includes a camera and a server, wherein the camera is used to collect conveyor belt crack pictures, and the server is used to implement the above-mentioned multimodal conveyor belt crack detection method by executing a program.

[0055] The technical effects of the present invention are:

[0056] (1) The present invention proposes an abnormal data collection method, which uses the YOLO detection model with only detection function to effectively collect the data required for the conveyor belt anomaly detection task and various types of text labels with little human intervention.

[0057] (2) The multimodal conveyor belt crack detection method provided by the present invention uses text and visual (image or video) two-modal data for detection and identification. The application of this multimodal technology can effectively distinguish and detect fine-grained conveyor belt abnormalities, such as micro defects, fine cracks, initial cracks, small cracks, medium cracks, severe cracks, fractures and other degrees of cracks as well as other types of abnormalities.

[0058] (3) By learning from historical data with text labels, the present invention can predict possible failures within a certain fault tolerance rate and take preventive measures in advance, thereby avoiding high maintenance costs caused by sudden failures. This predictive maintenance strategy can reduce the cost of unnecessary inspections and replacement of parts.

[0059] (4) The present invention can timely detect cracks in the conveyor belt, ensuring the normal and orderly transportation production, while avoiding problems such as material leakage and pollution caused by cracks in the conveyor belt, which is beneficial to protecting the environment and ecology. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1A flow chart of a conveyor belt crack detection method based on multi-modality according to an embodiment of the present invention;

[0061] Figure 2 This is a flow chart of a method for collecting abnormal data of a conveyor belt according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following is a further detailed description of the specific implementation of the present invention through the description of the embodiments with reference to the accompanying drawings, the purpose of which is to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation. It should be noted that the words "first", "second" and the like described in this application are only for the convenience of describing the technical solution to distinguish different components, and are not used to limit this application. In order to make the technical solution of the present invention clearer, the present invention is explained through the following embodiments.

[0063] The present invention provides a conveyor belt crack detection method based on multi-modality, such as Figure 1 As shown, including:

[0064] Step S1: Collecting a conveyor belt abnormality data set using a camera, a first detection model, and manual marking.

[0065] Specifically, the conveyor belt abnormality data collection method of the present invention is as follows: Figure 2 As shown, including:

[0066] Step S11, arrange n cameras at multiple key nodes of the conveyor belt for shooting, set the shooting time interval to t (a camera shoots the same position on the conveyor belt every t time), and upload all the taken pictures and corresponding information to the server, and the corresponding information includes timestamp and position. At the same time, in order to reduce the storage pressure of the server, this embodiment also sets the server, that is, for the picture data at a fixed position on the conveyor belt, the server is set to: only retain all the picture data collected from the current moment P+α time before the current moment, and the picture data before P+α time is deleted regularly. P is the advance warning time preset by the server, and α is the preset reserved time (buffer time). For example, P is 2 hours, α is 1 hour, and assuming that the time at the location is 4 o'clock, the server only retains the picture data of the location after 1 o'clock.

[0067] Step S12: The server uses a first detection model to roughly detect whether there are cracks in the image. In this embodiment, the first detection model uses a YOLO detection model, which is pre-trained with a public crack dataset and then deployed in the server.

[0068] Step S13, when the yolo detection model finds obvious cracks, the server gives a signal and sends all the images stored at that location and the corresponding timestamps to manual classification and labeling. After receiving the data, the manual determines whether there are cracks in all the images stored at that location in turn, and when there are cracks, determines the degree of the cracks and classifies them. As the timestamp moves toward the historical direction, the degree of the cracks should become lighter and lighter. Among them, the degree of cracks and timestamps in each picture are used as two text labels, which together with the corresponding pictures constitute the conveyor belt anomaly data set. The crack degree labels can be divided into fine-grained categories in advance according to safety specifications to form fine-grained labels, including complete, microscopic defects, fine cracks, initial cracks, small cracks, medium cracks, severe cracks, fractures, etc., so as to effectively distinguish the abnormal conditions of the conveyor belt with the help of more fine-grained detection.

[0069] Furthermore, when there are no cracks in the picture in the data and the timestamp is within the early warning time P, the data can be monitored and marked as "early prediction data". The marked data can be used for subsequent model training to improve the early warning capability of the model.

[0070] Step S14: Check and repair the cracks at the location, reset the pictures stored at the location, and continue monitoring and shooting.

[0071] Step S2: When the conveyor belt abnormality data set reaches a preset scale threshold, based on the data set, a second detection model is established and trained after the multimodal model is used as the base and the language model is introduced. In this embodiment, the second detection model uses CLIP as the multimodal model and introduces the language model (LM) for fine-tuning training. Specifically, step S2 of this embodiment includes:

[0072] Step S21: Introduce an adapter structure inside CLIP for fine-tuning model parameters and freeze the original CLIP parameters. Adapter is a method for fine-tuning models with high efficiency, which is widely used in Transformer-based neural networks and belongs to the existing technology. This application will not go into details.

[0073] Step S22: introducing a language model based on CLIP to form a CLIP-LM model as a second detection model, wherein CLIP is used to receive image data, and the language model is used to output text results.

[0074] Step S23, using the Lora structure for fine-tuning the language model; Lora is a low-rank adaptation technology in the prior art for fine-tuning large language models, which freezes the model weights of the original model pre-training and injects the trainable rank decomposition matrix into the original model, thereby achieving model fine-tuning. The application of the Lora structure can also effectively improve the training efficiency of the model.

[0075] Specifically, assuming that the parameters of the language model are , the parameter updated each time is For large language models, d and k represent the dimensions of input and output, which are usually very large numbers. Lora uses two low-rank matrices to Perform an approximate decomposition. That is, each update becomes:

[0076] ;

[0077] in, , . Freeze parameters during training , training only and This operation can reduce the time and resources required for model training.

[0078] Step S24: performing joint fine-tuning training on the CLIP-LM model on the conveyor belt anomaly data set. The joint fine-tuning training method based on the CLIP-LM model is as follows:

[0079] The model receives data from a multimodal conveyor belt anomaly dataset. Each piece of data contains an image, a timestamp label, and a crack degree label. , first through the word segmentation Vectorize it and get ,Right now:

[0080] ;

[0081] in, , is the vector length. CLIP uses ; Then use CLIP's text encoder ,right To encode:

[0082] ;

[0083] in, , is the feature dimension; The description of the degree of cracks in the conveyor belts of different types is combined into a classification matrix. Behavior, namely:

[0084] ;

[0085] Among them, the classification matrix ;

[0086] For 1 picture data in the dataset , C represents dimension, H represents height, W represents width, after data enhancement:

[0087] ;

[0088] Convert it to , represents the height after data enhancement, Indicates the width after data enhancement;

[0089] Encode using the CLIP image encoder:

[0090] ;

[0091] in, , is the feature dimension; the obtained image features and the classification matrix Multiply the product to get the judgment score of the crack category , used to calculate the crack classification loss .

[0092] In order to make the language model able to receive visual representations, the vision-language Adapter module is introduced. This module consists of MLP (Multilayer Perceptron, a feed-forward artificial neural network model), which is responsible for aligning the visual-text space and connecting the visual encoder and the language model. Input to the vision-language Adapter module:

[0093] ;

[0094] Then, a prompt text is set for the language model. In this embodiment, it is set to: "This is an encoding vector of a conveyor belt image. Please determine whether there is a crack. If so, please determine the degree of the crack and select an appropriate crack label from the following set ({no crack}, {slight gap}..., {break}); if not, please predict whether a crack will appear in the next P time."

[0095] Then, the prompt text is passed through the word segmenter in turn. After using the CLIP text encoder, we get ;

[0096] Will , and The result is input into the language model, and the crack judgment result is recalibrated through the language model to predict whether a crack will appear in the future P time. The loss is calculated for the result and label of the language model, and the loss function is combined to optimize the model training.

[0097] The loss function adopts the prediction score cross entropy loss function and the language model loss function, where:

[0098] The prediction score cross entropy loss function is:

[0099] ;

[0100] in, It is for The predicted scores of the cracks; It is The labels of the samples;

[0101] The language model loss function consists of two parts: the multi-classification loss of the crack degree and the binary classification loss of predicting whether cracks will occur in the future, namely:

[0102] ;

[0103] in, is a one-hot tag. is the prediction score of the language model for the i-th label; ; The prediction score for the language model to predict whether a crack will occur in the next P time;

[0104] The final loss function is:

[0105] ;

[0106] in, , which is used to control the proportion of the prediction score cross entropy loss function and the language model loss function.

[0107] In actual application, only the image data (including image, label) and prompt text need to be input into the model side for forward propagation calculation, without calculating loss function and back propagation. The judgment or prediction result is output by the language model, giving a real-time warning of the conveyor belt.

[0108] Step S3, replace the first detection model with the second detection model, and repeat the steps S1, S2, and S3. That is, multiple cameras are deployed to positions where they can capture key parts of the conveyor belt, and still take pictures at given time intervals and transmit them to the server. The server predicts through the second detection model, and gives an alarm when a crack is found; and processes all the data at this position before the crack is found, and places it in the data set as subsequent training data, forming a closed loop of data collection-training-detection.

[0109] The present invention can timely discover and predict cracks in the conveyor belt, ensure the normal and orderly transportation production, and avoid problems such as material leakage and pollution caused by cracks in the conveyor belt, which is beneficial to protecting the environment and ecology.

[0110] At the same time, the present application also proposes a multimodal conveyor belt crack detection system, the system comprising a camera and a server, wherein the camera is used to collect conveyor belt crack images, and the server is used to implement the above-mentioned multimodal conveyor belt crack detection method by executing a program.

[0111] The present invention is described above by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention; or the above concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A conveyor belt crack detection method based on multi-modality, characterized in that: The method comprises: Step S1, using a camera, a first detection model and manual marking to collect a conveyor belt abnormality data set; Step S2: When the conveyor belt abnormality data set reaches a preset scale threshold, based on the data set, a second detection model is established and trained after a multimodal model is used as a base and a language model is introduced; the second detection model uses CLIP as a multimodal model and introduces a language model for fine-tuning training. Specifically, step S2 includes: Step S21, introducing an adapter structure into CLIP to fine-tune model parameters, and freezing the original parameters of CLIP; Step S22: introducing a language model based on CLIP to form a CLIP-LM model as a second detection model, wherein CLIP is used to receive image data and the language model is used to output text results; Step S23: using the Lora structure for fine-tuning training of the language model; Step S24: performing joint fine-tuning training on the CLIP-LM model on the conveyor belt anomaly data set; wherein the joint fine-tuning training method is as follows: For the conveyor belt crack degree description text, first use the tokenizer to vectorize it to get text_token, that is: text_token = tokenizer(text) Where text_token∈R v , v is the vector length, CLIP uses v=77 by default; then use CLIP's text encoder encoder text , encode text_token: text_emb=encoder text (text_token) Where text_emb∈R 1×d , d is the feature dimension; the descriptions of the crack degree of N types of conveyor belts are combined into a classification matrix, and the i-th line of the matrix is: cls_matrix i =text_emb i Among them, the classification matrix cls matrix ∈R N×d ,i∈{1,…,N}; For one image data I∈R in the dataset C×H×W , C represents dimension, H represents height, W represents width, after data enhancement: I aug =augment(I) Convert it to I aug ∈R C×H′×W′ , H' represents the height after data enhancement, and W' represents the width after data enhancement; Encode using the CLIP image encoder: img_emb clip =encoder vis (I aug ) Among them, img_emb clip ∈R d , d is the feature dimension; the obtained image feature img_emb clip and the classification matrix cls matrix Multiply the product to get the judgment score logits∈R of the crack category N , used to calculate the crack classification loss; img_emb clip Input into the vision-languageAdapter module and get: img_emb adapter =Adapter v-l (img_emb clip ); Set prompt text prompttext for the language model, and pass it through the tokenizer and CLIP text encoder in turn to get text_emb prompt ; text_emb prompt 、img_emb adapter The logits are input into the language model, the crack judgment result is recalibrated through the language model, and whether cracks will appear in the future is predicted; the loss of the language model result and the label is calculated, and the model training optimization is performed with the joint loss function; the loss function adopts the prediction score cross entropy loss function and the language model loss function, where: The prediction score cross entropy loss function is: Among them, logits i is the prediction score for the i-th crack; y i is the label of the i-th sample; The language model loss function consists of two parts: the multi-classification loss of the crack degree and the binary classification loss of predicting whether cracks will occur in the future, namely: Among them, y,t is a one-hot label, clslogits i is the prediction score of the language model for the i-th label; j∈{0,1}; timelogits j A prediction score for the language model to predict whether a crack will occur in the future; The final loss function is: Among them, 0<α<1 is used to control the proportion of the prediction score cross entropy loss function and the language model loss function; Step S3, use the second detection model to replace the first detection model, and repeat steps S1, S2, and S3.

2. The multi-modal conveyor belt crack detection method according to claim 1, characterized in that: The step S1 comprises: Step S11, arranging n cameras at multiple nodes of the conveyor belt to take pictures, setting the shooting time interval to t, and uploading all the taken pictures and corresponding information to the server, wherein the corresponding information includes a timestamp and a location; Step S12: The server uses the first detection model to roughly detect whether there are cracks in the image; Step S13: When an obvious crack is found, the server gives a signal and sends all the images stored at the location and the corresponding timestamps to manual processing. The manual process determines whether there are cracks and the degree of cracks in all the images stored at the location. The degree of cracks and timestamps in each image are used as two text labels, which together with the corresponding images constitute a conveyor belt anomaly data set. Step S14: Check and repair the cracks at the location, reset the pictures stored at the location, and continue monitoring and shooting.

3. The multi-modal conveyor belt crack detection method according to claim 2, characterized in that: The crack degree labels include complete, microscopic defects, hairline cracks, initial cracks, small cracks, medium cracks, severe cracks, and breaks.

4. A conveyor belt crack detection method based on multi-modality according to claim 1, 2 or 3, characterized in that: The first detection model adopts the Yolo detection model and is pre-trained using a public crack dataset.

5. A conveyor belt crack detection method based on multi-modality according to claim 2 or 3, characterized in that: The step S11 includes: for the image data at a fixed position on a conveyor belt, the server is configured to only retain the image data collected from the position from P+α time before the current moment, where P is the advance warning time preset by the server, and α is the preset reserved time.

6. A conveyor belt crack detection system based on multi-modality, characterized by: The system includes a camera and a server, wherein the camera is used to collect images of conveyor belt cracks, and the server is used to implement a multi-modal conveyor belt crack detection method according to any one of claims 1 to 5 by executing a program.

Images