A multi-modal based output tape break detection algorithm

Abstract

A kind of output belt breakage detection algorithm based on multi-modal, comprising the following steps: S1, establishing contrast learning model, the model includes picture training model and sound training model;S2, on the basis of contrast learning model determined in step S1, the classification model is trained in the manner of finetune;S3, prediction stage, set prediction confidence threshold, picture collector and sound collector are arranged in the fixed position of conveying belt respectively to extract the picture and corresponding sound information on the conveying belt, the sound extraction frame is the sequence information of the set time before and after the picture, then the input information of the two is obtained by step S1 and step S2 Final prediction result of conveying belt breakage.The method in the application can monitor guide roller in real time, when guide roller is stuck or its rotation is abnormal, staff can find in time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of output belt breakage detection, and particularly relates to an output belt breakage detection algorithm based on multi-modal. BACKGROUND

[0002] The breakage detection of the output belt is an important factor in the safe use of the output belt. In order to detect the breakage of the output belt, the prior art has:

[0003] 1. (201510392423.7) A conveying belt tear detection method based on machine vision

[0004] The patent discloses a conveying belt tear detection method based on machine vision, which comprises the following steps: firstly, collecting real-time images, obtaining a gray image, detecting the edges of objects in the gray image by using an edge detector, then filtering the edge image based on a shape and direction filter template to obtain a straight edge image in the same direction as the conveying belt, marking the edges by using a one-dimensional vector, and determining whether the conveying belt is torn according to the known reference width and design width threshold of the conveying belt when the conveying belt is not torn. The method detects the tear of the conveying belt based on the traditional image detection algorithm. The method only detects the tear of the conveying belt at a fixed angle and detects the tear according to the edges of the conveying belt, and cannot detect the tear of the center part of the conveying belt, so that most of the tear conditions of the center part of the conveying belt cannot be recognized.

[0005] 2. (201510469098.X) Automatic detection method for surface breakage of conveying belt

[0006] The invention discloses an automatic detection method for surface breakage of a conveying belt. The conveying belt running image is collected by using a linear array camera, the consistency of the light sensed by the columns of the image is used to establish a suppression row vector, the column vector mean curve is used for edge fidelity denoising, the breakage characteristic parameters are extracted, and the surface breakage is quickly identified. The method adopts the traditional image feature extraction method, can only detect edge breakage, cannot detect different conveying belts at different angles, and has certain limitations. The patent only collects features through image information, and cannot extract features according to the noise generated by the wear of the conveying belt to better identify the wear of the conveying belt.

[0007] 3. (202011171382.9) A method for detecting the wear of a conveying belt based on a detection system and the conveying belt

[0008] The method discloses a detection conveying belt wear detection method and a conveying belt based on a detection system, relates to the technical field of material conveying, and the conveying belt comprises a first conveying device and a second conveying device, and the first conveying device and the second conveying device are reversely connected through a rotating shaft assembly; wherein the first conveying device comprises a first conveying section and a second conveying section, a first protective plate is arranged on the front and rear side walls of the first conveying section, the second conveying section is connected with the second conveying device, a second protective plate is arranged on the front and rear side walls of the second conveying device, and the length of the second protective plate is less than the length of the second conveying section. SUMMARY

[0009] To solve the problems in the background art, for this purpose, the application provides an output belt breakage detection algorithm based on multi-modal, and the specific scheme is as follows:

[0010] An output belt breakage detection algorithm based on multi-modal, comprising the following steps:

[0011] S1, a contrast learning model is established, the model comprises a picture training model and a sound training model, and the specific position is:

[0012] S11, collect the collection and shooting pictures of the conveying belt at different moments, and at the same time, the sound information around the picture collector before and after the corresponding pictures is also collected, and the picture information and the corresponding sound information are taken as matching pairs as training samples of the model;

[0013] S12, the collected pictures are input into a picture feature extraction module for picture feature extraction, and PANNs and EAT-M two kinds of speech feature extraction modes are selected to collect sound information;

[0014] S2, on the basis of the contrast learning model determined in step S1, the classification model is trained in a finetune manner;

[0015] S3, in the prediction stage, a prediction confidence threshold is set, the picture collector and the sound collector are arranged at fixed positions of the conveying belt, and the pictures on the conveying belt and the corresponding sound information are extracted, the sound extraction frame is the sequence information of the pictures before and after the set time, and then the input information of the two is input into steps S1 and S2 to obtain the final prediction result of the conveying belt breakage.

[0016] Specifically, the picture feature extraction step in step S12 is as follows:

[0017] SA121, the input picture is respectively subjected to 1x1 convolution to obtain a feature F1, and is input into a dilated convolution module with a dilated rate of 6, 12 and 18 and a convolution kernel size of 3x3 to obtain corresponding features F2, F3 and F4, and the picture is also subjected to 3x3 convolution, pooling operation and deconvolution operation to obtain a feature F5; the dilated convolution module comprises a dilated convolution, a BatchNorm and a corresponding GeLu activation operation function arranged in series;

[0018] SA122, the features F1, F2, F3, F4 and F5 are subjected to a concate module to obtain a feature map F c ;

[0019] SA123, the input picture is subjected to a patch division module and combined with the feature map F c After the concate module is combined, a reshape module and a self-attention encoder module of a Transformer model are arranged in series to obtain a feature F t ;

[0020] SA124, the feature F t is input into a sequencepooling module to obtain a feature F s ;

[0021] SA125, the feature F s is input into a CBAM module to obtain a feature F cbam ; the CBAM module is composed of a Channel Attention Module and a Spatial AttentionModule arranged in series;

[0022] SA136, the feature F cbam is input into a global pooling operation to obtain a final picture feature F image through a last three-layer MLP layer arranged in series; the activation function between the first two MLP layers is ReLu, and the activation function between the second and third MLP layers is SiLu.

[0023] Specifically, the sound extraction step in step S12 is as follows:

[0024] SB121, through two speech feature extraction methods of PANNs and EAT-M, features F panns and F eatm before the classification head of the two are extracted, respectively;

[0025] SB122, the features F panns and F eatmFusion, and then the output feature F is obtained through two-layer MLP voice The two-layer MLP is directly connected through a GELU activation function.

[0026] Specifically, step S2 is specifically: using the pre-training model of the contrast learning in step S1, freezing the weights of the picture feature encoding and the sound feature encoding in the model, inputting the obtained sound signal and picture signal into the corresponding encoder to obtain the sound and picture features, then connecting the features of the two through concate and obtaining attention feature information through SpatialAttention Module, outputting the attention feature information to the two-layer MLP module, connecting the two-layer MLP module through SiLu activation function, and finally outputting the binary classification output prediction result through sigmoid, which is divided into two categories of wear and no wear.

[0027] Specifically, the cross-entropy loss function is used for comparison calculation and classification with the real label in step S2.

[0028] Specifically, the prediction confidence threshold is set to 0.75.

[0029] Specifically, in step S1, if the anchor picture or its augmented version and the corresponding sound and its augmented sound are considered as positive sample pairs, the corresponding label is True, otherwise if the obtained picture and sound signal are non-matching pairs, they are considered as negative sample pairs, and the corresponding label is False.

[0030] The present application has the following advantages:

[0031] Compared with the prior art, the method in the present application can monitor the guide roller in real time, and when the guide roller is stuck or its rotation is abnormal, the staff can find it in time. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A structure diagram of a picture training model in an output belt damage detection algorithm based on multi-modal is provided.

[0033] Figure 2 A structure diagram of a sound training model in an output belt damage detection algorithm based on multi-modal is provided.

[0034] Figure 3 A structure diagram of a classification model in step S2 of an output belt damage detection algorithm based on multi-modal is provided. DETAILED DESCRIPTION

[0035] An output belt damage detection algorithm based on multi-modal includes the following steps: An output belt damage detection algorithm based on multi-modal includes the following steps:

[0036] S1, a contrast learning model is established, the model includes a picture training model and a sound training model, and the specific steps are as follows:

[0037] S11, collect the collection and shooting pictures of the conveying belt at different times, and at the same time, the sound information around the picture collector of the conveying belt 5 seconds before and after the corresponding picture is also collected, and the picture information and the corresponding sound information are taken as matching pairs as training samples of the model; specifically, the training samples include positive samples and negative samples, the original picture is set as an anchor picture, the picture positive sample selects the rotation, scaling, crop, cutcrop, color augmentation operation on the anchor picture. The corresponding sound signal adopts the augmentation method of adding different degrees of noise to obtain the positive sample of the sound. For the negative sample, the sound signal and its augmented signal of the non-anchor picture are used.

[0038] S12, input the collected pictures into the picture feature extraction module for picture feature extraction, and select PANNs and EAT-M two kinds of speech feature extraction methods to collect sound information;

[0039] As shown in Figure 1 , the picture feature extraction steps are as follows:

[0040] SA121, the input picture is respectively subjected to 1x1 convolution to obtain a feature F1, and is simultaneously input into a hollow convolution module with a hollow rate of 6, 12 and 18 and a convolution kernel size of 3x3 to obtain corresponding features F2, F3 and F4, and the picture is also subjected to 3x3 convolution and pooling operation and then subjected to deconvolution operation to obtain a feature F5; the hollow convolution module includes a hollow convolution, a BatchNorm and a corresponding GeLu activation operation function arranged in series;

[0041] SA122, the features F1, F2, F3, F4 and F5 are subjected to a concate module to obtain a feature map F c ;

[0042] SA123, the input picture is subjected to a patch partition module and then combined with the feature map F c after the concate module, and then subjected to a reshape module, a self-attention encoder module of a Transformer model in series to obtain a feature F t ;

[0043] SA124, the feature F t is input into a sequencepooling module to obtain a feature F s ;

[0044] SA125, the feature F sInput the feature F into the spatial attention and channel attention CBAM module cbam The spatial attention and channel attention CBAM module is composed of a Channel Attention Module and a Spatial Attention Module in series.

[0045] SA136, the feature F cbam is input into a global pooling operation, and the final picture feature F is obtained through the last three-layer MLP layer in series. image The activation function between the first two MLP layers is ReLu, and the activation function between the second and third MLP layers is SiLu.

[0046] As shown in Figure 2 , the sound extraction step is as follows:

[0047] SB121, through the PANNs and EAT-M two speech feature extraction methods, respectively extract the feature expression layer before the classification head of the two, the feature F panns and the feature F eatm ;

[0048] SB122, by adding the feature F panns and the feature F eatm , and then through two-layer MLP to obtain the output feature F voice , the two-layer MLP is directly connected through the GELU activation function.

[0049] During training, if the anchor picture or its augmentation and the corresponding sound and its augmented sound are considered as positive sample pairs, and the corresponding label is True, otherwise if the assigned picture and sound signal are not matched pairs, it is considered as negative sample pair, and the corresponding label is False. The loss function used in training is the cross-entropy loss function for training.

[0050] As shown in Figure 3As shown, S2, on the basis of the contrast learning model determined in step S1, the classification model is trained in a fine-tune manner. The specific steps are as follows: using the pre-training model of contrast learning in step S1, the weights of the picture feature encoding and the sound feature encoding in the model are frozen respectively, the sound signal and the picture signal obtained are input into the respective encoder to obtain the sound and picture features, then the features of the two are connected through concate and the attention feature information is obtained through Spatial Attention Module, the attention feature information is output to the two-layer MLP module, the two-layer MLP module is connected through SiLu activation function, and finally the output is obtained through sigmoid to obtain the output prediction result of binary classification, which is divided into two categories of wear and no wear, and the real label is compared to calculate the classification cross-entropy loss function.

[0051] S3, in the prediction stage, a prediction confidence threshold is set, the picture collector and the sound collector are arranged at fixed positions of the conveying belt to extract the picture and the corresponding sound information on the conveying belt, and the sound extraction frame is the sequence information of the set time before and after the picture. Then the input information of the two is input into step S1 and step S2 to obtain the final prediction result of the conveying belt damage. Here, the prediction confidence threshold is set to 0.75, and the set time before and after is 5s. If the confidence of the predicted damage is greater than 0.75, it is considered that the conveying belt at this position is damaged, and an alarm needs to be sent to remind the staff to check.

[0052] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-modal based output belt break detection algorithm, characterized in that, The method comprises the following steps: S1, a contrast learning model is established, the model comprising a picture training model and a sound training model, and the specific position is: S11, collecting pictures of a conveying belt at different times, and collecting sound information around the picture collector at the same time, and taking the picture information and the corresponding sound information as a matching pair as a training sample of the model; S12, inputting the collected pictures into a picture feature extraction module for picture feature extraction, and selecting PANNs and EAT-M two kinds of speech feature extraction methods to collect sound information; S2, training the classification model in a finetune manner on the basis of the contrast learning model determined in step S1; S3, in the prediction stage, a prediction confidence threshold is set, the picture collector and the sound collector are arranged at a fixed position of the conveying belt, and the picture and the corresponding sound information on the conveying belt are extracted, the sound extraction frame is a sequence information of a set time before and after the picture, and then the input information of the two is input into step S1 and step S2 to obtain a final prediction result of the conveying belt damage; The sound extraction step in step S12 is as follows: SB121, through PANNs and EAT-M two speech feature extraction methods, respectively extract the feature expression layer before the classification head of the two, feature F panns and feature F eatm ; SB122, the feature F panns and the feature F eatm are fused, and then the output feature F voice is obtained through a two-layer MLP; Step S2 is specifically: using the pre-training model of the contrast learning in step S1, freezing the weights of the picture feature encoding and the sound feature encoding in the model, inputting the obtained sound signal and picture signal into the corresponding encoder to obtain the sound and picture features, then connecting the features of the two through concate and obtaining attention feature information through a Spatial Attention Module, outputting the attention feature information to two layers of MLP modules, connecting the two layers of MLP modules through a SiLu activation function, and finally outputting a binary classification output prediction result through sigmoid, the result is divided into two categories of wear and no wear.

2. The multi-modal based output belt breakage detection algorithm as claimed in claim 1, wherein, The picture feature extraction step in step S12 is as follows: SA121, the input picture is respectively subjected to 1x1 convolution to obtain a feature F1, and is input into a hollow convolution module with a hollow rate of 6, 12 and 18 and a convolution kernel size of 3x3 to obtain corresponding features F2, F3 and F4, and the picture is also subjected to 3x3 convolution and pooling operation and then deconvolution operation to obtain a feature F5; the hollow convolution module comprises a hollow convolution, a BatchNorm and a corresponding GeLu activation operation function arranged in series; SA122, the features F1, F2, F3, F4, F5 are obtained after the concate module to get the feature map F c ; SA123, the input picture is divided by the patch partition module and combined with the feature map F c After the concate module is combined, the reshape module in series, the self-attention encoder module of the Transformer model, and the feature F are obtained t ; SA124, the feature F t input to the sequencepooling module to obtain the feature F s ; SA125, the feature F s input to the spatial attention and channel attention CBAM module to obtain the feature F cbam The spatial attention and channel attention CBAM module is composed of a channel attention module and a spatial attention module in series. SA136, feature F cbam input into a global pooling operation, to obtain the final picture feature F through the last three-layer MLP layer in series image The activation function between the first two MLP layers is ReLu, and the activation function between the second and third MLP layers is SiLu.

3. The multi-modal based output belt break detection algorithm as claimed in claim 1, wherein, In step S2, the cross-entropy loss function is used for comparing and calculating classification with the real label.

4. The multi-modal based output belt break detection algorithm as claimed in claim 1, wherein, The prediction confidence threshold is set to 0.

75.

5. The multi-modal based output band breakage detection algorithm as claimed in any one of claims 1-2, wherein, In step S1, if the anchor picture or its augmentation and the corresponding sound and its augmented sound are considered as a positive sample pair, the corresponding label is True, otherwise, if the assigned picture and sound signal are not a matching pair, it is considered as a negative sample pair, and the corresponding label is False.

Images