A belt abnormality monitoring method and device based on a multi-modal large model

Abstract

The application discloses a kind of based on multimodal large model's belt abnormal monitoring method and device, the method includes: respectively using sound sensor and camera to collect the audio signal and image data of belt operation, and constructs data set after being preprocessed;Build multimodal fusion model, and use data set to train, optimize model parameters in training process by joint loss function;Inference use stage will the audio signal of real-time acquisition after de-noising be converted into mel spectrum diagram and image data input into trained multimodal fusion model, output abnormal detection result and foreign matter position coordinate information;When detecting abnormality, trigger alarm and generate maintenance work order, simultaneously, abnormal detection result and foreign matter position coordinate information are input to large language model to generate maintenance suggestion.The application realizes the accurate detection and positioning of belt abnormality, and can more effectively improve job production quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring of industrial equipment, and particularly relates to a belt abnormality monitoring method and device based on a multi-modal large model. BACKGROUND

[0002] A belt conveyor is a core transportation device in industrial production, and its abnormal state such as foreign matter blockage, belt tearing, bearing wear, etc. can easily cause shutdown accidents. Therefore, it is necessary to monitor the belt conveyor for abnormalities, and the traditional monitoring methods have the following defects: ① single modal limitation: only relying on vibration sensors or image analysis, it is difficult to distinguish noise interference from real abnormalities under complex working conditions; ② poor environmental adaptability: sensor data is severely distorted in high-dust and strong-vibration scenes; ③ insufficient positioning accuracy: unable to output abnormal type and location information at the same time.

[0003] Therefore, in order to solve the above defects, a multi-modal fusion high-precision abnormality monitoring scheme is urgently needed. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and provides a belt abnormality monitoring method and device based on a multi-modal large model.

[0005] The purpose of the present application is achieved by the following technical solutions: the first aspect of the embodiment of the present application provides a belt abnormality monitoring method based on a multi-modal large model, which specifically comprises:

[0006] audio signals and image data during belt operation are collected simultaneously and preprocessed to construct a data set;

[0007] A multi-modal fusion model is constructed, and the multi-modal fusion model is trained using the data set. In the training process, the model parameters are optimized by a joint loss function to obtain a trained multi-modal fusion model;

[0008] In the inference use stage, audio signals and image data during belt operation are collected in real time, and the trained multi-modal fusion model is used to obtain abnormality detection results and foreign matter position coordinate information. When an abnormality is detected, an alarm is triggered and a maintenance work order is generated, and the abnormality detection results and foreign matter position coordinate information are input to a large language model to generate maintenance suggestions.

[0009] Further, the audio signals and image data during belt operation are collected simultaneously, specifically comprising:

[0010] The audio signals during belt operation are collected by a sound sensor, the image data during belt operation are collected by an industrial camera synchronously, and the audio signals and image data are timestamped and aligned by a time synchronization module.

[0011] Further, the time synchronization module adopts a hardware trigger signal to ensure that the audio signal and the image data are collected at a synchronized frequency; wherein the sampling rate of the audio signal is 44.1 kHz, and the frame rate of the image data is 30 fps; and each frame of image data is matched with a corresponding audio signal segment through timestamp alignment.

[0012] Further, the preprocessing specifically includes:

[0013] For the audio signal, a filtering method is used to remove the environmental noise therein to obtain a denoised audio signal, and then the denoised audio signal is converted into a mel-spectrogram, and an audio category label is labeled thereon; wherein the filtering method is wavelet transform; and the audio category includes a normal category, a belt breakage abnormal category, a foreign object blockage abnormal category, and a friction abnormal sound abnormal category.

[0014] For the image data, a foreign object category label is labeled thereon, and a foreign object coordinate frame is labeled simultaneously; wherein the foreign object category includes a no foreign object category, a mineral accumulation foreign object category, a metal scrap foreign object category, and a belt surface crack foreign object category.

[0015] Further, the multi-modal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first multi-layer perception module, a second multi-layer perception module, a cross-modal feature fusion module, and a cross-modal classifier.

[0016] The audio feature extraction network adopts ResNet-34 to extract mel-spectrogram features; the audio classifier is used to predict the audio category corresponding to the mel-spectrogram features; the image feature extraction network adopts the backbone network part of the YOLOV11 model to extract image features; the image detection module adopts the neck network part and the detection head of the YOLOV11 model to predict the foreign object category and the foreign object boundary box corresponding to the image features; the first multi-layer perception module and the second multi-layer perception module both adopt three fully connected layers, and each fully connected layer is connected with a batch normalization layer; the first multi-layer perception module is used to obtain an audio feature vector corresponding to the mel-spectrogram features, and the second multi-layer perception module is used to obtain an image feature vector corresponding to the image features; the cross-modal feature fusion module adopts a cross-attention mechanism to weight and aggregate the audio feature vector and the image feature vector to obtain a cross-modal fusion feature; and the cross-modal classifier is used to predict the joint abnormal state category corresponding to the cross-modal fusion feature.

[0017] Further, the multi-modal fusion model is trained using the data set, specifically including:

[0018] The mel-frequency spectrogram and image data in the data set are input into the multi-modal fusion model to obtain a predicted audio category, a foreign matter category and a foreign matter bounding box, and a joint abnormal state category; an audio classification loss is calculated according to the predicted audio category and a corresponding real audio category label thereof; a target classification loss and a distribution focus loss are calculated according to the predicted foreign matter category and a corresponding real foreign matter category label thereof, a bounding box regression loss is calculated according to the predicted foreign matter bounding box and a corresponding real foreign matter coordinate box thereof, and a sum of the target classification loss, the bounding box regression loss and the distribution focus loss is calculated as an image target detection loss; a cross-modal classification loss is calculated according to the predicted joint abnormal state category and a corresponding real abnormal state category thereof; a joint loss function is calculated according to the audio classification loss, the image target detection loss and the cross-modal classification loss; and model parameters of the multi-modal fusion model are optimized by using a stochastic gradient descent method with a minimum joint loss function as an optimization target until a preset training round is reached, so as to obtain a trained multi-modal fusion model.

[0019] Further, the calculation formula of the distribution focus loss is:

[0020] ;

[0021] In the formula, denotes the distribution focus loss, denotes a predicted probability that an i th sample output by a classification detection head of a YOLOV11 model belongs to a j th category, denotes a real foreign matter category label corresponding to the i th sample, and denote an index for adjusting a difficult sample weight and a parameter for controlling an influence degree of a category distribution deviation, respectively, is a prior frequency distribution of the j th category, , denotes a number of times that the j th category appears in batch training.

[0022] Further, the inference use stage specifically includes:

[0023] Real-time acquisition of audio signals and image data during belt operation, conversion of the audio signals into mel-frequency spectrograms after noise removal, and input of the image data into the trained multi-modal fusion model to obtain a predicted audio category, a foreign matter category and a foreign matter bounding box, and a joint abnormal state category;

[0024] Threshold judgment is performed on audio category probabilities and joint abnormal state category probabilities output by the multi-modal fusion model, and when the audio category probabilities and the joint abnormal state category probabilities both exceed a preset threshold, an alarm is given and a maintenance work order is generated;

[0025] According to the set prompt word template, the abnormal state category needing to be alarmed is filled into the prompt word template as the abnormal detection result and foreign matter position coordinate information, and the created prompt word is input into the large language model to generate a maintenance suggestion.

[0026] The second aspect of the embodiment of the present application provides a belt abnormality monitoring device based on a multi-modal large model, comprising one or more processors and a memory, the memory being coupled with the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the multi-modal large model-based belt abnormality monitoring method.

[0027] The third aspect of the embodiment of the present application provides a computer readable storage medium having a program stored thereon, and the program is executed by a processor to realize the multi-modal large model-based belt abnormality monitoring method.

[0028] The beneficial effects of the present application are: the present application solves the problems of high false alarm rate and weak positioning ability of a single sensor by fusing audio and visual modalities, realizes accurate detection and positioning of belt abnormalities, and is suitable for real-time fault detection and early warning of belt conveyors in mine, port, power plant and other scenes; at the same time, the large model gives a maintenance scheme and suggestion for the abnormal state of the belt, which can more effectively improve the operation production quality. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of the multi-modal large model-based belt abnormality monitoring method of the present application;

[0030] Figure 2 is an architecture flowchart of the multi-modal fusion model of the present application;

[0031] Figure 3 is a test calling flowchart of the multi-modal large model-based belt abnormality monitoring of the present application;

[0032] Figure 4 is an example diagram of calling a large model to generate a maintenance suggestion of the present application;

[0033] Figure 5 is a structural schematic diagram of the multi-modal large model-based belt abnormality monitoring device of the present application. DETAILED DESCRIPTION

[0034] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. As such, the following description should not be considered limiting of the scope of the application, as broadly described herein, but rather a description of some embodiments consistent with the application. Numerous specific details are described herein in order to provide a thorough understanding of the application. However, in some cases, well-known or otherwise conventional details are not described in order to not unnecessarily obscure the application. Finally, it is to be understood that the use of certain terms is intended to convey a particular meaning, unless otherwise understood from their use in a particular context.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0036] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is to be further understood that the terms "comprise", "comprising", "comprises", "including", "includes" or "contain" or "containing" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. It is to be understood that the terms "if' and "as if' as used herein can be interpreted either as "when" or "once" or "in response to determining" depending on the context.

[0037] The application will be described in detail herein with reference to the attached drawings. The features of the embodiments and implementation described below can be combined with each other in the case of no conflict.

[0038] Referring to Figure 1 The multi-modal large model-based belt abnormality monitoring method of the present application specifically comprises the following steps:

[0039] (1) Collecting audio signals and image data of the belt running at the same time, and preprocessing them to construct a data set.

[0040] Further, the audio signals and image data of the belt running at the same time are collected, specifically including: collecting the audio signals of the belt running through the sound sensor, synchronously collecting the image data of the belt running through the industrial camera, and aligning the audio signals and image data through the time synchronization module to ensure the consistency of the time stamps.

[0041] Further, the time synchronization module adopts a hardware trigger signal to ensure that the audio signal and the image data are collected at a synchronized frequency; the sampling rate of the audio signal is 44.1 kHz, and the frame rate of the image data is 30 fps; each frame of image data is matched with a corresponding audio signal segment through timestamp alignment.

[0042] Further, the preprocessing specifically includes: for the audio signal, using a filtering method to remove the environmental noise therein to obtain a denoised audio signal, and then converting the denoised audio signal into a Mel spectrum graph and labeling the audio category label thereof; the filtering method is wavelet transform; since the audio state is divided into a normal state and an abnormal state, the abnormal state includes belt breakage, foreign object blockage, friction abnormal sound, etc., and therefore the audio category includes a normal category, a belt breakage abnormal category, a foreign object blockage abnormal category, a friction abnormal sound abnormal category, etc. For the image data, a foreign object category label is labeled, and a foreign object coordinate frame is labeled at the same time; since there can be a foreign object in the image, and there can be no foreign object, and when there is a foreign object, the foreign object includes ore accumulation, metal debris, belt surface crack, etc., and therefore the foreign object category includes a no-foreign-object category, an ore accumulation foreign object category, a metal debris foreign object category, a belt surface crack foreign object category, etc.

[0043] It should be noted that the various category labels involved in the present embodiment are encoded using one-hot encoding, for example, an audio category label of 0010 indicates that the corresponding audio belongs to the foreign object blockage abnormal category, and an audio category label of 1000 indicates that the corresponding audio belongs to the normal category.

[0044] In the present embodiment, the audio signal is subjected to wavelet decomposition using a wavelet basis Daubechies, and then a soft threshold is selected for threshold processing, and the wavelets that exceed the soft threshold range are removed, so that the environmental noise in the audio signal is removed, and finally an inverse wavelet transform is performed to obtain a denoised audio signal. In the present embodiment, the PyWavelets library in python is directly used for denoising processing, and the denoised audio signal is converted into a Mel spectrum graph using the librosa library.

[0045] (2) A multi-modal fusion model is constructed, and the multi-modal fusion model is trained using a data set; during the training process, the model parameters are optimized through a joint loss function to obtain a trained multi-modal fusion model.

[0046] In the present embodiment, the multi-modal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first Multilayer Perceptron (MLP) module, a second MLP module, a cross-modal feature fusion module, and a cross-modal classifier, as shown in Figure 2The audio feature extraction network adopts ResNet-34, which is used to extract Mel spectrum feature; the audio classifier is used to predict the audio category corresponding to the Mel spectrum feature; the image feature extraction network adopts the backbone network part of the YOLOV11 model, which is used to extract image features; the image detection module adopts the neck network part and the detection head of the YOLOV11 model, which is used to predict the foreign matter category and the foreign matter bounding box corresponding to the image feature; the first MLP module and the second MLP module both adopt three fully connected layers, and each fully connected layer is connected with a batch normalization layer, the first MLP module is used to obtain the audio feature vector corresponding to the Mel spectrum feature, and the second MLP module is used to obtain the image feature vector corresponding to the image feature; the cross-modal feature fusion module adopts a cross-attention mechanism to weight and aggregate the audio feature vector and the image feature vector to obtain the cross-modal fusion feature; and the cross-modal classifier is used to predict the joint abnormal state category corresponding to the cross-modal fusion feature.

[0047] Specifically, the Mel spectrum and the image data are input into the multi-modal fusion model, the Mel spectrum is input into the ResNet-34 to obtain the corresponding Mel spectrum feature; the Mel spectrum feature is input into the audio classifier to obtain the predicted audio category. The image data is input into the backbone network part of the YOLOV11 model to obtain the corresponding image feature; the image feature is further input into the neck network part and the detection head of the YOLOV11 model to obtain the predicted foreign matter category and the foreign matter bounding box. The Mel spectrum feature is mapped to the vector space by the first MLP module to obtain the audio feature vector, denoted as , and the vector length is , and the dimension is 1024 in this embodiment; the image feature is mapped to the vector space by the second MLP module to obtain the image feature vector, denoted as , and the vector length is , and the dimension is 1024 in this embodiment. The audio feature vector and the image feature vector are input into the cross-modal feature fusion module, and the cross-modal feature fusion module adopts a cross-attention mechanism to weight and aggregate the audio feature vector and the image feature vector to obtain the cross-modal fusion feature. Specifically, the implementation process of the cross-modal feature fusion module includes: ① mapping the audio feature vector into a query (Query) vector, and mapping the image feature vector into a key (Key) vector and a value (Value) vector, which are respectively denoted as:

[0048] ;

[0049] ;

[0050] ;

[0051] In the formula, Q, K, V respectively represent a query vector, a key vector, and a value vector; 、 、 are respectively learnable query parameter matrices, key parameter matrices, and value parameter matrices, and the dimensions of the parameter matrices in this embodiment are all 256*1024 dimensions. ②Calculate the attention score and normalize, denoted as:

[0052] ;

[0053] In the formula, denotes the normalized attention score, denotes a Softmax activation function, and the superscript denotes the transpose of a matrix or a vector, denotes the vector length of the audio feature vector , which is 1024. ③Weighted aggregation, denoted as:

[0054] ;

[0055] In the formula, denotes a cross-modal fusion feature. The cross-modal fusion feature is obtained by a cross-modal classifier to predict a joint abnormal state category.

[0056] In this embodiment, the multi-modal fusion model is trained using a data set, specifically including: as Figure 2As shown, the Mel spectrogram and image data in the data set are input into the multi-modal fusion model to obtain the predicted audio category, foreign object category and foreign object bounding box, and joint abnormal state category; the audio classification loss is calculated according to the predicted audio category and the corresponding real audio category label thereof; the target classification loss and the distribution focal loss are calculated according to the predicted foreign object category and the corresponding real foreign object category label thereof, the bounding box regression loss is calculated according to the predicted foreign object bounding box and the corresponding real foreign object coordinate box, and the sum of the target classification loss, the bounding box regression loss and the distribution focal loss is taken as the image target detection loss; the cross-modal classification loss is calculated according to the predicted joint abnormal state category and the corresponding real abnormal state category; the joint loss function is calculated according to the audio classification loss, the image target detection loss and the cross-modal classification loss; the model parameters of the multi-modal fusion model are optimized by the stochastic gradient descent (SGD) method with the optimization target of minimizing the joint loss function, the learning rate can be set to 0.001, and the training is performed until the preset training round is reached, so as to obtain the trained multi-modal fusion model.

[0057] Further, the calculation formula of the audio classification loss is:

[0058]

[0059] In the formula, denotes the audio classification loss, denotes the predicted probability that the i th sample output by the audio classifier belongs to the j th category, denotes the real audio category label (0 or 1) of the i th sample corresponding to the j th category, n denotes the total number of samples in a batch training, and C denotes the total number of categories. In this embodiment, n = 64, indicating that the total number of training samples in a batch is 64; C = 4, including 3 abnormal categories and 1 normal category.

[0060] Further, the calculation formula of the image target detection loss is:

[0061] ;

[0062] In the formula, denotes the image target detection loss; denotes the target classification loss, which is commonly used in the YOLOV11 model and will not be described here; denotes the bounding box regression loss, which is commonly used in the YOLOV11 model and will not be described here; denotes the distribution focal loss, and the calculation formula thereof is:

[0063] ;

[0064] wherein, represents the prediction probability of the i-th sample output by the classification detection head of the YOLOV11 model belonging to the j-th class, represents the real foreign object class label (0 or 1) corresponding to the i-th sample belonging to the j-th class, and respectively represent the index for adjusting the weight of the difficult sample and the parameter for controlling the influence degree of the deviation of the class distribution, is the prior frequency distribution of the j-th class, , represents the number of times the j-th class appears in the batch training.

[0065] Further, the calculation formula of the cross-modal classification loss is:

[0066] ;

[0067] wherein, represents the prediction probability of the i-th sample output by the cross-modal classifier belonging to the j-th class, represents the real abnormal state class (0 or 1, taking the real audio class label as the real abnormal state class) corresponding to the i-th sample belonging to the j-th class.

[0068] Further, the calculation formula of the joint loss function is:

[0069] ;

[0070] wherein, represents the joint loss function, , and respectively represent the weight coefficients of the audio classification loss, the image target detection loss and the cross-modal classification loss.

[0071] (3) In the inference use stage, the audio signal and the image data during the running of the belt are collected in real time, the trained multi-modal fusion model is used to obtain the abnormal detection result and the foreign object position coordinate information; when an abnormality is detected, an alarm is triggered and a maintenance work order is generated, and the abnormal detection result and the foreign object position coordinate information are input to a large language model (Large Language Model, LLM, referred to as a large model) to generate maintenance suggestions, as shown in Figure 3 .

[0072] (3.1) Real-time acquisition of audio signals and image data during belt operation, denoising of the audio signals, conversion into Mel spectrograms, and input of the image data into a trained multi-modal fusion model to obtain predicted audio categories, foreign object categories and foreign object bounding boxes, and joint abnormal state categories. The Mel spectrogram is input into a ResNet-34 and an audio classifier to obtain the predicted audio category, the image data is input into a YOLOV11 model to obtain the predicted foreign object category and foreign object bounding box, and the cross-modal feature fusion module and cross-modal classifier are used to obtain the predicted joint abnormal state category.

[0073] (3.2) Threshold judgment of the audio category probability and the joint abnormal state category probability output by the multi-modal fusion model, for example, a fixed threshold of 0.7 can be used, and when the audio category probability and the joint abnormal state category probability both exceed the preset threshold, an alarm is generated and a maintenance work order is generated.

[0074] (3.3) Input of the abnormal state category and the foreign object position coordinate information that need to be alarmed into a large language model to generate maintenance suggestions, specifically: according to a set prompt word template, fill the abnormal state category that needs to be alarmed as the abnormal detection result and the foreign object position coordinate information into the prompt word template, and input the created prompt word into the large language model to generate the maintenance suggestion. The abnormal state category that needs to be alarmed is determined according to the threshold judgment of step (3.2); the foreign object position coordinate information is determined according to the predicted foreign object bounding box.

[0075] Further, the prompt word template is: "You are a belt maintenance personnel, the belt has the following abnormal state now, and foreign objects are detected on the belt, the following is the abnormal state information: <>, please give maintenance scheme and suggestions according to this information." The created prompt word is input into the large language model, and the corresponding maintenance suggestion is returned. As shown in Figure 4 , one example of the prompt word is: "You are a belt maintenance personnel, the belt has the following abnormal state now, and foreign objects are detected on the belt, the following is the abnormal state information: the belt abnormal type is tearing, the foreign object position coordinates are <23, 567>, <67, 456>, please give maintenance scheme and suggestions according to this information.", and the corresponding maintenance suggestion generated by the large language model is as shown in Figure 4 .

[0076] Corresponding to the foregoing embodiment of the belt abnormality monitoring method based on the multi-modal large model, the application also provides an embodiment of a belt abnormality monitoring device based on the multi-modal large model.

[0077] Referring to Figure 5The embodiment of the present application provides a belt abnormity monitoring device based on a multi-modal large model, which comprises one or more processors and a memory, and the memory is coupled with the processor; wherein the memory is used for storing program data, and the processor is used for executing the program data to realize the multi-modal large model based belt abnormity monitoring method in the above embodiment.

[0078] The embodiment of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, hardware or a combination of software and hardware. Taking software realization as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability. Figure 5 As shown in the figure, it is a hardware structure diagram of the device with data processing capability of the present application, in addition to the processor, the memory, the network interface and the non-volatile memory shown in the figure, the device with data processing capability in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, and details are not described here. Figure 5 As shown in the figure, it is a hardware structure diagram of the device with data processing capability of the present application, in addition to the processor, the memory, the network interface and the non-volatile memory shown in the figure, the device with data processing capability in the embodiment usually includes other hardware according to the actual function of the device with data processing capability, and details are not described here.

[0079] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and details are not described here.

[0080] For the device embodiment, since it basically corresponds to the method embodiment, the related part can be referred to the part of the method embodiment. The above described device embodiment is only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to realize the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0081] The embodiment of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the multi-modal large model based belt abnormity monitoring method in the above embodiment.

[0082] The computer readable storage medium can be an internal storage unit of any of the aforementioned devices with data processing capability, such as a hard disk or a memory. The computer readable storage medium can also be any of the aforementioned devices with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can also include both an internal storage unit of any of the aforementioned devices with data processing capability and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the aforementioned devices with data processing capability, and can also be used to temporarily store data that has been output or will be output.

[0083] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A belt anomaly monitoring method based on a multimodal large model, characterized in that, Specifically, it includes: Simultaneously, audio signals and image data from the belt operation are collected and preprocessed to construct a dataset; A multimodal fusion model is constructed and trained using a dataset. During training, the model parameters are optimized using a joint loss function to obtain a well-trained multimodal fusion model. Specifically, training the multimodal fusion model using the dataset includes: The Mel spectrograms and image data from the dataset are input into the multimodal fusion model to obtain the predicted audio category, foreign object category, foreign object bounding box, and joint anomalous state category. Audio classification loss is calculated based on the predicted audio category and its corresponding ground truth audio category label. Target classification loss and distribution focus loss are calculated based on the predicted foreign object category and its corresponding ground truth foreign object category label. Bounding box regression loss is calculated based on the predicted foreign object bounding box and its corresponding ground truth foreign object coordinate box. The sum of the target classification loss, bounding box regression loss, and distribution focus loss is used as the image target detection loss. Cross-modal classification loss is calculated based on the predicted joint anomalous state category and its corresponding ground truth anomalous state category. A joint loss function is calculated based on the audio classification loss, image target detection loss, and cross-modal classification loss. The model parameters of the multimodal fusion model are optimized using stochastic gradient descent with the goal of minimizing the joint loss function, until a preset number of training epochs are reached to obtain a trained multimodal fusion model. During the inference and usage phase, audio signals and image data of the belt during operation are collected in real time. Using a trained multimodal fusion model, anomaly detection results and foreign object location coordinates are obtained. When an anomaly is detected, an alarm is triggered and a maintenance work order is generated. At the same time, the anomaly detection results and foreign object location coordinates are input into the large language model to generate maintenance suggestions.

2. The belt anomaly monitoring method based on a multimodal large model according to claim 1, characterized in that, The simultaneous acquisition of audio signals and image data during belt operation specifically includes: The system collects audio signals from the belt during operation using a sound sensor, simultaneously captures image data from the belt using an industrial camera, and then timestamps the audio signals and image data using a time synchronization module.

3. The belt anomaly monitoring method based on a multimodal large model according to claim 2, characterized in that, The time synchronization module uses a hardware trigger signal to ensure that the acquisition frequencies of audio signals and image data are synchronized; the sampling rate of the audio signal is 44.1kHz, and the acquisition frame rate of the image data is 30fps; timestamp alignment is used to match the corresponding audio signal segment with each frame of image data.

4. The belt anomaly monitoring method based on a multimodal large model according to claim 1, characterized in that, The preprocessing specifically includes: For audio signals, a filtering method is used to remove environmental noise to obtain a denoised audio signal. The denoised audio signal is then converted into a Mel spectrogram, and its audio category is labeled. The filtering method is wavelet transform. The audio categories include normal category, belt breakage abnormal category, foreign object jamming abnormal category, and friction noise abnormal category. For image data, foreign object category labels are marked, along with foreign object coordinate boxes; the foreign object categories include no foreign object category, ore accumulation foreign object category, metal debris foreign object category, and belt surface crack foreign object category.

5. The belt anomaly monitoring method based on a multimodal large model according to claim 1, characterized in that, The multimodal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first multilayer perceptron module, a second multilayer perceptron module, a cross-modal feature fusion module, and a cross-modal classifier; The audio feature extraction network uses ResNet-34 to extract Mel spectrogram features; the audio classifier predicts the audio category corresponding to the Mel spectrogram features; the image feature extraction network uses the backbone of the YOLOv11 model to extract image features; the image detection module uses the neck network and detection head of the YOLOv11 model to predict the foreign object category and foreign object bounding box corresponding to the image features; both the first and second multilayer perceptron modules use three fully connected layers, with each fully connected layer followed by a batch normalization layer. The first multilayer perceptron module obtains the audio feature vector corresponding to the Mel spectrogram features, and the second multilayer perceptron module obtains the image feature vector corresponding to the image features; the cross-modal feature fusion module uses a cross-attention mechanism to weight and aggregate the audio and image feature vectors to obtain cross-modal fused features; the cross-modal classifier predicts the joint abnormal state category corresponding to the cross-modal fused features.

6. The belt anomaly monitoring method based on a multimodal large model according to claim 1, characterized in that, The formula for calculating the distribution focus loss is as follows: ； In the formula, Indicates the distribution focus loss. This represents the predicted probability that the i-th sample belongs to the j-th class, as output by the classification detection head of the YOLOv11 model. This represents the true foreign object category label corresponding to the j-th category for the i-th sample. and These represent the index that adjusts the weights of easy and difficult samples, and the parameter that controls the degree of influence of class distribution bias, respectively. Let j be the prior frequency distribution of the j-th category. , Let represent the number of times the j-th category appears in the batch training, n represent the total number of samples in the batch training, and C represent the total number of categories.

7. The belt anomaly monitoring method based on a multimodal large model according to claim 1, characterized in that, The reasoning usage phase specifically includes: The system collects audio signals and image data during belt operation in real time. After denoising, the audio signals are converted into bit-Mel spectrograms and the image data are input into a trained multimodal fusion model to obtain the predicted audio category, foreign object category, foreign object bounding box, and joint abnormal state category. Threshold judgment is performed on the audio category probability and joint abnormal state category probability output by the multimodal fusion model. When both the audio category probability and the joint abnormal state category probability exceed the preset threshold, an alarm is triggered and a maintenance work order is generated. According to the set prompt word template, the abnormal status category that needs to be alarmed is filled into the prompt word template as the abnormal detection result and the location coordinate information of the foreign object. The created prompt words are then input into the large language model to generate maintenance suggestions.

8. A belt anomaly monitoring device based on a multimodal large model, comprising one or more processors and a memory, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the belt anomaly monitoring method based on a multimodal large model as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, is used to implement the belt anomaly monitoring method based on a multimodal large model as described in any one of claims 1-7.

Images