Belt abnormity monitoring method and device based on multi-modal large model

Abstract

The invention discloses a belt abnormity monitoring method and device based on a multi-modal large model, and the method comprises the steps: collecting audio signals and image data during the operation of a belt through a sound sensor and a camera, and carrying out the preprocessing of the audio signals and image data, and constructing a data set; a multi-modal fusion model is constructed, a data set is used for training, and model parameters are optimized through a joint loss function in the training process; in the reasoning and using stage, audio signals collected in real time are denoised and then converted into a Mel spectrogram and image data, the Mel spectrogram and the image data are input into the trained multi-modal fusion model, and an anomaly detection result and foreign matter position coordinate information are output; and when an anomaly is detected, triggering an alarm and generating a maintenance work order, and inputting an anomaly detection result and foreign matter position coordinate information to the large language model to generate a maintenance suggestion. According to the invention, accurate detection and positioning of belt abnormity are realized, and the operation production quality can be more effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent monitoring of industrial equipment, and in particular to a belt anomaly monitoring method and device based on a multimodal large model. Background Art

[0002] Belt conveyors are core transportation equipment in industrial production. Abnormal conditions such as foreign object jamming, belt tearing, and bearing wear can easily cause downtime. Therefore, belt conveyor anomaly monitoring is necessary, but traditional monitoring methods have the following drawbacks: ① Single-modality limitations: Relying solely on vibration sensors or image analysis, it is difficult to distinguish between noise interference and true anomalies in complex operating conditions; ② Poor environmental adaptability: Sensor data is severely distorted in high dust and strong vibration environments; ③ Inadequate positioning accuracy: The inability to simultaneously output anomaly type and location information.

[0003] Therefore, in order to solve the above defects, a high-precision anomaly monitoring solution based on multimodal fusion is urgently needed. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide a belt anomaly monitoring method and device based on a multimodal large model.

[0005] The object of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a belt abnormality monitoring method based on a multimodal large model, specifically comprising: At the same time, the audio signal and image data of the belt are collected and preprocessed to construct a data set; Construct a multimodal fusion model and use the dataset to train the multimodal fusion model. During the training process, the model parameters are optimized through the joint loss function to obtain a trained multimodal fusion model. During the inference and use phase, audio signals and image data of the belt are collected in real time during operation, and the trained multimodal fusion model is used to obtain anomaly detection results and foreign object location coordinate information. When an anomaly is detected, an alarm is triggered and a maintenance work order is generated. At the same time, the anomaly detection results and foreign object location coordinate information are input into the large language model to generate maintenance suggestions.

[0006] Furthermore, the simultaneous collection of audio signals and image data during belt operation specifically includes: The audio signal of the belt during operation is collected through the sound sensor, the image data of the belt during operation is collected synchronously through the industrial camera, and the audio signal and image data are time-stamped and aligned through the time synchronization module.

[0007] Furthermore, the time synchronization module uses a hardware trigger signal to ensure synchronization of the acquisition frequencies of the audio signal and image data; wherein, the sampling rate of the audio signal is 44.1kHz, and the acquisition frame rate of the image data is 30fps; and timestamp alignment is used to match each frame of image data to the corresponding audio signal segment.

[0008] Furthermore, the preprocessing specifically includes: For audio signals, a filtering method is used to remove ambient noise to obtain a denoised audio signal. The denoised audio signal is then converted into a Mel-spectrogram and labeled with audio category labels. The filtering method is wavelet transform. Audio categories include normal, belt breakage, foreign object jamming, and friction noise. For the image data, the foreign body category labels are annotated, and the foreign body coordinate frames are also annotated; among them, the foreign body categories include no foreign body category, ore accumulation foreign body category, metal debris foreign body category, and belt surface crack foreign body category.

[0009] Furthermore, the multimodal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first multi-layer perceptron module, a second multi-layer perceptron module, a cross-modal feature fusion module and a cross-modal classifier; Among them, the audio feature extraction network adopts ResNet-34 to extract Mel-spectrogram features; the audio classifier is used to predict the audio category corresponding to the Mel-spectrogram features; the image feature extraction network adopts the backbone network part of the YOLOV11 model to extract image features; the image detection module adopts the neck network part and detection head of the YOLOV11 model to predict the foreign object category and foreign object bounding box corresponding to the image features; the first multi-layer perceptron module and the second multi-layer perceptron module both adopt three fully connected layers, and each fully connected layer is connected to a batch normalization layer. The first multi-layer perceptron module is used to obtain the audio feature vector corresponding to the Mel-spectrogram feature, and the second multi-layer perceptron module is used to obtain the image feature vector corresponding to the image feature; the cross-modal feature fusion module adopts a cross-attention mechanism to perform weighted aggregation on the audio feature vector and the image feature vector to obtain the cross-modal fusion feature; the cross-modal classifier is used to predict the joint abnormal state category corresponding to the cross-modal fusion feature.

[0010] Furthermore, the multimodal fusion model is trained using the dataset, specifically including: The Mel-spectrogram and image data in the dataset are input into the multimodal fusion model to obtain the predicted audio category, foreign object category, foreign object bounding box and joint abnormal state category; the audio classification loss is calculated based on the predicted audio category and its corresponding true audio category label; the target classification loss and distribution focus loss are calculated based on the predicted foreign object category and its corresponding true foreign object category label, the bounding box regression loss is calculated based on the predicted foreign object bounding box and its corresponding true foreign object coordinate box, and the sum of the target classification loss, bounding box regression loss and distribution focus loss is calculated as the image target detection loss; the cross-modal classification loss is calculated based on the predicted joint abnormal state category and its corresponding true abnormal state category; the joint loss function is calculated based on the audio classification loss, image target detection loss and cross-modal classification loss; with minimizing the joint loss function as the optimization goal, the model parameters of the multimodal fusion model are optimized using the stochastic gradient descent method until the preset training rounds are reached to obtain a trained multimodal fusion model.

[0011] Furthermore, the calculation formula of the distributed focus loss is: ; Where, represents the distribution focal loss, Represents the predicted probability that the i-th sample output by the classification detection head of the YOLOV11 model belongs to the j-th category, Indicates the true foreign body category label of the i-th sample corresponding to the j-th category, and They represent the index for adjusting the weight of difficult and easy samples and the parameters for controlling the degree of influence of category distribution deviation, is the prior frequency distribution of the j-th category, , Indicates the number of times the jth category appears in the batch training.

[0012] Furthermore, the inference and use phase specifically includes: The audio signal and image data of the belt are collected in real time during operation. The audio signal is converted into a Mel-spectrogram after denoising, and the image data is input into the trained multimodal fusion model to obtain the predicted audio category, foreign object category and foreign object bounding box, and joint abnormal state category. Threshold judgment is performed on the audio category probability and the joint abnormal state category probability output by the multimodal fusion model. When both the audio category probability and the joint abnormal state category probability exceed the preset threshold, an alarm is issued and a maintenance work order is generated; According to the set prompt word template, the abnormal state category that needs to be alarmed is filled into the prompt word template as the abnormal detection result and the foreign object location coordinate information. The created prompt word is input into the large language model to generate maintenance suggestions.

[0013] A second aspect of an embodiment of the present invention provides a belt abnormality monitoring device based on a multimodal large model, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned belt abnormality monitoring method based on the multimodal large model.

[0014] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the program is used to implement the above-mentioned belt abnormality monitoring method based on a multimodal large model.

[0015] The beneficial effects of the present invention are: by integrating audio and visual modalities, the present invention solves the problems of high false alarm rate and weak positioning ability of a single sensor, realizes accurate detection and positioning of belt abnormalities, and is suitable for real-time fault detection and early warning of belt conveyors in scenarios such as mines, ports, and power plants; at the same time, combined with a large model, it provides maintenance plans and suggestions for abnormal belt conditions, which can more effectively improve the quality of operational production. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the belt abnormality monitoring method based on the multimodal large model of the present invention; Figure 2 It is a flow chart of the architecture of the multimodal fusion model of the present invention; Figure 3 This is a test call flow chart of the belt abnormality monitoring based on the multi-modal large model of the present invention; Figure 4 This is an example diagram of calling a large model to generate maintenance suggestions according to the present invention; Figure 5 It is a structural schematic diagram of the belt anomaly monitoring device based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0017] The exemplary embodiments will be described in detail herein, with examples shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Obviously, the drawings used in the following description are only some embodiments of the present invention, and it is possible for a person of ordinary skill in the art to derive other drawings based on these drawings without inventive effort. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0018] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0019] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0020] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0021] See also Figure 1 The belt abnormality monitoring method based on the multimodal large model of the present invention specifically includes the following steps: (1) The audio signal and image data of the belt are collected simultaneously and preprocessed to construct a dataset.

[0022] Furthermore, audio signals and image data of the belt are collected simultaneously during operation, specifically including: collecting audio signals of the belt during operation through a sound sensor, synchronously collecting image data of the belt during operation through an industrial camera, and aligning the timestamps of the audio signals and image data through a time synchronization module to ensure that the timestamps are consistent.

[0023] Furthermore, the time synchronization module uses a hardware trigger signal to ensure synchronization of the acquisition frequencies of the audio signal and image data. The sampling rate of the audio signal is 44.1kHz, and the acquisition frame rate of the image data is 30fps. Timestamp alignment is used to ensure that each frame of image data matches the corresponding audio signal segment.

[0024] Furthermore, preprocessing specifically includes: for audio signals, filtering is used to remove ambient noise to obtain a denoised audio signal, which is then converted into a Mel-spectrogram and annotated with audio category labels. The filtering method is wavelet transform. Since audio states are categorized as normal and abnormal, with abnormal states including belt breakage, foreign object obstruction, and friction noise, the audio categories include normal, belt breakage abnormal, foreign object obstruction abnormal, and friction noise abnormal. For image data, foreign object category labels are annotated, along with foreign object coordinate frames. Since images can contain or not contain foreign objects, and foreign objects present can include ore accumulation, metal debris, and belt surface cracks, the foreign object categories include no foreign object, ore accumulation foreign object, metal debris foreign object, and belt surface crack foreign object.

[0025] It should be noted that the various category labels involved in this embodiment use one-hot encoding. For example, when the audio category label is 0010, it means that the corresponding audio belongs to the foreign object obstruction abnormality category; when the audio category label is 1000, it means that the corresponding audio belongs to the normal category.

[0026] In this embodiment, the audio signal is decomposed using the Daubechies wavelet basis. A soft threshold is then selected for thresholding. Wavelets outside the soft threshold are removed to remove ambient noise from the audio signal. Finally, an inverse wavelet transform is performed to obtain the denoised audio signal. This embodiment directly uses the PyWavelets library in Python for denoising, and the denoised audio signal is converted into a Mel-spectrogram using the librosa library.

[0027] (2) Construct a multimodal fusion model and use the data set to train the multimodal fusion model. During the training process, the model parameters are optimized through the joint loss function to obtain a trained multimodal fusion model.

[0028] In this embodiment, the multimodal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first multilayer perceptron (MLP) module, a second MLP module, a cross-modal feature fusion module, and a cross-modal classifier. Figure 2As shown in the figure, the audio feature extraction network uses ResNet-34 to extract Mel spectrogram features; the audio classifier is used to predict the audio category corresponding to the Mel spectrogram features; the image feature extraction network uses the backbone network part of the YOLOV11 model to extract image features; the image detection module uses the neck network part (neck) and detection head (head) of the YOLOV11 model to predict the foreign object category and foreign object bounding box corresponding to the image features; the first MLP module and the second MLP module both use three fully connected layers, and each fully connected layer is connected to a batch normalization layer. The first MLP module is used to obtain the audio feature vector corresponding to the Mel spectrogram features, and the second MLP module is used to obtain the image feature vector corresponding to the image features; the cross-modal feature fusion module uses a cross-attention mechanism to perform weighted aggregation on the audio feature vector and the image feature vector to obtain the cross-modal fusion feature; the cross-modal classifier is used to predict the joint abnormal state category corresponding to the cross-modal fusion feature.

[0029] Specifically, the Mel spectrogram and image data are input into the multimodal fusion model. The Mel spectrogram is passed through ResNet-34 to obtain the corresponding Mel spectrogram features; the Mel spectrogram features are passed through the audio classifier to obtain the predicted audio category. The image data is passed through the backbone network part of the YOLOV11 model to obtain the corresponding image features; the image features are then passed through the neck network part and detection head of the YOLOV11 model to obtain the predicted foreign object category and foreign object bounding box. The Mel spectrogram features are mapped to the vector space through the first MLP module to obtain the audio feature vector, which is recorded as , whose vector length is , in this embodiment, its dimension is 1024; the image features are mapped to the vector space through the second MLP module to obtain the image feature vector, which is recorded as , whose vector length is In this embodiment, its dimension is 1024. Audio feature vector and image feature vector The input is sent to the cross-modal feature fusion module, which uses a cross-attention mechanism to and image feature vector Perform weighted aggregation to obtain cross-modal fusion features. The implementation process of the cross-modal feature fusion module specifically includes: ① The audio feature vector Mapped to a query vector, the image feature vector Mapped into key vectors and value vectors, represented as: ; ; ;

[0030] Where Q, K, and V represent query vector, key vector, and value vector, respectively; 、 、 are the query parameter matrix, key parameter matrix, and value parameter matrix, respectively. In this embodiment, the dimension of the parameter matrix is ​​256*1024. ② Calculate the attention score and normalize it, which is expressed as: ;

[0031] Where, represents the normalized attention score, Represents the Softmax activation function, the superscript represents the transpose of a matrix or vector, Represents the audio feature vector The vector length is 1024. ③ Weighted aggregation is expressed as: ;

[0032] Where, Represents cross-modal fusion features. Cross-modal fusion features are used to obtain the predicted joint abnormal state category through a cross-modal classifier.

[0033] In this embodiment, the multimodal fusion model is trained using a data set, specifically including: Figure 2 As shown, the Mel spectrogram and image data in the dataset are input into the multimodal fusion model to obtain the predicted audio category, foreign object category, foreign object bounding box, and joint abnormal state category; the audio classification loss is calculated based on the predicted audio category and its corresponding true audio category label; the target classification loss and distribution focal loss are calculated based on the predicted foreign object category and its corresponding true foreign object category label, the bounding box regression loss is calculated based on the predicted foreign object bounding box and its corresponding true foreign object coordinate box, and the sum of the target classification loss, bounding box regression loss, and distribution focal loss is calculated as the image target detection loss; the cross-modal classification loss is calculated based on the predicted joint abnormal state category and its corresponding true abnormal state category; the joint loss function is calculated based on the audio classification loss, image target detection loss, and cross-modal classification loss; with minimizing the joint loss function as the optimization goal, the stochastic gradient descent (SGD) method is used to optimize the model parameters of the multimodal fusion model. The learning rate can be set to 0.001 until the preset training round is reached to obtain a trained multimodal fusion model.

[0034] Furthermore, the audio classification loss is calculated as:

[0035] Where, represents the audio classification loss, represents the predicted probability that the i-th sample output by the audio classifier belongs to the j-th category, represents the true audio category label (0 or 1) of the jth category corresponding to the i-th sample, n represents the total number of samples in the training batch, and C represents the total number of categories. In this example, n = 64, indicating that the total number of training samples in a batch is 64; C = 4, including 3 abnormal categories and 1 normal category.

[0036] Furthermore, the calculation formula for image target detection loss is: ;

[0037] Where, represents the image target detection loss; Represents the target classification loss. Its calculation method is commonly used in the YOLOV11 model and will not be repeated here; Represents the bounding box regression loss, and its calculation method is commonly used in the YOLOV11 model and will not be repeated here; represents the distribution focal loss, which is calculated as: ;

[0038] Where, Represents the predicted probability that the i-th sample output by the classification detection head of the YOLOV11 model belongs to the j-th category, Indicates the true foreign body category label (0 or 1) of the jth category corresponding to the i-th sample, and They represent the index for adjusting the weight of difficult and easy samples and the parameters for controlling the degree of influence of category distribution deviation, is the prior frequency distribution of the j-th category, , Indicates the number of times the jth category appears in the batch training.

[0039] Furthermore, the calculation formula of cross-modal classification loss is: ;

[0040] Where, represents the predicted probability that the i-th sample output by the cross-modal classifier belongs to the j-th category, Indicates the true abnormal state category of the jth category corresponding to the i-th sample (0 or 1, with the real audio category label as the true abnormal state category mentioned here).

[0041] Furthermore, the calculation formula of the joint loss function is: ;

[0042] Where, represents the joint loss function, 、 and Represent the weight coefficients of audio classification loss, image target detection loss and cross-modal classification loss respectively.

[0043] (3) In the inference and use phase, the audio signal and image data of the belt are collected in real time during operation, and the trained multimodal fusion model is used to obtain the abnormality detection results and the coordinate information of the foreign object position. When an abnormality is detected, an alarm is triggered and a maintenance work order is generated. At the same time, the abnormality detection results and the coordinate information of the foreign object position are input into the Large Language Model (LLM) to generate maintenance suggestions, such as Figure 3 shown.

[0044] (3.1) Real-time audio signals and image data are collected during belt operation. The audio signals are de-noised and converted into Mel-spectrograms. The image data is then input into a trained multimodal fusion model to obtain predicted audio categories, foreign object categories, foreign object bounding boxes, and joint abnormal state categories. The Mel-spectrogram is passed through a ResNet-34 and audio classifier to obtain the predicted audio category. The image data is passed through a YOLO V11 model to obtain the predicted foreign object category and foreign object bounding box. The cross-modal feature fusion module and cross-modal classifier are used to obtain the predicted joint abnormal state category.

[0045] (3.2) Perform threshold judgment on the audio category probability and the joint abnormal state category probability output by the multimodal fusion model. For example, a fixed threshold of 0.7 can be used. When both the audio category probability and the joint abnormal state category probability exceed the preset threshold, an alarm will be issued and a maintenance work order will be generated.

[0046] (3.3) Input the abnormal condition category requiring an alarm and the foreign object location coordinates into the large language model to generate maintenance recommendations. Specifically, according to the set prompt word template, the abnormal condition category requiring an alarm is used as the abnormality detection result and the foreign object location coordinates are entered into the prompt word template. The created prompt word is input into the large language model to generate maintenance recommendations. The abnormal condition category requiring an alarm is determined based on the threshold judgment in step (3.2); the foreign object location coordinates are determined based on the predicted foreign object bounding box.

[0047] Furthermore, the prompt word template is: "You are a belt maintenance worker. The belt is now in the following abnormal state and a foreign object is detected on the belt. The following is the abnormal state information: <>. Please provide maintenance plans and suggestions based on this information." Among them, the abnormal state category and foreign object location coordinate information that need to be alarmed are filled in <>. The created prompt word is input into the large language model to return the corresponding maintenance suggestions. Figure 4 As shown, an example of a prompt word is: "You are a belt maintenance worker. The belt is now in the following abnormal state and a foreign object is detected on the belt. The following is the abnormal state information: the belt abnormality type is tearing, and the foreign object location coordinates are <23,567>, <67,456>. Please provide maintenance plans and suggestions based on this information." Correspondingly, the maintenance suggestions generated by the large language model are as follows: Figure 4 shown.

[0048] Corresponding to the aforementioned embodiment of the belt abnormality monitoring method based on a multi-modal large model, the present invention also provides an embodiment of a belt abnormality monitoring device based on a multi-modal large model.

[0049] See also Figure 5 An embodiment of the present invention provides a belt abnormality monitoring device based on a multimodal large model, including one or more processors and a memory, and the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the belt abnormality monitoring method based on the multimodal large model in the above embodiment.

[0050] The embodiment of the belt anomaly monitoring device based on the multimodal large model of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where the belt abnormality monitoring device based on the multi-modal large model of the present invention is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0051] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0052] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0053] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the belt abnormality monitoring method based on a multimodal large model in the above embodiment is implemented.

[0054] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A belt abnormality monitoring method based on a multimodal large model, characterized in that: Specifically include: At the same time, the audio signal and image data of the belt are collected and preprocessed to construct a data set; Construct a multimodal fusion model and use the dataset to train the multimodal fusion model. During the training process, the model parameters are optimized through the joint loss function to obtain a trained multimodal fusion model. During the inference and use phase, audio signals and image data of the belt are collected in real time during operation, and the trained multimodal fusion model is used to obtain anomaly detection results and foreign object location coordinate information. When an anomaly is detected, an alarm is triggered and a maintenance work order is generated. At the same time, the anomaly detection results and foreign object location coordinate information are input into the large language model to generate maintenance suggestions.

2. The belt abnormality monitoring method based on a multimodal large model according to claim 1 is characterized in that: The simultaneous collection of audio signals and image data during belt operation specifically includes: The audio signal of the belt during operation is collected through the sound sensor, the image data of the belt during operation is collected synchronously through the industrial camera, and the audio signal and image data are time-stamped and aligned through the time synchronization module.

3. The belt abnormality monitoring method based on multimodal large model according to claim 2 is characterized in that: The time synchronization module uses a hardware trigger signal to ensure synchronization of the acquisition frequencies of the audio signal and image data. The sampling rate of the audio signal is 44.1kHz, and the acquisition frame rate of the image data is 30fps. Timestamp alignment is used to ensure that each frame of image data matches the corresponding audio signal segment.

4. The belt abnormality monitoring method based on a multimodal large model according to claim 1 is characterized in that: The pretreatment specifically includes: For audio signals, a filtering method is used to remove ambient noise to obtain a denoised audio signal. The denoised audio signal is then converted into a Mel-spectrogram and labeled with audio category labels. The filtering method is wavelet transform. Audio categories include normal, belt breakage, foreign object jamming, and friction noise. For the image data, the foreign body category labels are annotated, and the foreign body coordinate frames are also annotated; among them, the foreign body categories include no foreign body category, ore accumulation foreign body category, metal debris foreign body category, and belt surface crack foreign body category.

5. The belt abnormality monitoring method based on multimodal large model according to claim 1 is characterized in that: The multimodal fusion model includes an audio feature extraction network, an audio classifier, an image feature extraction network, an image detection module, a first multi-layer perceptron module, a second multi-layer perceptron module, a cross-modal feature fusion module and a cross-modal classifier; Among them, the audio feature extraction network adopts ResNet-34 to extract Mel-spectrogram features; the audio classifier is used to predict the audio category corresponding to the Mel-spectrogram features; the image feature extraction network adopts the backbone network part of the YOLOV11 model to extract image features; the image detection module adopts the neck network part and detection head of the YOLOV11 model to predict the foreign object category and foreign object bounding box corresponding to the image features; the first multi-layer perceptron module and the second multi-layer perceptron module both adopt three fully connected layers, and each fully connected layer is connected to a batch normalization layer. The first multi-layer perceptron module is used to obtain the audio feature vector corresponding to the Mel-spectrogram feature, and the second multi-layer perceptron module is used to obtain the image feature vector corresponding to the image feature; the cross-modal feature fusion module adopts a cross-attention mechanism to perform weighted aggregation on the audio feature vector and the image feature vector to obtain the cross-modal fusion feature; the cross-modal classifier is used to predict the joint abnormal state category corresponding to the cross-modal fusion feature.

6. The belt abnormality monitoring method based on multimodal large model according to claim 1 is characterized in that: The multimodal fusion model is trained using the dataset, specifically including: The Mel-spectrogram and image data in the dataset are input into the multimodal fusion model to obtain the predicted audio category, foreign object category, foreign object bounding box and joint abnormal state category; the audio classification loss is calculated based on the predicted audio category and its corresponding true audio category label; the target classification loss and distribution focus loss are calculated based on the predicted foreign object category and its corresponding true foreign object category label, the bounding box regression loss is calculated based on the predicted foreign object bounding box and its corresponding true foreign object coordinate box, and the sum of the target classification loss, bounding box regression loss and distribution focus loss is calculated as the image target detection loss; the cross-modal classification loss is calculated based on the predicted joint abnormal state category and its corresponding true abnormal state category; the joint loss function is calculated based on the audio classification loss, image target detection loss and cross-modal classification loss; with minimizing the joint loss function as the optimization goal, the model parameters of the multimodal fusion model are optimized using the stochastic gradient descent method until the preset training rounds are reached to obtain a trained multimodal fusion model.

7. The belt abnormality monitoring method based on multimodal large model according to claim 6 is characterized in that: The calculation formula of the distributed focus loss is: ； Where, represents the distribution focal loss, Represents the predicted probability that the i-th sample output by the classification detection head of the YOLOV11 model belongs to the j-th category, Indicates the true foreign body category label of the i-th sample corresponding to the j-th category, and They represent the index for adjusting the weight of difficult and easy samples and the parameters for controlling the degree of influence of category distribution deviation, is the prior frequency distribution of the j-th category, , Indicates the number of times the jth category appears in the batch training.

8. The belt abnormality monitoring method based on multimodal large model according to claim 1 is characterized in that: The reasoning and use phase specifically includes: The audio signal and image data of the belt are collected in real time during operation. The audio signal is converted into a Mel-spectrogram after denoising, and the image data is input into the trained multimodal fusion model to obtain the predicted audio category, foreign object category and foreign object bounding box, and joint abnormal state category. Threshold judgment is performed on the audio category probability and the joint abnormal state category probability output by the multimodal fusion model. When both the audio category probability and the joint abnormal state category probability exceed the preset threshold, an alarm is issued and a maintenance work order is generated; According to the set prompt word template, the abnormal state category that needs to be alarmed is filled into the prompt word template as the abnormal detection result and the foreign object location coordinate information. The created prompt word is input into the large language model to generate maintenance suggestions.

9. A belt anomaly monitoring device based on a multimodal large model, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the belt abnormality monitoring method based on a multimodal large model according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, it is used to implement the belt abnormality monitoring method based on a multimodal large model as described in any one of claims 1-8.

Images