A belt conveyor coal flow foreign matter detection method based on a multi-modal visual language large model

Abstract

The application discloses a belt conveyor coal flow foreign matter detection method based on a multi-modal visual language large model, and is characterized in that the method comprises the following steps: foreign matter preliminary detection: a picture-text response mapping unit is generated through a foreign matter semantic prior generation unit and a coal flow image layered representation unit, and then a semantic weight modulation unit and a candidate foreign matter extraction unit are constructed to respectively realize preliminary detection and candidate target output; semantic review and timing confirmation: a continuous frame consistency confirmation unit is generated through a candidate target layered re-determination unit and a working condition perception fusion unit, and then a timing confidence smoothing unit and a feature memory updating unit are constructed to respectively realize online learning and feature library updating; and a foreign matter detection result is obtained. The method can improve the detection accuracy, detection stability and continuous adaptability of the coal flow foreign matter under complex working conditions.

Description

Technical Field

[0001] This invention relates to a method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model, belonging to the technical field of mining belt conveyors. Background Technology

[0002] Belt conveyors are commonly used continuous conveying equipment in coal mines, ports, and thermal power plants. During operation, the coal flow often contains foreign objects such as metal parts, wood blocks, stones, ropes, and fabric. Once these foreign objects enter the conveying system, they can easily cause belt scratches and tears, idler jamming, equipment blockage, and even safety accidents. Therefore, timely detection of foreign objects in the coal flow is of great importance.

[0003] Existing methods for detecting foreign objects in coal flow mainly rely on machine vision technology. These methods acquire images of the coal flow using industrial cameras and then utilize object detection networks to identify and locate foreign objects. This type of method is characterized by its ease of deployment and fast response time, and has already been applied in some industrial scenarios. With the development of deep learning technology, lightweight single-stage object detection methods are gradually becoming a common solution in industrial online inspection due to their better real-time performance. Meanwhile, multimodal visual language models with image-text joint understanding capabilities have also demonstrated strong scene semantic analysis capabilities, providing a new technical approach for complex foreign object identification.

[0004] However, the foreign object detection scenario in coal flow presents significant industrial complexity. On the one hand, coal blocks and foreign objects share certain similarities in color, texture, and shape; on the other hand, factors such as dust obstruction, changes in illumination, motion blur, and irregular foreign object postures can significantly interfere with image quality and target discrimination. Under these conditions, existing technologies still struggle to simultaneously achieve real-time detection, recognition accuracy, and output stability. This is mainly manifested in the following ways: detection methods based on single visual features rely heavily on target appearance information for discrimination, failing to adequately utilize the semantics and hazardous attributes of foreign object categories, and are prone to false positives and false negatives when coal blocks and foreign objects are similar in appearance; large multimodal models with strong semantic understanding capabilities typically have high computational costs, making them unsuitable for meeting the timeliness requirements of online detection in high-speed coal flow scenarios; insufficient collaborative utilization of various judgment information under different operating conditions, lacking a judgment mechanism that can automatically adjust with belt speed, illumination changes, and coal flow disturbances; and existing methods output based on single-frame results, lacking utilization of the continuous motion characteristics of foreign objects, resulting in insufficient stability of detection results under conditions of rapid passage, partial occlusion, and image jitter. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model. This method can improve the detection accuracy, detection stability and continuous adaptability of foreign objects in coal flow under complex working conditions.

[0006] To achieve the above objectives, this invention provides a method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model, comprising the following steps:

[0007] S1. Foreign object initial detection: The image and text response mapping unit is generated by the foreign object semantic prior generation unit and the coal flow image hierarchical representation unit. Then, the semantic weight modulation unit and the candidate foreign object extraction unit are constructed to realize the initial detection and candidate target output respectively.

[0008] S2, Semantic verification and temporal confirmation: A continuous frame consistency confirmation unit is generated through the candidate target hierarchical review unit and the working condition perception fusion unit, and then a temporal confidence smoothing unit and a feature memory update unit are constructed to realize online learning and feature library update respectively;

[0009] S3. Obtain the foreign object detection results.

[0010] Furthermore, the specific process of S1 is as follows:

[0011] S1.1 Constructing a priori semantic generation unit for foreign objects:

[0012] To enable the detection system to have prior knowledge of the types of foreign objects in the coal flow, a set of text prompts corresponding to the foreign object scenarios is first established. This set includes metal, wood, rope, stone, and cloth foreign objects, and can be expanded to other hazardous foreign object categories based on the actual situation on site. The text prompt set is represented as follows: Where T represents the set of semantic cues for foreign objects; Let K represent the k-th foreign object cue word; K represents the total number of cue words; input each cue word into the text encoding terminal of the multimodal visual language model to obtain the corresponding text semantic representation, the expression of which is:

[0013] ;

[0014] In the formula, This represents the text embedding vector obtained by the text encoder for the k-th cue word; This represents the text encoding function in a multimodal visual language model. Considering that text features and visual features are usually inconsistent in dimensional space, a linear mapping layer is introduced to project the text embedding onto the detector's visual feature space for subsequent image-text association calculations. Its expression is:

[0015] ;

[0016] In the formula, This represents the k-th text projection vector obtained after linear mapping; This represents the text feature projection weight matrix; Indicates the projection layer bias term;

[0017] S1.2 Constructing hierarchical representation units for coal flow images:

[0018] The coal flow image acquired on-site is input into the lightweight single-stage target detector. Let the input coal flow image be... ,in, This represents the input coal flow image; H represents the real space in which the image tensor resides; W represents the image height; 3 represents the image width; and 3 represents the number of color channels in the image.

[0019] The detector's backbone network and feature pyramid structure perform layer-by-layer analysis of the input image, extracting visual feature maps at different scales to simultaneously preserve the surface texture of the coal flow, the target contour boundary, and the high-level semantic structure. Its expression is as follows:

[0020] ;

[0021] In the formula, Indicates the first The visual feature map output by the layer; Indicates the first The height of the layer feature map; Indicates the first Width of the layer feature map; Indicates the number of visual feature channels; For feature layer index; Indicates the total number of feature layers;

[0022] Since foreign objects in coal flow often resemble the coal block itself in appearance, it is difficult to achieve stable differentiation relying solely on single-scale features. Therefore, multi-layer feature output is retained to enhance the representation of small-sized foreign objects, partially occluded foreign objects, and irregularly shaped foreign objects. A unified visual feature map is obtained after multi-scale integration:

[0023] ;

[0024] In the formula, This represents the unified visual feature map obtained after multi-scale integration. Indicates the height of the uniform feature map; Indicates the width of the uniform feature map;

[0025] S1.3, Generating graphic response mapping unit:

[0026] To achieve an effective correspondence between text semantics and visual regions, visual features from different scales are first unified to the same spatial resolution. Then, a unified visual feature representation is formed through channel concatenation and convolution. Subsequently, the similarity between the text projection vector corresponding to each type of foreign object cue and the local features at each spatial location on the unified visual feature map is calculated to obtain the response intensity of that location to different foreign object concepts. The expression is as follows:

[0027] ;

[0028] In the formula, This represents the spatial location of the k-th text projection vector. The corresponding original semantic response value; Indicates the transpose operation; Indicates the location of the unified visual feature map The channel vector at the location; i represents the vertical position index of the feature map; j represents the horizontal position index of the feature map;

[0029] The above dot product form can be further written as a channel-wise summation form as follows:

[0030] ;

[0031] In the formula, c is the channel index; This represents the component of the k-th text projection vector in the c-th channel; Indicates the location of the unified visual feature map Feature value on the c-th channel;

[0032] To highlight the dominant semantic features of the suspected foreign object region, the strongest response value is selected on the category dimension to represent the most relevant semantic relationship between this spatial location and all foreign object concepts. Its expression is as follows:

[0033] ;

[0034] In the formula, Indicates spatial location The comprehensive semantic guide value at the location; max indicates the maximum value operation;

[0035] S1.4 Constructing the semantic weight modulation unit:

[0036] Because the numerical distribution of image-text similarity results varies significantly across different images and working conditions, directly using them as weights can easily cause oscillations during training or unstable inference output. Therefore, this unit standardizes and activates the comprehensive semantic response map to form a stable spatial weight matrix. First, the global mean of the comprehensive semantic guidance map is calculated:

[0037] ;

[0038] In the formula, This represents the global mean of the comprehensive semantic guidance graph; Indicates the height of the comprehensive semantic guidance graph; Indicates the width of the comprehensive semantic guidance graph; This represents the total number of spatial locations in the guide diagram;

[0039] Next, calculate the global standard deviation of the integrated semantic guidance map:

[0040] ;

[0041] In the formula, This represents the global standard deviation of the integrated semantic guidance graph; Represents the square root operation;

[0042] Standardize the comprehensive semantic guidance graph:

[0043] ;

[0044] In the formula, Indicates position The standardized semantic response value;

[0045] Finally, the final semantic weights are obtained through the Sigmoid activation function:

[0046] ;

[0047] In the formula, Indicates position The semantic weight value at that location; This represents the Sigmoid activation function; Represents the natural exponential function;

[0048] S1.5 Constructing candidate foreign object extraction units:

[0049] After obtaining the semantic modulation weights, these weights are applied to the unified visual feature map to enhance suspected foreign object regions and suppress the pure coal flow background, thereby forming a semantically enhanced feature map, the expression of which is:

[0050] ;

[0051] In the formula, This indicates that the enhanced feature map is located at... Feature value on the c-th channel;

[0052] After the enhanced feature map is input into the detection head, the bounding box coordinates of the candidate targets, the target confidence score, and the category prediction result are output, realizing the first stage of foreign object initial detection. The set of candidate targets in the nth frame image is represented as:

[0053] ;

[0054] In the formula, This represents the set of candidate targets in the nth frame image; This represents the m-th candidate target in the n-th frame; m is the index of the candidate target. This represents the total number of candidate targets in the nth frame; n represents the video frame index.

[0055] Wherein, a single candidate target is written as:

[0056] ;

[0057] In the formula, and These represent the x and y coordinates of the center point of the candidate box, respectively. and These represent the width and height of the candidate box, respectively. This represents the confidence level of the candidate target output by the detector; This indicates the candidate target category output by the detector.

[0058] Furthermore, the specific process of S2 is as follows:

[0059] S2.1 Constructing a Hierarchical Reassessment Unit for Candidate Targets: Considering the presence of numerous texture confusions, local occlusions, and lighting disturbances in the coal flow scene, misjudgments may still occur relying solely on the initial detection results. Therefore, a hierarchical reassessment unit for candidate targets is established. This unit segments the candidate boxes based on the confidence levels of the candidate boxes output by the detector, with the following threshold relationship:

[0060] ;

[0061] In the formula, Indicates a low threshold; Indicates the threshold value; Indicates a high threshold;

[0062] For candidate targets entering the review process, the corresponding regions are extracted from the original image, and inference is performed using visual question-and-answer prompts. The visual semantic feature vector of the candidate region is represented as follows:

[0063] ;

[0064] In the formula, This represents the semantic feature vector obtained by visual encoding the m-th candidate region in the n-th frame; Represents the visual encoding function in a large-scale multimodal visual language model; Represents the original image from the nth frame. Middle of the candidate box The cropped region image;

[0065] The multimodal visual language large model outputs the presence markers, semantic categories, and hazard level scores of candidate regions, along with corresponding semantic confidence scores, expressed as follows:

[0066] ;

[0067] In the formula, This represents the semantic confidence score of the m-th candidate target in the n-th frame after multimodal re-judgment. This represents the category probability distribution output by a large multimodal visual language model.

[0068] S2.2 Constructing a working condition perception fusion unit:

[0069] After obtaining the initial detection results and multimodal re-judgment results from the detector, the credibility of the two types of evidence is determined based on the on-site working conditions. By setting up a working condition perception fusion unit, the belt speed, ambient illuminance, and coal flow parameters are combined to form a working condition vector, the expression of which is:

[0070] ;

[0071] In the formula, This represents the vector of operating parameters at time n. Represents a column vector; Indicates the belt speed; Indicates ambient illuminance; Indicates coal flow rate; Indicates dust concentration; Indicates image sharpness; This indicates the vibration intensity of the equipment; among them, dust concentration, image clarity, and vibration intensity are optional parameters and can be added selectively.

[0072] After standardizing the operating condition parameter vector, we obtain:

[0073] ;

[0074] In the formula, This represents the standardized operating condition parameter vector; A vector representing the mean values ​​of the operating parameters; A vector representing the standard deviation of operating parameters;

[0075] The standardized operating condition parameter vector is input into a multilayer perceptron network, and the output of the first hidden layer is represented as:

[0076] ;

[0077] In the formula, This represents the output vector of the first hidden layer; Represents the linear rectification activation function; This indicates a batch normalization operation; This represents the first-layer weight matrix; Indicates the first-level bias term;

[0078] The output of the second hidden layer is represented as:

[0079] ;

[0080] In the formula, This represents the output vector of the second hidden layer; Indicates a random deactivation operation; This represents the weight matrix of the second layer; Indicates the second-level bias term;

[0081] The fusion weights for the two categories of decisions are obtained through Softmax normalization:

[0082] ;

[0083] In the formula, This represents the fusion weights corresponding to the detector confidence levels; This represents the fusion weight corresponding to the multimodal re-judgment confidence level; Represents the normalization function; This represents the third-layer weight matrix; Indicates the third-level bias term;

[0084] The two types of fusion weights satisfy:

[0085] ;

[0086] Based on this, the final fusion confidence of the candidate targets is obtained:

[0087] ;

[0088] In the formula, This represents the final fusion confidence of the m-th candidate target in the n-th frame;

[0089] S2.3, Generation of continuous frame consistency confirmation unit:

[0090] Because foreign objects exhibit continuous motion on the conveyor belt, single-frame results are often insufficient to support stable alarms. Therefore, a continuous frame consistency verification unit is introduced. This unit correlates the trajectories of candidate targets in adjacent frames within a sliding time window. A sequence of N consecutive frames within this window is represented as follows:

[0091] ;

[0092] In the formula, This represents a sequence of N consecutive frames within a sliding time window; N represents the length of the time window.

[0093] Extract the local representation vector of each candidate box from the enhanced feature map, denoted as: , This represents the local region feature vector of the m-th candidate target in the n-th frame; Indicates the dimension of regional features;

[0094] To further introduce semantic constraints, a set of cross-frame related semantic cues is constructed:

[0095] ;

[0096] In the formula, Represents a set of semantic cues used for cross-frame association; This represents the p-th associated semantic cue; p is the cue index; P represents the total number of associated semantic cuees;

[0097] The corresponding text embedding vector is represented as The semantic similarity component of the candidate target relative to the p-th associated semantic cue is represented as:

[0098] ;

[0099] The above formula represents the cosine similarity between the two, where, This represents the semantic similarity score between the m-th candidate target and the p-th associated semantic cue in the n-th frame; The norm of the region's eigenvectors; The norm of the text embedding vector;

[0100] Combining appearance and semantic information, a comprehensive correlation matrix is ​​constructed between candidate targets across frames, with the element expression as follows:

[0101] ;

[0102] In the formula, Indicates the first The first frame The semantic feature vector of each candidate target (composed of the similarity components of the target to each associated semantic cues). Indicates the first The first frame Semantic feature vectors of candidate targets; This represents the overall correlation between the a-th candidate target in the i-th frame and the b-th candidate target in the j-th frame; Weighting coefficients representing the similarity of regional appearances; Weight coefficients representing semantic similarity; Represents the cosine similarity function;

[0103] S2.4 Constructing a time-series confidence smoothing unit:

[0104] After completing cross-frame association, to further suppress confidence fluctuations caused by belt vibration, image jitter, and momentary occlusion, temporal smoothing is performed on the trajectory-level detection results. The expression is as follows:

[0105] ;

[0106] In the formula, This represents the smooth confidence level at the current moment; This represents the smooth confidence level at the previous time step; This represents the smoothing coefficient, with a value ranging from 0 to 1; This indicates the weight of the current observation;

[0107] S2.5 Constructing Feature Memory Update Units:

[0108] To ensure the system maintains its identification capability under conditions of coal type change, lighting variation, conveyor belt wear, and long-term field drift, a feature memory update unit is implemented. This unit maintains the foreign object feature database, and its expression is:

[0109] ;

[0110] In the formula, G represents the foreign object feature library; This represents the feature vector of the g-th template in the library; This represents the category identifier corresponding to the template feature; g represents the template index;

[0111] For the new sample features confirmed by reassessment, the original template is modified using a momentum update method, expressed as:

[0112] ;

[0113] In the formula, This represents the updated g-th template feature; This represents the g-th template feature before the update; This represents the feature vector of the new input sample; This represents the momentum update coefficient.

[0114] This invention improves the ability to distinguish foreign object targets in coal flow scenarios by introducing semantic information related to foreign objects during the detection process. This is achieved through the synergistic effect of semantic information and visual features, enhancing detection performance under conditions where coal blocks and foreign objects are confused. By combining rapid candidate detection with semantic verification and adaptively fusing judgment results from different sources based on on-site conditions, the invention avoids applying the same processing method to all candidate targets while meeting the needs of industrial online detection. This improves the reliability of candidate target judgment results and enhances recognition accuracy under complex interference conditions, while maintaining overall processing efficiency. Furthermore, by utilizing the correlation between consecutive frames to confirm the consistency of detection results and combining this with a temporal smoothing strategy to output alarm results, the invention reduces the impact of abnormal fluctuations in a single frame on the system output, improving the continuity and stability of foreign object detection. Finally, by setting a feature memory update mechanism, the invention adjusts and supplements the foreign object representation based on new samples appearing during operation, enabling the system to gradually adapt to changes in coal type, environment, and equipment status, thereby improving its adaptability to long-term changes in operating conditions. Attached Figure Description

[0115] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0116] The invention will now be further described with reference to the accompanying drawings.

[0117] like Figure 1 As shown, a method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model includes the following steps:

[0118] S1. Foreign object initial detection: The image and text response mapping unit is generated by the foreign object semantic prior generation unit and the coal flow image hierarchical representation unit. Then, the semantic weight modulation unit and the candidate foreign object extraction unit are constructed to realize the initial detection and candidate target output respectively.

[0119] S2, Semantic verification and temporal confirmation: A continuous frame consistency confirmation unit is generated through the candidate target hierarchical review unit and the working condition perception fusion unit, and then a temporal confidence smoothing unit and a feature memory update unit are constructed to realize online learning and feature library update respectively;

[0120] S3. Obtain the foreign object detection results.

[0121] As a preferred embodiment, the specific process of S1 is as follows:

[0122] S1.1 Constructing a priori semantic generation unit for foreign objects:

[0123] To enable the detection system to have prior knowledge of the types of foreign objects in the coal flow, a set of text prompts corresponding to the foreign object scenarios is first established. This set includes metal, wood, rope, stone, and cloth foreign objects, and can be expanded to other hazardous foreign object categories based on the actual situation on site. The text prompt set is represented as follows: Where T represents the set of semantic cues for foreign objects; Let K represent the k-th foreign object cue word; K represents the total number of cue words; input each cue word into the text encoding terminal of the multimodal visual language model to obtain the corresponding text semantic representation, the expression of which is:

[0124] ;

[0125] In the formula, This represents the text embedding vector obtained by the text encoder for the k-th cue word; This represents the text encoding function in a multimodal visual language model. Considering that text features and visual features are usually inconsistent in dimensional space, a linear mapping layer is introduced to project the text embedding onto the detector's visual feature space for subsequent image-text association calculations. Its expression is:

[0126] ;

[0127] In the formula, This represents the k-th text projection vector obtained after linear mapping; This represents the text feature projection weight matrix; Indicates the projection layer bias term;

[0128] S1.2 Constructing hierarchical representation units for coal flow images:

[0129] The coal flow image acquired on-site is input into the lightweight single-stage target detector. Let the input coal flow image be... ,in, This represents the input coal flow image; H represents the real space in which the image tensor resides; W represents the image height; 3 represents the image width; and 3 represents the number of color channels in the image.

[0130] The detector's backbone network and feature pyramid structure perform layer-by-layer analysis of the input image, extracting visual feature maps at different scales to simultaneously preserve the surface texture of the coal flow, the target contour boundary, and the high-level semantic structure. Its expression is as follows:

[0131] ;

[0132] In the formula, Indicates the first The visual feature map output by the layer; Indicates the first The height of the layer feature map; Indicates the first Width of the layer feature map; Indicates the number of visual feature channels; For feature layer index; Indicates the total number of feature layers;

[0133] Since foreign objects in coal flow often resemble the coal block itself in appearance, it is difficult to achieve stable differentiation relying solely on single-scale features. Therefore, multi-layer feature output is retained to enhance the representation of small-sized foreign objects, partially occluded foreign objects, and irregularly shaped foreign objects. A unified visual feature map is obtained after multi-scale integration:

[0134] ;

[0135] In the formula, This represents the unified visual feature map obtained after multi-scale integration. Indicates the height of the uniform feature map; Indicates the width of the uniform feature map;

[0136] S1.3, Generating graphic response mapping unit:

[0137] To achieve an effective correspondence between text semantics and visual regions, visual features from different scales are first unified to the same spatial resolution. Then, a unified visual feature representation is formed through channel concatenation and convolution. Subsequently, the similarity between the text projection vector corresponding to each type of foreign object cue and the local features at each spatial location on the unified visual feature map is calculated to obtain the response intensity of that location to different foreign object concepts. The expression is as follows:

[0138] ;

[0139] In the formula, This represents the spatial location of the k-th text projection vector. The corresponding original semantic response value; Indicates the transpose operation; Indicates the location of the unified visual feature map The channel vector at the location; i represents the vertical position index of the feature map; j represents the horizontal position index of the feature map;

[0140] The above dot product form can be further written as a channel-wise summation form as follows:

[0141] ;

[0142] In the formula, c is the channel index; This represents the component of the k-th text projection vector in the c-th channel; Indicates the location of the unified visual feature map Feature value on the c-th channel;

[0143] To highlight the dominant semantic features of the suspected foreign object region, the strongest response value is selected on the category dimension to represent the most relevant semantic relationship between this spatial location and all foreign object concepts. Its expression is as follows:

[0144] ;

[0145] In the formula, Indicates spatial location The comprehensive semantic guide value at the location; max indicates the maximum value operation;

[0146] S1.4 Constructing the semantic weight modulation unit:

[0147] Because the numerical distribution of image-text similarity results varies significantly across different images and working conditions, directly using them as weights can easily cause oscillations during training or unstable inference output. Therefore, this unit standardizes and activates the comprehensive semantic response map to form a stable spatial weight matrix. First, the global mean of the comprehensive semantic guidance map is calculated:

[0148] ;

[0149] In the formula, This represents the global mean of the comprehensive semantic guidance graph; Indicates the height of the comprehensive semantic guidance graph; Indicates the width of the comprehensive semantic guidance graph; This represents the total number of spatial locations in the guide diagram;

[0150] Next, calculate the global standard deviation of the integrated semantic guidance map:

[0151] ;

[0152] In the formula, This represents the global standard deviation of the integrated semantic guidance graph; Represents the square root operation;

[0153] Standardize the comprehensive semantic guidance graph:

[0154] ;

[0155] In the formula, Indicates position The standardized semantic response value;

[0156] Finally, the final semantic weights are obtained through the Sigmoid activation function:

[0157] ;

[0158] In the formula, Indicates position The semantic weight value at that location; This represents the Sigmoid activation function; Represents the natural exponential function;

[0159] S1.5 Constructing candidate foreign object extraction units:

[0160] After obtaining the semantic modulation weights, these weights are applied to the unified visual feature map to enhance suspected foreign object regions and suppress the pure coal flow background, thereby forming a semantically enhanced feature map, the expression of which is:

[0161] ;

[0162] In the formula, This indicates that the enhanced feature map is located at... Feature value on the c-th channel;

[0163] After the enhanced feature map is input into the detection head, the bounding box coordinates of the candidate targets, the target confidence score, and the category prediction result are output, realizing the first stage of foreign object initial detection. The set of candidate targets in the nth frame image is represented as:

[0164] ;

[0165] In the formula, This represents the set of candidate targets in the nth frame image; This represents the m-th candidate target in the n-th frame; m is the index of the candidate target. This represents the total number of candidate targets in the nth frame; n represents the video frame index.

[0166] Wherein, a single candidate target is written as:

[0167] ;

[0168] In the formula, and These represent the x and y coordinates of the center point of the candidate box, respectively. and These represent the width and height of the candidate box, respectively. This represents the confidence level of the candidate target output by the detector; This indicates the candidate target category output by the detector.

[0169] As a preferred embodiment, the specific process of S2 is as follows:

[0170] S2.1 Constructing a Hierarchical Reassessment Unit for Candidate Targets: Considering the presence of numerous texture confusions, local occlusions, and lighting disturbances in the coal flow scene, misjudgments may still occur relying solely on the initial detection results. Therefore, a hierarchical reassessment unit for candidate targets is established. This unit segments the candidate boxes based on the confidence levels of the candidate boxes output by the detector, with the following threshold relationship:

[0171] ;

[0172] In the formula, Indicates a low threshold; Indicates the threshold value; Indicates a high threshold;

[0173] For candidate targets entering the review process, the corresponding regions are extracted from the original image, and inference is performed using visual question-and-answer prompts. The visual semantic feature vector of the candidate region is represented as follows:

[0174] ;

[0175] In the formula, This represents the semantic feature vector obtained by visual encoding the m-th candidate region in the n-th frame; Represents the visual encoding function in a large-scale multimodal visual language model; Represents the original image from the nth frame. Middle of the candidate box The cropped region image;

[0176] The multimodal visual language large model outputs the presence markers, semantic categories, and hazard level scores of candidate regions, along with corresponding semantic confidence scores, expressed as follows:

[0177] ;

[0178] In the formula, This represents the semantic confidence score of the m-th candidate target in the n-th frame after multimodal re-judgment. This represents the category probability distribution output by a large multimodal visual language model.

[0179] S2.2 Constructing a working condition perception fusion unit:

[0180] After obtaining the initial detection results and multimodal re-judgment results from the detector, the credibility of the two types of evidence is determined based on the on-site working conditions. By setting up a working condition perception fusion unit, the belt speed, ambient illuminance, and coal flow parameters are combined to form a working condition vector, the expression of which is:

[0181] ;

[0182] In the formula, This represents the vector of operating parameters at time n. Represents a column vector; Indicates the belt speed; Indicates ambient illuminance; Indicates coal flow rate; Indicates dust concentration; Indicates image sharpness; This indicates the vibration intensity of the equipment; among them, dust concentration, image clarity, and vibration intensity are optional parameters and can be added selectively.

[0183] After standardizing the operating condition parameter vector, we obtain:

[0184] ;

[0185] In the formula, This represents the standardized operating condition parameter vector; A vector representing the mean values ​​of the operating parameters; A vector representing the standard deviation of operating parameters;

[0186] The standardized operating condition parameter vector is input into a multilayer perceptron network, and the output of the first hidden layer is represented as:

[0187] ;

[0188] In the formula, This represents the output vector of the first hidden layer; Represents the linear rectification activation function; This indicates a batch normalization operation; This represents the first-layer weight matrix; Indicates the first-level bias term;

[0189] The output of the second hidden layer is represented as:

[0190] ;

[0191] In the formula, This represents the output vector of the second hidden layer; Indicates a random deactivation operation; This represents the weight matrix of the second layer; Indicates the second-level bias term;

[0192] The fusion weights for the two categories of decisions are obtained through Softmax normalization:

[0193] ;

[0194] In the formula, This represents the fusion weights corresponding to the detector confidence levels; This represents the fusion weight corresponding to the multimodal re-judgment confidence level; Represents the normalization function; This represents the third-layer weight matrix; Indicates the third-level bias term;

[0195] The two types of fusion weights satisfy:

[0196] ;

[0197] Based on this, the final fusion confidence of the candidate targets is obtained:

[0198] ;

[0199] In the formula, This represents the final fusion confidence of the m-th candidate target in the n-th frame;

[0200] S2.3, Generation of continuous frame consistency confirmation unit:

[0201] Because foreign objects exhibit continuous motion on the conveyor belt, single-frame results are often insufficient to support stable alarms. Therefore, a continuous frame consistency verification unit is introduced. This unit correlates the trajectories of candidate targets in adjacent frames within a sliding time window. A sequence of N consecutive frames within this window is represented as follows:

[0202] ;

[0203] In the formula, This represents a sequence of N consecutive frames within a sliding time window; N represents the length of the time window.

[0204] Extract the local representation vector of each candidate box from the enhanced feature map, denoted as: , This represents the local region feature vector of the m-th candidate target in the n-th frame; Indicates the dimension of regional features;

[0205] To further introduce semantic constraints, a set of cross-frame related semantic cues is constructed:

[0206] ;

[0207] In the formula, Represents a set of semantic cues used for cross-frame association; This represents the p-th associated semantic cue; p is the cue index; P represents the total number of associated semantic cuees;

[0208] The corresponding text embedding vector is represented as The semantic similarity component of the candidate target relative to the p-th associated semantic cue is represented as:

[0209] ;

[0210] The above formula represents the cosine similarity between the two, where, This represents the semantic similarity score between the m-th candidate target and the p-th associated semantic cue in the n-th frame; The norm of the region's eigenvectors; The norm of the text embedding vector;

[0211] Combining appearance and semantic information, a comprehensive correlation matrix is ​​constructed between candidate targets across frames, with the element expression as follows:

[0212] ;

[0213] In the formula, Indicates the first The first frame The semantic feature vector of each candidate target (composed of the similarity components of the target to each associated semantic cues). Indicates the first The first frame Semantic feature vectors of candidate targets; This represents the overall correlation between the a-th candidate target in the i-th frame and the b-th candidate target in the j-th frame; Weighting coefficients representing the similarity of regional appearances; Weight coefficients representing semantic similarity; Represents the cosine similarity function;

[0214] S2.4 Constructing a time-series confidence smoothing unit:

[0215] After completing cross-frame association, to further suppress confidence fluctuations caused by belt vibration, image jitter, and momentary occlusion, temporal smoothing is performed on the trajectory-level detection results. The expression is as follows:

[0216] ;

[0217] In the formula, This represents the smooth confidence level at the current moment; This represents the smooth confidence level at the previous time step; This represents the smoothing coefficient, with a value ranging from 0 to 1; This indicates the weight of the current observation;

[0218] S2.5 Constructing Feature Memory Update Units:

[0219] To ensure the system maintains its identification capability under conditions of coal type change, lighting variation, conveyor belt wear, and long-term field drift, a feature memory update unit is implemented. This unit maintains the foreign object feature database, and its expression is:

[0220] ;

[0221] In the formula, G represents the foreign object feature library; This represents the feature vector of the g-th template in the library; This represents the category identifier corresponding to the template feature; g represents the template index;

[0222] For the new sample features confirmed by reassessment, the original template is modified using a momentum update method, expressed as:

[0223] ;

[0224] In the formula, This represents the updated g-th template feature; This represents the g-th template feature before the update; This represents the feature vector of the new input sample; This represents the momentum update coefficient.

[0225] The method of this invention is based on several processing units. Each unit unfolds sequentially around coal flow image input, semantic information injection, candidate target generation, semantic verification, result fusion, temporal confirmation, and online updating, forming a complete foreign object detection process. In the example, the multimodal visual language large model adopts Qwen-VL, and the lightweight single-stage target detector adopts YOLO11. However, this invention is not limited to the above model selection, and other models with the same functions can also be applied.

Claims

1. A method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model, characterized in that, Includes the following steps: S1. Foreign object initial detection: The image and text response mapping unit is generated by the foreign object semantic prior generation unit and the coal flow image hierarchical representation unit. Then, the semantic weight modulation unit and the candidate foreign object extraction unit are constructed to realize the initial detection and candidate target output respectively. S2, Semantic verification and temporal confirmation: A continuous frame consistency confirmation unit is generated through the candidate target hierarchical review unit and the working condition perception fusion unit, and then a temporal confidence smoothing unit and a feature memory update unit are constructed to realize online learning and feature library update respectively; S3. Obtain the foreign object detection results.

2. The method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model according to claim 1, characterized in that, The specific process of S1 is as follows: S1.1 Constructing a priori semantic generation unit for foreign objects: Establish a set of text prompts corresponding to foreign object scenarios. The prompt set includes metal foreign objects, wooden foreign objects, rope-like foreign objects, stone-like foreign objects, and cloth-like foreign objects, and can be expanded to other hazardous foreign object categories based on the actual situation on site. The text prompt set is represented as follows: Where T represents the set of semantic cues for foreign objects; Let K represent the k-th foreign object cue word; K represents the total number of cue words; input each cue word into the text encoding terminal of the multimodal visual language model to obtain the corresponding text semantic representation, the expression of which is: ； In the formula, This represents the text embedding vector obtained by the text encoder for the k-th cue word; This represents the text encoding function in a multimodal visual language model. Considering that text features and visual features are usually inconsistent in dimensional space, a linear mapping layer is introduced to project the text embedding onto the detector's visual feature space for subsequent image-text association calculations. Its expression is: ； In the formula, This represents the k-th text projection vector obtained after linear mapping; This represents the text feature projection weight matrix; Indicates the projection layer bias term; S1.2 Constructing hierarchical representation units for coal flow images: The coal flow image acquired on-site is input into the lightweight single-stage target detector. Let the input coal flow image be... ,in, This represents the input coal flow image; H represents the real space in which the image tensor resides; W represents the image height; 3 represents the image width; and 3 represents the number of color channels in the image. The detector's backbone network and feature pyramid structure perform layer-by-layer analysis of the input image, extracting visual feature maps at different scales to simultaneously preserve the surface texture of the coal flow, the target contour boundary, and the high-level semantic structure. Its expression is as follows: ； In the formula, Indicates the first The visual feature map output by the layer; Indicates the first The height of the layer feature map; Indicates the first Width of the layer feature map; Indicates the number of visual feature channels; For feature layer index; Indicates the total number of feature layers; A unified visual feature map is obtained after multi-scale integration: ； In the formula, This represents the unified visual feature map obtained after multi-scale integration. Indicates the height of the uniform feature map; Indicates the width of the uniform feature map; S1.3, Generating graphic response mapping unit: To achieve an effective correspondence between text semantics and visual regions, visual features from different scales are first unified to the same spatial resolution. Then, a unified visual feature representation is formed through channel concatenation and convolution. Subsequently, the similarity between the text projection vector corresponding to each type of foreign object cue and the local features at each spatial location on the unified visual feature map is calculated to obtain the response intensity of that location to different foreign object concepts. The expression is as follows: ； In the formula, This represents the spatial location of the k-th text projection vector. The corresponding original semantic response value; Indicates the transpose operation; Indicates the location of the unified visual feature map The channel vector at the location; i represents the vertical position index of the feature map; j represents the horizontal position index of the feature map; The above dot product form can be further written as a channel-wise summation form as follows: ； In the formula, c is the channel index; This represents the component of the k-th text projection vector in the c-th channel; Indicates the location of the unified visual feature map Feature value on the c-th channel; To highlight the dominant semantic features of the suspected foreign object region, the strongest response value is selected on the category dimension to represent the most relevant semantic relationship between this spatial location and all foreign object concepts. Its expression is as follows: ； In the formula, Indicates spatial location The comprehensive semantic guide value at the location; max indicates the maximum value operation; S1.4 Constructing the semantic weight modulation unit: First, calculate the global mean of the comprehensive semantic guidance graph: ； In the formula, This represents the global mean of the comprehensive semantic guidance graph; Indicates the height of the comprehensive semantic guidance graph; Indicates the width of the comprehensive semantic guidance graph; This represents the total number of spatial locations in the guide diagram; Next, calculate the global standard deviation of the integrated semantic guidance map: ； In the formula, This represents the global standard deviation of the integrated semantic guidance graph; Represents the square root operation; Standardize the comprehensive semantic guidance graph: ； In the formula, Indicates position The standardized semantic response value; Finally, the final semantic weights are obtained through the Sigmoid activation function: ； In the formula, Indicates position The semantic weight value at that location; This represents the Sigmoid activation function; Represents the natural exponential function; S1.5 Constructing candidate foreign object extraction units: After obtaining the semantic modulation weights, these weights are applied to the unified visual feature map to enhance suspected foreign object regions and suppress the pure coal flow background, thereby forming a semantically enhanced feature map, the expression of which is: ； In the formula, This indicates that the enhanced feature map is located at... Feature value on the c-th channel; After the enhanced feature map is input into the detection head, the bounding box coordinates of the candidate targets, the target confidence score, and the category prediction result are output, realizing the first stage of foreign object initial detection. The set of candidate targets in the nth frame image is represented as: ； In the formula, This represents the set of candidate targets in the nth frame image; This represents the m-th candidate target in the n-th frame; m is the index of the candidate target. This represents the total number of candidate targets in the nth frame; n represents the video frame index. Wherein, a single candidate target is written as: ； In the formula, and These represent the x and y coordinates of the center point of the candidate box, respectively. and These represent the width and height of the candidate box, respectively. This represents the confidence level of the candidate target output by the detector; This indicates the candidate target category output by the detector.

3. The method for detecting foreign objects in coal flow of a belt conveyor based on a multimodal visual language large model according to claim 1, characterized in that, The specific process of S2 is as follows: S2.1 Constructing a hierarchical review unit for candidate targets: The hierarchical review unit for candidate targets performs segmentation processing based on the confidence level of the candidate boxes output by the detector, and its threshold relationship satisfies: ； In the formula, Indicates a low threshold; Indicates the threshold value; Indicates a high threshold; For candidate targets entering the review process, the corresponding regions are extracted from the original image, and inference is performed using visual question-and-answer prompts. The visual semantic feature vector of the candidate region is represented as follows: ； In the formula, This represents the semantic feature vector obtained by visual encoding the m-th candidate region in the n-th frame; Represents the visual encoding function in a large-scale multimodal visual language model; Represents the original image from the nth frame. Middle of the candidate box The cropped region image; The multimodal visual language large model outputs the presence markers, semantic categories, and hazard level scores of candidate regions, along with corresponding semantic confidence scores, expressed as follows: ； In the formula, This represents the semantic confidence score of the m-th candidate target in the n-th frame after multimodal re-judgment. This represents the category probability distribution output by a large multimodal visual language model. S2.2 Constructing a working condition perception fusion unit: After obtaining the initial detection results and multimodal re-judgment results from the detector, the credibility of the two types of evidence is determined based on the on-site working conditions. By setting up a working condition perception fusion unit, the belt speed, ambient illuminance, and coal flow parameters are combined to form a working condition vector, the expression of which is: ； In the formula, This represents the vector of operating parameters at time n. Represents a column vector; Indicates the belt speed; Indicates ambient illuminance; Indicates coal flow rate; Indicates dust concentration; Indicates image sharpness; Indicates the vibration intensity of the equipment; After standardizing the operating condition parameter vector, we obtain: ； In the formula, This represents the standardized operating condition parameter vector; A vector representing the mean values ​​of the operating parameters; A vector representing the standard deviation of operating parameters; The standardized operating condition parameter vector is input into a multilayer perceptron network, and the output of the first hidden layer is represented as: ； In the formula, This represents the output vector of the first hidden layer; Represents the linear rectification activation function; This indicates a batch normalization operation; This represents the first-layer weight matrix; Indicates the first-level bias term; The output of the second hidden layer is represented as: ； In the formula, This represents the output vector of the second hidden layer; Indicates a random deactivation operation; This represents the weight matrix of the second layer; Indicates the second-level bias term; The fusion weights for the two categories of decisions are obtained through Softmax normalization: ； In the formula, This represents the fusion weights corresponding to the detector confidence levels; This represents the fusion weight corresponding to the multimodal re-judgment confidence level; Represents the normalization function; This represents the third-layer weight matrix; Indicates the third-level bias term; The two types of fusion weights satisfy: ； Based on this, the final fusion confidence of the candidate targets is obtained: ； In the formula, This represents the final fusion confidence of the m-th candidate target in the n-th frame; S2.3, Generation of continuous frame consistency confirmation unit: The consecutive frame consistency verification unit performs trajectory association on candidate targets in adjacent frames within a sliding time window. The sequence of N consecutive frames within the window is represented as follows: ； In the formula, This represents a sequence of N consecutive frames within a sliding time window; N represents the length of the time window. Extract the local representation vector of each candidate box from the enhanced feature map, denoted as: , This represents the local region feature vector of the m-th candidate target in the n-th frame; Indicates the dimension of regional features; To further introduce semantic constraints, a set of cross-frame related semantic cues is constructed: ； In the formula, Represents a set of semantic cues used for cross-frame association; This represents the p-th associated semantic cue; p is the cue index; P represents the total number of associated semantic cuees; The corresponding text embedding vector is represented as The semantic similarity component of the candidate target relative to the p-th associated semantic cue is represented as: ； In the formula, This represents the semantic similarity score between the m-th candidate target and the p-th associated semantic cue in the n-th frame; The norm of the region's eigenvectors; The norm of the text embedding vector; Combining appearance and semantic information, a comprehensive correlation matrix is ​​constructed between candidate targets across frames, with the element expression as follows: ； In the formula, Indicates the first The first frame Semantic feature vectors of candidate targets; Indicates the first The first frame Semantic feature vectors of candidate targets; This represents the overall correlation between the a-th candidate target in the i-th frame and the b-th candidate target in the j-th frame; Weighting coefficients representing the similarity of regional appearances; Weight coefficients representing semantic similarity; Represents the cosine similarity function; S2.4 Constructing a time-series confidence smoothing unit: After completing cross-frame association, to further suppress confidence fluctuations caused by belt vibration, image jitter, and momentary occlusion, temporal smoothing is performed on the trajectory-level detection results. The expression is as follows: ； In the formula, This represents the smooth confidence level at the current moment; This represents the smooth confidence level at the previous time step; This represents the smoothing coefficient, with a value ranging from 0 to 1; This indicates the weight of the current observation; S2.5 Constructing Feature Memory Update Units: The feature memory update unit is used to maintain the foreign object feature library, and its expression is: ； In the formula, G represents the foreign object feature library; This represents the feature vector of the g-th template in the library; This represents the category identifier corresponding to the template feature; g represents the template index; For the new sample features confirmed by reassessment, the original template is modified using a momentum update method, expressed as: ； In the formula, This represents the updated g-th template feature; This represents the g-th template feature before the update; This represents the feature vector of the new input sample; This represents the momentum update coefficient.

Images