Metal-based solid waste sample classification method, device and equipment based on multi-modal fusion

Abstract

This application relates to a method, apparatus, and device for classifying metal-based solid waste samples based on multimodal fusion. The method includes: inputting the elemental information of the metal-based solid waste sample to be tested into a pre-trained elemental classification model, outputting a first confidence matrix and a first reliability score of the first confidence matrix; inputting the image information of the metal-based solid waste sample to be tested into a pre-trained image classification model, outputting a second confidence matrix and a second reliability score of the second confidence matrix; when the first category corresponding to the highest confidence in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence in the second confidence matrix, and the category difference between the first and second categories is within a preset range, calculating the weighted total score of each category associated with the first and second matrices, and determining the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested. This method has high classification accuracy and strong anti-interference ability.

Description

Technical Field

[0001] This application relates to the field of metal-based solid waste sample classification technology, and in particular to a method, apparatus and equipment for metal-based solid waste sample classification based on multimodal fusion. Background Technology

[0002] Accurate classification of metal-based solid waste samples is a key prerequisite for their resource utilization. In particular, in the simultaneous smelting process of lead-zinc ore combined with metal-based solid waste samples, rapid and accurate classification of solid waste samples directly affects the rationality of intelligent batching and production stability.

[0003] Existing classification methods for metal-based solid waste samples mainly rely on single-modal information: element-based classification methods utilize techniques such as X-ray fluorescence spectroscopy to obtain elemental composition; image-based classification methods distinguish solid waste sample morphology, color, and texture through visual features. However, current metal-based solid waste sample classification methods suffer from low accuracy and weak anti-interference capabilities. Therefore, this application provides a metal-based solid waste sample classification method based on multimodal fusion, which offers high accuracy and strong anti-interference capabilities. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, and medium for classifying metal-based solid waste samples based on multimodal fusion that has high accuracy and strong anti-interference ability, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for classifying metal-based solid waste samples based on multimodal fusion, including:

[0006] S101. Input the element information of the metal-based solid waste sample to be tested into the pre-trained element classification model, and output the first confidence matrix and the first reliability score of the first confidence matrix. Input the image information of the metal-based solid waste sample to be tested into the pre-trained image classification model, and output the second confidence matrix and the second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample to be tested belonging to each category.

[0007] S102. When the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix, calculate the category difference between the first category and the second category.

[0008] S103. When the category difference is within a preset range, use Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. iLet Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0009] In the aforementioned multimodal fusion-based metal-based solid waste sample classification method, a pre-trained elemental classification model is used to process the elemental information of the metal-based solid waste sample under test, and a pre-trained image classification model is used to process the image information of the metal-based solid waste sample under test. This combines the stability of elemental information with the feature discrimination advantages of image information. Furthermore, the output not only provides a confidence matrix but also a reliability score, enabling real-time quantification of the credibility of the elemental and image modes under the current input environment. This effectively avoids misleading results caused by data distortion in a single mode, significantly enhancing the system's fault tolerance and anti-interference capabilities. Further, for situations where the prediction results between modes are inconsistent but the difference is within a preset range, this application can perform soft fusion based on the probability confidence and reliability score of the corresponding category in the confidence matrix, fully utilizing the complementary information of the two modes to achieve a higher final score S. i It can more accurately reflect the true category, improve the anti-interference ability, and effectively solve the problem of classification error caused by single-mode data distortion in complex working conditions of metal-based solid waste samples, with high accuracy.

[0010] In one embodiment, the method further includes a training process for an element classification model, the training process of which includes:

[0011] S201. Using the elemental information of multiple metal-based solid waste samples with attached category labels as input features, initialize the hyperparameters of the elemental classification model. The elemental information includes element type and content.

[0012] S202. Extract multiple morphological features from the sample images of each metal-based solid waste sample, and stitch together the morphological features and input features of the metal-based solid waste sample to obtain the enhanced input features of each metal-based solid waste sample.

[0013] S203. Multi-scale decomposition of the enhanced input features is performed by wavelet transform to obtain the low-frequency global features and high-frequency detail features of each metal-based solid waste sample.

[0014] S204. When the total number of low-frequency global features and high-frequency detail features of the metal-based solid waste sample is greater than the first threshold or less than the second threshold, data augmentation processing is performed based on the low-frequency global features and high-frequency detail features to obtain the augmented dataset of each metal-based solid waste sample, and the first training set is selected from each augmented dataset, where the first threshold is greater than the second threshold.

[0015] S205. Sample the first training set and input the low-frequency global features of the sampled first samples into the first-level global random forest of the element classification model. Based on the principle of minimizing Gini impurity, split the nodes and output the first-level classification results of each first sample and the first-level classification confidence of each first-level classification result. Select the difficult samples with the first-level classification confidence lower than the third threshold.

[0016] S206. The primary classification results and primary classification confidence of the difficult-to-classify samples are converted into numerical prior features and spliced ​​with the high-frequency detail features of the difficult-to-classify samples to obtain spliced ​​features.

[0017] S207. Input the spliced ​​features of the difficult-to-distinguish samples into the secondary detail random forest of the element classification model, and output the secondary classification results and secondary classification confidence.

[0018] S208. Input the high-frequency detail features of the second samples that were not sampled in the first training set into the element classification model in training, output the predicted classification results of each second sample, and compare the predicted classification results of each second sample with the true classification results to obtain the first classification error rate.

[0019] S209. Exchange the morphological features between the metal-based solid waste samples to obtain the exchange morphological features of each metal-based solid waste sample. Based on the exchange morphological features of the metal-based solid waste samples and the enhanced input features after splicing the input features, execute S203, S204, S205, S206, S207 and S208 to obtain the second classification error rate. Calculate the difference between the first classification error rate and the second classification error rate to obtain the cross-modal feature contribution.

[0020] S210. When the first classification error rate is less than the fourth threshold and the cross-modal feature contribution is greater than the fifth threshold, the trained element classification model is obtained.

[0021] In the above embodiments, by using a cascaded structure and multi-scale feature fusion, both global element features and detailed features are taken into account. The two-stage training specifically improves the classification accuracy of difficult-to-distinguish samples, and the cross-modal feature guidance further enhances the ability to distinguish between solid waste samples with similar elements, ultimately resulting in an element classification model that supports accurate classification.

[0022] In one embodiment, the method further includes a training process for an image classification model, the training process of which includes:

[0023] S301. Collect image data of each metal-based solid waste sample and label the metal-based solid waste samples in the image data to obtain labeled images;

[0024] S302. Using a pre-trained initial element prediction model, predict the element information of metal-based solid waste samples in the labeled image to obtain element category pseudo-labels. Then, associate the image data, the labeling information of the image data, and the element category pseudo-labels to form a dual-label image dataset.

[0025] S303. Extract the second training set from the dual-label image dataset and perform data augmentation on the second training set to obtain the image augmentation training set.

[0026] S304. Initialize the model training parameters of the image classification model. Based on the pre-acquired unlabeled image dataset, pre-train the initialized image classification model to obtain pre-trained weights, and load the pre-trained weights into the image classification model. The image classification model after loading the pre-trained weights includes at least a backbone module, a neck module, a head module, and a pseudo-label supervision branch. The backbone module includes at least a feature extraction module, an attention module, and a spatial pyramid pooling module. The neck module includes at least a feature fusion module and a cross-modal feature interaction module.

[0027] S305. Input the image augmentation training set into the feature extraction module to obtain the mid-level semantic features of the image augmentation training set. Then, use the attention module to perform weighted enhancement of the mid-level semantic features of the image by the channel dimension and the spatial dimension to obtain the image semantic enhancement features.

[0028] S306. Input the image semantic enhancement features into the spatial pyramid pooling module to obtain the global deep features of the image, and reduce the dimensionality of the global deep features of the image to obtain the backbone feature map.

[0029] S307. The feature fusion module performs internal residual fusion on the backbone feature map to obtain deep semantic features of the image. The cross-modal feature interaction module performs cross-channel attention fusion on the pseudo-label feature vector corresponding to the pseudo-label of the element category and the deep semantic features of the image to obtain attention fusion features. The attention fusion features are then upsampled and concatenated with the mid-level semantic features of the image.

[0030] S308. The spliced ​​features are bidirectionally fused through the neck module to obtain multi-scale cross-modal fusion features;

[0031] S309. Input the multi-scale cross-modal fusion features into the head module to obtain the image classification result, and obtain the image classification loss based on the image classification result and annotation information;

[0032] S310. Through the pseudo-label supervision branch, the multi-scale cross-modal fusion features are mapped to the element feature space to obtain the element category prediction results. Based on the element category prediction results and the element category pseudo-labels, the pseudo-label prediction loss is obtained.

[0033] S311. Based on the multi-scale cross-modal fusion features, obtain the fine-tuned features corresponding to the image data. Using the image classification model with pre-trained weights, generate the target features corresponding to the image data. And obtain the contrastive learning regularization loss by minimizing the difference between the fine-tuned features and the target features.

[0034] S312. Based on the image classification loss, label prediction loss, and contrastive learning regularization loss, optimize the image classification model to obtain the image classification model.

[0035] In the above embodiments, the image classification model is specifically improved by strengthening the extraction of key image features through an attention module and achieving deep fusion of element features and image features through a cross-modal feature interaction module, making the classification basis of metal-based solid waste more scientific and comprehensive. In the first stage, the model is pre-trained to learn the general features of solid waste images, solving the problem of poor model generalization in small sample scenarios. In the second stage, pre-trained weights are loaded, pseudo-label supervision branches are added, and multi-task loss is jointly calculated to complete fine-tuning, ensuring that the model can simultaneously fit the correlation between image features and element attributes. The accuracy of the trained model is significantly improved.

[0036] In one embodiment, the second training set is augmented to obtain an image augmented training set, including:

[0037] S401, in response to performing a random fluctuation enhancement operation on the element feature vector corresponding to the image data in the second training set, determine the image perturbation parameters corresponding to the image data, and perform perturbation adjustment on the image data of the metal-based solid waste sample based on the image perturbation parameters to obtain the first enhanced dataset;

[0038] S402, in response to performing a stitching and fusion operation on the image data of multiple metal-based solid waste samples, obtain the information of the elements to be enhanced corresponding to each image data, and perform weighted average fusion processing on the information of the elements to be enhanced to obtain the second enhanced dataset;

[0039] S403, add the first augmented dataset and the second augmented dataset to the second training set to obtain the image augmented training set.

[0040] In the above embodiments, by using two cross-modal linkage enhancement methods, when one modality is enhanced, the other modality will make corresponding and logically consistent adjustments to maintain the semantic consistency of cross-modal data, thereby significantly improving the generalization ability of the model in real industrial mixed scenarios.

[0041] In one embodiment, the metal-based solid waste sample classification method based on multimodal fusion further includes:

[0042] S501. Obtain the element classification weights w of the element classification model. eImage classification weights w in image classification models i ,in, ;

[0043] S502, When the first category and the second category are consistent, use Calculate the fused confidence matrix and P merge The category with the highest probability was determined as the category of the metal-based solid waste sample to be tested, P. merge Let Pe be the fused confidence matrix, and Pi be the first confidence matrix and Pi be the second confidence matrix.

[0044] S503. When the class difference exceeds the preset difference threshold, extract the important element feature content in the information of the element to be tested and the important morphological feature content in the image information to be tested. Based on the important element feature content, the important morphological feature content, the first confidence matrix and the second confidence matrix, construct a conflict discrimination vector and input the conflict discrimination vector into the pre-trained conflict resolution classifier to obtain the class of the metal-based solid waste sample to be tested.

[0045] In the above embodiments, differentiated resolution strategies are adopted for different conflict levels to address the problem of insufficient accuracy in traditional single resolution methods. When no conflict is triggered, the resolution is based on the element classification weight w. e Image classification weight w i Output P merge The category with the highest probability is selected. When a conflict is triggered, the result is output after hierarchical resolution.

[0046] In one embodiment, S501 includes:

[0047] S601. Construct a reliability assessment function, and based on the reliability assessment function, calculate the historical element classification reliability score and historical image classification reliability score under different historical environmental conditions during the historical classification process of metal-based solid waste samples.

[0048] S602. Using historical environmental information under different historical environmental conditions as input features, and the normalized values ​​of historical element classification reliability scores and historical image classification reliability scores corresponding to historical environmental information as output labels, train a nonlinear mapping model between input features and output features.

[0049] S603. Input the current environmental information corresponding to the metal-based solid waste sample to be tested into the nonlinear mapping model to obtain the initial element classification weights w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0 ;

[0050] S604. Obtain the historical weight distribution information of metal-based solid waste samples, and construct a prior probability model based on the historical weight distribution information; wherein, the historical weight distribution information includes the initial weight w of the historical element classification corresponding to the historical classification process where the classification result is correct. e0历 Initial weights w for historical image classification i0历 Initial weights w for historical element classification e0历 The optimized historical element classification target weight w e历 Initial weights w for historical image classification i0历 The optimized historical image classification target weights w i历 ;

[0051] S605. The optimization objective is to maximize the maximum confidence value in the fused confidence matrix. Based on the prior probability model, initial weights w are assigned to the element classification. e0 Image classification initial weights w i0 Through iterative optimization, the target weight w for element classification is obtained. e Image classification target weight w i .

[0052] In the above embodiments, in the four-level decision fusion architecture, the first level quantifies the current prediction confidence of the bimodal model through a reliability evaluation function; the second level fits the nonlinear relationship between environmental information, modal reliability, and weights, replacing the traditional fixed threshold and linear fitting, which is more in line with complex industrial conditions; the third level adaptively updates the initial weights, aiming to maximize the classification confidence after fusion, to achieve dynamic optimal solution of the weights and obtain the accurate element classification target weight w. e Image classification target weight w i This allows for accurate classification of the metal-based solid waste samples to be tested.

[0053] In one embodiment, the method further includes a process for acquiring information about the element to be tested and information about the image to be tested, the process of acquiring information about the element to be tested and information about the image to be tested including:

[0054] S701, perform coordinate matching between the detection area of ​​the element detection device and the shooting area of ​​the image capturing device, so that the acquired image information to be tested and the data of the element to be tested correspond to the same detection area;

[0055] S702 generates synchronized element detection trigger signals and image acquisition trigger signals. The element detection trigger signal triggers the element detection device to detect the element information of the metal-based solid waste sample to be tested. At the same time, the image acquisition trigger signal triggers the image capturing device to acquire the image information of the metal-based solid waste sample to be tested.

[0056] In the above embodiments, the element-image spatiotemporal alignment calibration solves the problem of regional mismatch and time asynchrony between element detection and image acquisition in industrial scenarios, ensuring the semantic consistency of paired samples; and the cross-modal linkage data augmentation strategy solves the problem of cross-modal data semantic disconnect caused by traditional single-modal augmentation, improving the quality and generalization of the dataset.

[0057] Secondly, this application also provides a metal-based solid waste sample classification device based on multimodal fusion, characterized in that the device comprises:

[0058] The single-modal analysis module is used to input the element information of the metal-based solid waste sample to be tested into a pre-trained element classification model and output a first confidence matrix and a first reliability score of the first confidence matrix. It also inputs the image information of the metal-based solid waste sample to be tested into a pre-trained image classification model and outputs a second confidence matrix and a second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample belonging to each category.

[0059] The multimodal analysis module is used to calculate the class difference between the first and second categories when the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix.

[0060] The classification module is used to determine the appropriate classification method when the class difference falls within a preset range. Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0061] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described multimodal fusion-based metal-based solid waste sample classification method.

[0062] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for classifying metal-based solid waste samples based on multimodal fusion.

[0063] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described multimodal fusion-based metal-based solid waste sample classification method. Attached Figure Description

[0064] Figure 1 This is an application environment diagram of a metal-based solid waste sample classification method based on multimodal fusion in one embodiment;

[0065] Figure 2 This is a flowchart illustrating a multimodal fusion-based method for classifying metal-based solid waste samples in one embodiment.

[0066] Figure 3 Here is a complete structural diagram of the improved YOLOv5s model in one embodiment;

[0067] Figure 4 This is a flowchart illustrating a multimodal fusion-based method for classifying metal-based solid waste samples in a detailed application example.

[0068] Figure 5 This is a structural block diagram of a metal-based solid waste sample classification device based on multimodal fusion in one embodiment.

[0069] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] Existing classification methods for metal-based solid waste samples mainly rely on single-modal information. Among them, classification methods based on elemental information use techniques such as X-ray fluorescence spectroscopy to obtain elemental composition, but they are prone to misclassification of solid waste samples with similar elemental types and contents. Classification methods based on image information distinguish the shape, color and texture of solid waste samples through visual features, but they are easily affected by environmental factors such as light, temperature and humidity, resulting in insufficient classification stability.

[0072] Current classification models can establish a mapping relationship between input features and output labels by learning from large amounts of data, enabling rapid classification of unknown data. Among them, the random forest model has good robustness to high-dimensional sparse data and fast training and prediction speeds. However, existing conventional random forests are single-stage structures, unable to simultaneously take into account global element content features and trace element detail features, resulting in weak discrimination ability for difficult-to-distinguish samples with similar elements, and failing to achieve deep fusion of cross-modal features. The YOLOv5 model, as a single-stage object detection algorithm, can efficiently extract image features and achieve rapid classification. However, conventional YOLOv5s have insufficient feature extraction capabilities for overlapping solid waste samples and powdered solid waste samples, making them prone to errors. Feature omission and the prevalence of purely supervised learning in training processes result in poor generalization ability in small sample scenarios. Multimodal fusion technology can integrate the advantages of different modalities, compensate for the deficiencies of a single modality, and improve classification accuracy and anti-interference ability. However, existing multimodal fusion schemes mostly adopt simple weighted fusion strategies, failing to consider the reliability differences of the modalities themselves, the nonlinear relationship between environmental information and weights, and modal conflicts, thus limiting the fusion effect. Furthermore, the training processes of dual-modal models are independent of each other, failing to form collaborative optimization, and the model structure is not specifically improved for the characteristics of metal-based solid waste samples. The training method does not explore the correlation value of cross-modal data, making it difficult to fully leverage the complementary advantages of multimodal data. Therefore, this paper proposes a rapid classification method for metal-based solid waste samples based on multimodal fusion, which has high classification accuracy and strong anti-interference ability.

[0073] The metal-based solid waste sample classification method based on multimodal fusion provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 1002 interacts with server 1004 via a wired / wireless channel. The data storage system can store the data that server 1004 needs to process.

[0074] This metal-based solid waste sample classification method can be applied independently to terminal 1002. For example, when a user triggers the metal-based solid waste sample classification control on terminal 1002, terminal 1002 acquires the element information and image information of the metal-based solid waste sample uploaded by the user. The element information is input into a pre-trained element classification model, which outputs a first confidence matrix and a first reliability score for the first confidence matrix. The image information is input into a pre-trained image classification model, which outputs a second confidence matrix and a second reliability score for the second confidence matrix. The elements in the first and second confidence matrices represent the probability confidence of the metal-based solid waste sample belonging to each category. When the first category corresponding to the highest confidence in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence in the second confidence matrix, terminal 1002 calculates the category difference between the first and second categories. When the category difference is within a preset range, the method is used... Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0075] This metal-based solid waste sample classification method can also be applied between terminal 1002 and server 1004. For example, when a user triggers the metal-based solid waste sample classification control on terminal 1002, terminal 1002 obtains the element information and image information of the metal-based solid waste sample to be tested uploaded by the user, and uploads the element information and image information to server 1004. Server 1004 then performs the above-mentioned metal-based solid waste sample classification steps based on the element information and image information, which will not be elaborated here.

[0076] The terminal 1002 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. The server 1004 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers.

[0077] In one embodiment, such as Figure 2 As shown, a multimodal fusion-based method for classifying metal-based solid waste samples is provided, and this method is applied to... Figure 1 Taking server 1004 as an example, including S101~S103:

[0078] S101. Input the element information of the metal-based solid waste sample to be tested into the pre-trained element classification model, and output the first confidence matrix and the first reliability score of the first confidence matrix. Input the image information of the metal-based solid waste sample to be tested into the pre-trained image classification model, and output the second confidence matrix and the second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample belonging to each category.

[0079] Metal-based solid waste samples refer to solid waste containing metals such as lead and zinc and their compounds, generated during the mining, smelting, and processing of metals. Elemental information refers to the composition, content, distribution characteristics, and physicochemical forms of chemical elements in solid waste or materials, determined through chemical analysis. The image classification model is a machine learning model whose task is to receive an image as input and output the category label to which the image belongs. The elemental classification model is a model specifically designed for classifying elements.

[0080] Probability confidence refers to the model's subjective certainty about a certain prediction result. Each numerical element in the confidence matrix represents the probability confidence that the current metal-based solid waste sample is classified into a certain preset solid waste sample category by the corresponding classification model. It is a quantification of the model's prediction confidence of the current sample category, with a value range of 0 to 1. The sum of all elements in the matrix is ​​1. For example, the first confidence matrix [metallized pellets 45.2%, metallized ball sieve powder 42.8%] means that the element classification model has a confidence of 45.2% in classifying the metal-based solid waste sample as metallized pellets, a confidence of 42.8% in classifying it as metallized ball sieve powder, and the remaining confidence is the probability of belonging to one of the other 9 solid waste sample categories.

[0081] The reliability score is a standardized score that quantifies the overall credibility of a single-modal classification model (elemental classification model or image classification model) in predicting the current test sample. The higher the score, the more reliable the prediction result of the modal model for the current test sample, and the higher the weight assigned in the subsequent weighted fusion; conversely, the lower the score, the lower the weight.

[0082] Specifically, elemental detection and image acquisition are performed on the metal-based solid waste sample to be tested to obtain the elemental information and image information of the sample; pre-trained elemental classification model and image classification model are obtained from the database; the elemental information to be tested is input into the trained elemental classification model, and the elemental classification model outputs the first classification confidence matrix and the corresponding modal reliability score; the image information to be tested is input into the trained image classification model, and the image classification model outputs the second classification confidence matrix and the corresponding modal reliability score.

[0083] S102. When the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix, calculate the category difference between the first category and the second category.

[0084] Specifically, when the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix, it may be due to interference in the single-modal model detection process or the sample being tested being located in the boundary zone of category judgment. This leads to a conflict between the categories of the sample detected by the two methods: elemental information and image information. In this case, a category cannot be simply randomly selected; instead, a hierarchical conflict resolution mechanism needs to be activated to resolve the discrepancy in the bimodal prediction and ensure classification accuracy. When activating the hierarchical conflict resolution mechanism, the concept of category difference between the first and second categories needs to be introduced for hierarchical decision-making. That is, the category of the sample being tested can be determined based on whether the category difference is within a preset range.

[0085] S103. When the category difference is within a preset range, use Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0086] Specifically, when the class dissimilarity is within a preset range, the class with the highest bimodal confidence level exhibits a slight conflict. This triggers the first-level dissimilarity conflict resolution rule, also known as reliability score-weighted voting: [The rule is then implemented using...] Calculate the weighted total score for each category in the association between the first confidence matrix and the second matrix, where S i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i The probability confidence of the tested metal-based solid waste sample belonging to category i in the second confidence matrix is ​​given, and the category with the highest weighted total score is determined as the category of the tested metal-based solid waste sample.

[0087] Taking 11 types of metal-based solid waste samples as an example, the sample to be tested was low-zinc powder. After testing, the first confidence matrix was: low-zinc powder 40%, high-zinc powder 55% (highest category is high-zinc powder), and the first reliability score R... e=0.65; Second confidence matrix: low zinc powder 90%, high zinc powder 5% (low zinc powder is the highest category), second reliability score Ri=0.90; the cosine distance between the two highest categories is calculated to be 0.45, which is within the preset range of 0.3~0.6, triggering the first-level weighted voting resolution; at this time, with Re and Ri as voting weights, the votes for low zinc powder = 0.65×0+0.90×1=0.90, and the votes for high zinc powder = 0.65×1+0.90×0=0.65. The votes for low zinc powder are higher, and the final classification result is low zinc powder.

[0088] In the aforementioned multimodal fusion-based metal-based solid waste sample classification method, a pre-trained elemental classification model is used to process the elemental information of the metal-based solid waste sample under test, and a pre-trained image classification model is used to process the image information of the metal-based solid waste sample under test. This combines the stability of elemental information with the feature discrimination advantages of image information. Furthermore, the output not only provides a confidence matrix but also a reliability score, enabling real-time quantification of the credibility of the elemental and image modes under the current input environment. This effectively avoids misleading results caused by data distortion in a single mode, significantly enhancing the system's fault tolerance and anti-interference capabilities. Further, for situations where the prediction results between modes are inconsistent but the difference is within a preset range, this application can perform soft fusion based on the probability confidence and reliability score of the corresponding category in the confidence matrix, fully utilizing the complementary information of the two modes to achieve a higher final score S. i It can more accurately reflect the true category, improve the anti-interference ability, and effectively solve the problem of classification error caused by single-mode data distortion in complex working conditions of metal-based solid waste samples, with high accuracy.

[0089] In one embodiment, the method further includes a training process for an element classification model, which includes the following steps S201 to S210:

[0090] S201. Using the elemental information of multiple metal-based solid waste samples with attached category labels as input features, initialize the hyperparameters of the elemental classification model. The elemental information includes element type and content.

[0091] Hyperparameters are configuration variables that are manually set before training begins, and are used to control the model's structure and learning process.

[0092] Specifically, this application trains a single-modality elemental classification model based on elemental information from multiple metal-based solid waste samples with attached category labels. The elemental information of the metal-based solid waste samples can be obtained through methods such as spectral or chemical analysis. The elemental classification model can be a random forest, gradient boosting tree model, support vector machine, K-nearest neighbor model, or logistic regression model. For large samples, it can also be a fully connected neural network or a deep learning model such as an autoencoder + classifier. Furthermore, a two-stage progressive training method is used to optimize the elemental classification model, resulting in a well-trained elemental classification model.

[0093] In the two-stage progressive training process, firstly, each metal-based solid waste sample is numbered, and an input feature matrix is ​​constructed based on the element type and content of each sample (the rows of the matrix represent sample 1, sample 2, ..., sample M, and the columns represent element types). An output label array is constructed based on the sample numbers, and the hyperparameters of the element classification model are initialized. Taking the element classification model as a random forest as an example, the hyperparameters initialized for the random forest model include the number of decision trees in the first-level global forest and the second-level detail forest, the maximum depth of a single tree, the size of the feature subset of the node split, and the minimum number of samples in the leaf node.

[0094] S202. Extract multiple morphological features from the sample images of each metal-based solid waste sample, and stitch together the morphological features and input features of the metal-based solid waste sample to obtain the enhanced input features of each metal-based solid waste sample.

[0095] Specifically, morphological features, including target area, perimeter, texture entropy, and color mean, are extracted from the sample images of each metal-based solid waste sample. The morphological features of all samples are normalized to form an auxiliary feature matrix (typically N rows × 4 columns) with the same row dimension as the input feature matrix (N rows × M columns, where N = total number of samples, M = number of element types). This auxiliary feature matrix is ​​then concatenated column-wise to the element feature columns of the input feature matrix in a fixed order of target area - perimeter - texture entropy - color mean, forming an enhanced input feature matrix (N rows × (M+4) columns). In an optional embodiment, morphological feature extraction is implemented using OpenCV, the color mean is calculated based on the average of the RGB three channels, and the texture entropy is calculated using the gray-level co-occurrence matrix P(i,j). ,in, It is a minimal constant, P(i,j) refers to the probability that grayscale values ​​i and j appear adjacent to each other in a specified direction 0 and at a distance d, and satisfies .

[0096] It should be noted that morphological features are cross-modal global auxiliary features that reflect the visual morphological attributes of the sample. They are used to help the model distinguish solid waste samples with similar elemental content but different morphologies (such as metallized pellets and pellet screening powder). Therefore, there is no need to match them according to the element column position. However, the model training needs to learn the mapping relationship between the feature column position and the category label. If the column order of morphological features is random each time, the model will not be able to stably learn the feature rules. Therefore, morphological features must follow a fixed column order and cannot be spliced ​​arbitrarily.

[0097] S203. The enhanced input features are decomposed into low-frequency global features and high-frequency detailed features of each metal-based solid waste sample by wavelet transform.

[0098] Wavelet transform is a tool for analyzing signals that can simultaneously tell you the types of frequencies in a signal and when those frequencies occur.

[0099] Specifically, the enhanced input features are decomposed into low-frequency global features (major element content features) and high-frequency detail features (trace element content, image texture detail features) through wavelet transform. A subset of global features is constructed based on the low-frequency global features, and a subset of detail features is constructed based on the high-frequency detail features. Each sample will generate a set of low-frequency global features and a set of high-frequency detail features.

[0100] In an optional embodiment, the wavelet transform uses the db4 wavelet basis, with a decomposition level of 3. The low-frequency global features are the approximation coefficients of the 3rd level, and the high-frequency detail features are the reconstructed features of the detail coefficients of the 1st to 3rd levels.

[0101] S204. When the total number of low-frequency global features and high-frequency detail features of the metal-based solid waste sample is greater than the first threshold or less than the second threshold, data augmentation processing is performed based on the low-frequency global features and high-frequency detail features to obtain the augmented dataset of each metal-based solid waste sample, and the first training set is selected from each augmented dataset, wherein the first threshold is greater than the second threshold.

[0102] Specifically, when the total number of low-frequency global features and high-frequency detail features of a metal-based solid waste sample is greater than the first threshold or less than the second threshold, a data augmentation algorithm can be used to address the sample imbalance problem, thereby obtaining an augmented dataset for each metal-based solid waste sample. The augmented dataset is then divided into a first training set and a first test set according to a preset ratio, which can be 8:2, 7:3, etc., and is not limited here.

[0103] Furthermore, for low-frequency global features and high-frequency detail features with sufficient sample quantity, KNN (K-Nearest Neighbors) interpolation can be used to generate new sample augmentation datasets. For low-frequency global features and high-frequency detail features with insufficient sample quantity, a stochastic fluctuation algorithm can be used to generate new samples. Among them, the KNN interpolation method generates new samples based on the feature similarity of similar samples, which is suitable for scenarios with sufficient sample quantity. The new samples have high authenticity, consistent with the feature distribution of the original samples, and no false features are introduced. Specifically, based on the feature interpolation of the nearest neighbor samples of similar solid waste samples, the generated element content and morphological features are all within the reasonable feature range of similar samples, which conforms to the physical laws of actual detection, avoids feature distortion caused by random generation, and retains the core feature correlation of similar solid waste samples (such as the content ratio of a certain element with other elements, the matching relationship between morphological features and element features), improves the model's learning effect on the feature patterns of similar samples, and the interpolated new samples can effectively expand the dataset size, alleviate sample imbalance, and will not interfere with the feature distribution of the original samples. It is suitable for solid waste sample categories with sufficient sample quantity and stable feature distribution. The random fluctuation algorithm generates new samples based on small perturbations of single-sample features. It is suitable for scenarios with very few samples (no sufficient neighboring samples for interpolation), does not rely on other samples, is simple to implement, and can preserve the core categorical features of the original sample. Specifically, it performs small-range random fluctuations on the elemental content of the original sample. Where x represents the original elemental content data, and x' represents the perturbed elemental content data. The fluctuation coefficient is 0 < δ ≤ 1. To ensure that the random numbers follow a standard normal distribution, without changing the element types or deviating from the core characteristics of the solid waste sample, the new sample still belongs to the original category and does not require support from neighboring samples of the same type. This solves the problem that small sample categories cannot be used for KNN interpolation, and quickly expands the sample size. The slight feature fluctuations simulate the instrument errors and slight heterogeneity of samples in actual detection, allowing the model to learn the feature tolerance range of the solid waste sample and improve the model's generalization ability to small sample categories.

[0104] Furthermore, the enhanced datasets for each metal-based solid waste sample are complete paired samples consisting of elemental information and corresponding morphological features, consistent with the original sample structure: at the elemental level, new elemental types (unchanged) and elemental content data (values ​​after KNN interpolation / random fluctuation) are generated; at the image level, corresponding images are synchronously generated according to cross-modal linkage rules (e.g., when elements fluctuate randomly, the image is simultaneously slightly adjusted for brightness / contrast; when KNN interpolation generates elemental data, the image is simultaneously subjected to feature interpolation / morphological fine-tuning of neighboring sample images); ensuring semantic consistency between elemental and image data, and meeting the sample requirements for spatiotemporal alignment calibration.

[0105] It should be noted that the data augmentation here differs from the cross-modal linkage data augmentation strategy. Cross-modal linkage data augmentation targets the entire dataset, providing high-quality, semantically consistent, and scenario-comprehensive original paired datasets for multimodal model training, thus solving the cross-modal data matching problem. In contrast, this augmentation targets element classification model training, performing targeted augmentation on categories with imbalanced sample sizes to address the bias problem caused by sample imbalance in the model. Furthermore, the augmentation process follows the cross-modal linkage rules to ensure that the elements of the new samples match the semantics of the images.

[0106] S205. Sample the metal-based solid waste samples in the first training set, and input the low-frequency global features of the sampled first samples into the first-level global random forest of the element classification model. Based on the principle of minimizing Gini impurity, perform node splitting, output the first-level classification results of each first sample and the first-level classification confidence of each first-level classification result, and screen out the difficult samples with the first-level classification confidence lower than the third threshold.

[0107] Specifically, taking the element classification model as a random forest model as an example, the Bootstrap method is used to sample the global feature subset of the first training set. The low-frequency global features of the first sample obtained after sampling are used as the input of the first-level global random forest of the element classification model. The label array of the metal-based solid waste sample number of the corresponding row is used as the supervised output to train the first-level global random forest. The nodes are split based on the principle of minimizing Gini impurity. The first-level classification result of each first sample and the first-level classification confidence of each first-level classification result are output. Then, the difficult samples with the first-level classification confidence lower than the third threshold are screened out.

[0108] S206. The primary classification results and primary classification confidence of the difficult-to-classify samples are converted into numerical prior features and spliced ​​with the high-frequency detail features of the difficult-to-classify samples to obtain spliced ​​features.

[0109] Specifically, for difficult-to-distinguish samples, the high-frequency detail features of the difficult-to-distinguish samples are used as input, and the corresponding solid waste sample number label array is used as the supervised output. A secondary detail random forest is trained using high-frequency detail features. During the splitting process, the splitting priority of cross-modal auxiliary features (morphological features) is prioritized. That is, in the splitting stage of the decision tree node of the secondary detail random forest, the secondary forest can accurately classify difficult-to-distinguish samples with highly similar elemental contents through differences in morphological features, making up for the classification deficiency of the primary global forest that only relies on elemental features. This distinguishes difficult solid waste samples with similar elemental contents but different morphologies (such as metallized pellets and pellet screening powder). The specific steps include at least:

[0110] 1. Feature weight assignment: When initializing the decision tree of the secondary forest, assign a higher splitting weight coefficient (e.g., set to 1.2) to the cross-modal auxiliary feature column, and assign a default weight of 1.0 to the element feature column;

[0111] 2. Node splitting candidate screening: When splitting at each node, the decision tree calculates the Gini impurity reduction value of all features (the core splitting basis) and amplifies the Gini impurity reduction value of cross-modal auxiliary features by weighting the coefficient (i.e., reduction value × 1.2).

[0112] 3. Splitting Feature Selection: The model prioritizes the feature with the largest decrease in Gini impurity after amplification as the node splitting feature. Because the value of cross-modal auxiliary features is amplified, their probability of being selected as splitting features is much higher than that of ordinary element features.

[0113] 4. Full-tree iterative execution: Every decision tree and every split node in the secondary forest is executed according to this rule, ultimately achieving the priority preservation and utilization of cross-modal auxiliary features by the entire tree.

[0114] Furthermore, the first-level classification results and confidence scores of the difficult samples output by the first-level forest are used as numerical prior features input to the second-level forest, and concatenated with the high-frequency detail features of the difficult samples. The resulting concatenated features are then used as input to the second-level forest, allowing the second-level forest to utilize the classification information from the first-level forest during training / prediction, thus achieving joint optimization of the two-level models. The specific steps are as follows:

[0115] 1. For each hard-to-classify sample selected, extract the two core outputs of the primary forest and convert them into numerical features to form a 2-dimensional prior feature vector: Feature 1: Primary classification category code (e.g., if the primary forest determines it to be category 3, then the code is 3; if it is multi-class, one-hot coding can be used); Feature 2: Primary classification confidence (e.g., if the primary forest determines the confidence of this category to be 0.65, then the value 0.65 is directly taken).

[0116] 2. The above 2D prior feature vector is column-wise concatenated to the end of the high-frequency detail feature subset of the difficult-to-distinguish sample (e.g., 378-dimensional detail features → concatenated to 380-dimensional features) to form the final input features of the secondary forest;

[0117] 3. Training phase: Using the spliced ​​380-dimensional features as input and the solid waste sample number label array as supervised output, the training of the secondary forest is completed according to the conventional random forest training logic. The model automatically learns the mapping relationship between the primary prediction results + high-frequency detail features and the true category.

[0118] 4. Prediction stage: For the sample to be tested, the prediction results are first obtained through the primary forest and quantified. Then, the results are combined with the detailed features of the sample and input into the secondary forest to obtain the final classification result.

[0119] At this point, for difficult-to-distinguish samples with similar elemental content but different morphologies, the priority of cross-modal auxiliary features is increased, allowing the model to grasp the core distinguishing point of morphological differences. This makes the secondary detail forest a precise complement to the primary global forest. Furthermore, by incorporating the prior features of the primary forest, the secondary forest does not need to relearn the basic classification rules. Instead, it makes fine-tuning corrections for difficult-to-distinguish samples based on the classification of the primary forest, significantly improving training efficiency and classification accuracy. Ultimately, the primary forest is responsible for coarse classification of global elemental features, while the secondary forest is responsible for fine classification of detail features and cross-modal features. The two-level model works together to complete efficient and accurate classification of all samples.

[0120] S207. Input the spliced ​​features of the difficult-to-distinguish samples into the secondary detail random forest of the element classification model, and output the secondary classification results and secondary classification confidence.

[0121] S208. Input the high-frequency detail features of the second samples that were not sampled in the first training set into the element classification model in training, output the predicted classification results of each second sample, and compare the predicted classification results of each second sample with the actual classification results to obtain the first classification error rate.

[0122] Specifically, when training the first / second level forest, the Bootstrap sample of unselected out-of-bag samples is used as the second sample. The high-frequency detail features of the second sample and the first-level prior features are used as inputs and substituted into the element classification model in training to obtain the predicted classification result of each second sample. The predicted classification result of each second sample is compared with the true classification result to obtain the first classification error rate, which is the out-of-bag error. It is used to measure the model's generalization ability on unseen training data, and the smaller the better.

[0123] S209. Exchange the morphological features between the metal-based solid waste samples to obtain the exchange morphological features of each metal-based solid waste sample. Based on the exchange morphological features of the metal-based solid waste samples and the enhanced input features after splicing the input features, execute S203, S204, S205, S206, S207 and S208 to obtain the second classification error rate. Calculate the difference between the first classification error rate and the second classification error rate to obtain the cross-modal feature contribution.

[0124] Specifically, ideally, if a feature has no effect on classification, then shuffling its order should result in almost no change in the model's accuracy, with the accuracy difference approaching zero. However, in reality, if a feature (such as morphological features) is crucial for classification, shuffling its order causes the model to lose key discriminative information, leading to a significant drop in accuracy. Therefore, the importance ranking method can be used to exchange the morphological feature values ​​among the various metal-based solid waste samples, obtaining the exchanged morphological features for each sample. Based on the exchanged morphological features and the enhanced input features obtained by concatenating the input features, steps S203, S204, S205, S206, S207, and S208 are executed. The classification accuracy of the cascaded model before and after the exchange on the test set is compared to obtain the second classification error rate. The difference between the first and second classification error rates represents the cross-modal feature contribution. When the cross-modal feature contribution (accuracy difference) is greater than the preset threshold, it indicates that the addition of this morphological feature improves the model's classification accuracy, proving that it has a significant core role in the model's classification, which is in line with expectations; if the difference is very small, it indicates that the feature has almost no contribution to the model and is redundant.

[0125] In an optional embodiment, the specific operation of exchanging the values ​​of morphological features among the metal-based solid waste samples is as follows: 1. Keep the elemental features (non-cross-modal features) of all samples unchanged; 2. For the column (or multiple columns) of morphological features (cross-modal features), randomly shuffle the order of their values ​​across all samples; for example, if the morphological entropy of sample 1 is 0.8, sample 2 is 0.6, and sample 3 is 0.9, after the exchange, it may become sample 1 is 0.6, sample 2 is 0.9, and sample 3 is 0.8, in order to disrupt the true correlation between morphological features and the true labels of the samples, thereby assessing the importance of the feature for model prediction.

[0126] S210. When the first classification error rate is less than the fourth threshold and the cross-modal feature contribution is greater than the fifth threshold, the trained element classification model is obtained.

[0127] Specifically, if the first classification error rate is less than the fourth threshold and the cross-modal feature contribution is greater than the fifth threshold, it indicates that the trained element classification model has both good generalization ability and effectively utilizes cross-modal features. At this point, a qualified trained element classification model is obtained. Otherwise, return to S205 and readjust the hyperparameters of the cascaded random forest.

[0128] In an optional embodiment, the specific adjustment directions of the hyperparameters of the cascaded random forest for optimizing the fitting effect of the model are as follows: 1. First-level global forest hyperparameters: Number of decision trees: appropriately increase (e.g., from 80 to 100 / 120) to improve the model's ensemble learning ability; Size of node splitting feature subset: increase (e.g., from 8 to 10 / 12) to allow the decision trees to use more features for splitting; Maximum depth: moderately increase (e.g., from 8 to 10 / 12) to improve the model's ability to fit global features.

[0129] 2. Second-level detail forest hyperparameters: Number of decision trees: significantly increased (e.g., from 120 to 150 / 200), enhancing the ability to classify difficult samples in detail; Minimum number of leaf node samples: decreased (e.g., from 3 to 1 / 2), allowing the model to learn fine-grained features of difficult samples; Cross-modal feature splitting weight coefficient: increased (e.g., from 1, 2 to 1, 3 / 1, 5), further enhancing the splitting priority of morphological features.

[0130] 3. The third threshold for screening difficult samples: fine-tune the confidence threshold of the primary classification (e.g., from 0.7 to 0.65 / 0.75) to adjust the number of difficult samples entering the secondary forest, and avoid model overfitting / underfitting caused by too many / too few samples.

[0131] In the above embodiments, by using a cascaded structure and multi-scale feature fusion, both global element features and detailed features are taken into account. The two-stage training specifically improves the classification accuracy of difficult-to-distinguish samples, and the cross-modal feature guidance further enhances the ability to distinguish between solid waste samples with similar elements, ultimately resulting in an element classification model that supports accurate classification.

[0132] In one embodiment, the method further includes a training process for an image classification model, which includes steps S301 to S313:

[0133] S301. Collect image data of each metal-based solid waste sample and label the metal-based solid waste samples in the image data to obtain labeled images;

[0134] Specifically, the image classification model can be an improved YOLOv5s model based on the embedded CBAM (Convolutional Block Attention Module) attention mechanism and cross-modal feature interaction module. The training process is as follows: First, image data of each metal-based solid waste sample is collected, and the image data is labeled using the LabelImg tool. The label information is saved in XML (Extensible Markup Language) format. In practical applications, image classification models can also include Faster R-CNN (Faster Region-based Convolutional Neural Networks), lightweight convolutional neural networks (MobileNetV2 / V3, ShuffleNetV2), classic CNN models (ResNet18 / 34 (Residual Network 18 layers / 34 layers), DenseNet121 (Densely Convolutional Network 121 layers), and few-shot models such as Siamese Network and Prototypical Network are also very suitable.

[0135] For example, during the annotation process, blocky solid waste samples are discrete individuals. Even when overlapping, they possess independent edge and morphological features. The model can learn the features of individual individuals to achieve classification. Therefore, overlapping blocky samples are annotated with individual bounding boxes, with the box precisely fitting the edge of each blocky sample and excluding the area of ​​other overlapping individuals. Powdered solid waste samples, on the other hand, are continuous accumulations without independent individual boundaries, making it impossible to split and select them. They need to be annotated as a whole region. Therefore, powdered samples are annotated with region bounding boxes, with the box covering the entire accumulation area of ​​powder, without needing to distinguish the powder details within the area. One bounding box corresponds to the solid waste sample category. Through this annotation, individual-level bounding box annotation allows the model to learn the individual features of blocky solid waste samples and accurately identify each block in an overlapping scene. Region-level bounding box annotation allows the model to learn the regional texture and color features of powdered solid waste samples, adapting to their continuous distribution characteristics. This avoids confusion in model feature learning caused by meaningless splitting and selecting, and the differentiated annotations perfectly match the classification needs of actual production, improving the model's industrial adaptability. It should be noted that the category labeling requirements for both types of annotation are consistent: each box must be bound to the corresponding solid waste sample category number / category name (e.g., 1=high zinc powder, 2=electroplating sludge) in the XML annotation file. Differentiated design in terms of the granularity and range of the box selection is essentially to make the annotation data more closely match the characteristics of solid waste samples of different forms, thereby improving the feature extraction and classification accuracy of the model.

[0136] S302. Using a pre-trained initial element prediction model, predict the element information of metal-based solid waste samples in the labeled image to obtain element category pseudo-labels. Then, associate the image data, the labeling information of the image data, and the element category pseudo-labels to form a dual-label image dataset.

[0137] The pre-trained initial element prediction model corresponds to a simplified version of the element classification model in this application, which omits the S202 cross-modal feature fusion step during training. It does not perform the cross-modal auxiliary feature concatenation of S202, but instead trains by constructing an input feature matrix using the original element types and contents. The input to this initial element prediction model is the element type and content feature matrix extracted from S201, and the labels are the actual solid waste sample category labels corresponding to the element information. After training, the initial element prediction model learns the mapping relationship from element information to solid waste sample categories.

[0138] Specifically, the elemental information of the metal-based solid waste sample in the labeled image is obtained through spectral or chemical analysis. The elemental information of the metal-based solid waste sample in the labeled image is predicted using a trained initial elemental prediction model, such as an initial random forest model. The initial elemental prediction model outputs the elemental category prediction result of the sample, i.e., the elemental category pseudo-label. The image data, the labeling information of the image data and the elemental category pseudo-label are associated to form a dual-label image dataset.

[0139] S303. Extract the second training set from the dual-label image dataset and perform data augmentation on the second training set to obtain the image augmentation training set.

[0140] Specifically, the dual-label image dataset is divided into a second training set and a second test set according to a preset ratio. The preset ratio can be 7:3, 8:2, etc., which is not limited here. The training set is expanded by Mosaic+ cross-modal constraint data augmentation technology to obtain the expanded image training set.

[0141] S304. Initialize the model training parameters of the image classification model. Based on the pre-acquired unlabeled image dataset, pre-train the initialized image classification model to obtain pre-trained weights, and load the pre-trained weights into the image classification model. The image classification model after loading the pre-trained weights includes at least a backbone module, a neck module, a head module, and a pseudo-label supervision branch. The backbone module includes at least a feature extraction module, an attention module, and a spatial pyramid pooling module. The neck module includes at least a feature fusion module and a cross-modal feature interaction module.

[0142] Specifically, the model training parameters of the image classification model are initialized, including: initializing the input image size to 640×640, the number of iterations to 300, the batch size to 8, and the learning rate to 0.01.

[0143] Taking the improved YOLOv5s model as an example, the original YOLOv5s model includes at least a backbone module, a neck module, and a head module. The backbone module includes at least a feature extraction module CSP1_X and a spatial pyramid pooling module SPPF, and the neck module includes at least a feature fusion module CSP2_X. The improved YOLOv5s model has made certain improvements to the original YOLOv5s model: 1. A CBAM attention module is added after the CSP1_X module to enhance the extracted image features with channel and spatial dimensions; 2. A cross-modal feature interaction module is added to the Neck module to map the category features of the pseudo-labels to feature vectors and perform cross-channel attention fusion with the image features to output enhanced multimodal fusion features; 3. A pseudo-label supervision branch is connected to the output of the Neck module to adapt to the dual-label training requirements. It should be noted that the CBAM module weights and enhances the low-level / mid-level visual features (such as edges, textures, color blocks, contours, local shapes, etc.) extracted by the model autonomously from the original RGB image, rather than the morphological features extracted manually.

[0144] At this point, the first stage of contrastive learning pre-training is performed on the image classification model. Based on the pre-acquired unlabeled image dataset of metal-based solid waste samples, the MoCo3 contrastive learning framework is used to pre-train the Backbone and Neck modules to construct positive and negative sample pairs. Then, the general feature representation of the unlabeled image dataset is learned through the contrastive loss function to obtain the pre-trained weights. The second stage of pseudo-label supervised fine-tuning is then performed on the image classification model. The pre-trained weights obtained are loaded into the image classification model, and the underlying parameters of the Backbone module are frozen. This allows the updated image classification model to learn the general visual features of metal-based solid waste (such as shape, texture, and contour), improving the feature extraction capability and model generalization in small sample scenarios.

[0145] In an optional embodiment, after pre-training the improved YOLOv5s Backbone+Neck with MoCov3, the output pre-trained weights do not include the parameters of the Head detection head and the pseudo-label supervision branch. Instead, they are the convolution kernel parameters, bias parameters, and weight / coefficient parameters of the attention module for all trainable layers in the Backbone+Neck module for image feature extraction. Specifically, they are broken down as follows: Backbone part: convolution parameters of the Focus structure, convolution / residual layer parameters of all CSP1_X modules, channel / spatial attention weight parameters of the CBAM attention module after each CSP1_X, and pooling and convolution parameters of the SPPF module; Neck part: convolution parameters of all CSP2_X modules, parameters of the two 1×1 convolutional layers of the cross-modal feature interaction module, correlation calculation / weight parameters of the cross-channel attention layer, and convolution fusion parameters of the upsampling / feature splicing layer. The above parameters are the core trainable weights for the model to extract image features. After pre-training, the optimal values ​​of these parameters will be saved as a weight file. During pseudo-label supervised fine-tuning, this file will be loaded directly. After freezing the underlying parameters of the Backbone, the upper-layer parameters will be fine-tuned and optimized without retraining the core feature extraction layer.

[0146] S305. Input the image augmentation training set into the feature extraction module to obtain the mid-level semantic features of the image augmentation training set. Then, use the attention module to perform weighted enhancement of the mid-level semantic features of the image by channel dimension and spatial dimension to obtain the image semantic enhancement features.

[0147] Specifically, based on the image classification model with pre-trained weights, joint training of feature extraction and object detection is performed. First, the image augmentation training set is sliced, downsampled, and convolutionally mapped through the Focus structure at the input end to extract shallow basic features of the image, resulting in an initial feature map of fixed dimensions. The initial feature map is then sequentially input into the feature extraction module CSP1_X to extract the mid-level semantic features of the image augmentation training set. Then, the mid-level semantic features of the image are weighted and enhanced by the CBAM attention module to strengthen the key visual features of the solid waste sample, resulting in image semantic enhancement features.

[0148] S306. Input the image semantic enhancement features into the spatial pyramid pooling module to obtain the global deep features of the image, and reduce the dimensionality of the global deep features of the image to obtain the backbone feature map.

[0149] Specifically, the image semantic enhancement features are sampled in parallel by multi-size pooling kernels of the Spatial Pyramid Pooling Module (SPPF) to achieve multi-receptive field fusion, extract global deep features of the image, and output a unified-dimensional backbone feature map through convolutional dimensionality reduction.

[0150] S307. The feature fusion module performs internal residual fusion on the backbone feature map to obtain deep semantic features of the image. The cross-modal feature interaction module performs cross-channel attention fusion on the pseudo-label feature vector corresponding to the pseudo-label of the element category and the deep semantic features of the image to obtain attention fusion features. The attention fusion features are then upsampled and concatenated with the mid-level semantic features of the image.

[0151] Specifically, the backbone feature map is input into the Neck module. First, the feature fusion module CSP2_X completes the internal residual fusion of the backbone feature map to purify the deep semantic features of the image. Then, through the cross-modal feature interaction module, the pseudo-label of the element category is mapped to the pseudo-label feature vector, and the pseudo-label feature vector is fused with the image features through cross-channel attention to obtain the attention fusion feature. Subsequently, the size is enlarged by the upsampling module and stitched together with the mid-level semantic features of the image from the skip connection of the Backbone module.

[0152] The process of cross-channel attention fusion between the pseudo-label feature vector and the image feature is as follows: 1. Dimension matching: The mapped element pseudo-label feature vector is spatially expanded by a 1×1 convolution to generate an element feature map with the same size and number of channels as the corresponding scale image feature; 2. Channel attention calculation: The image feature map and the element feature map are globally averaged to obtain channel feature vectors, which are then normalized by dot product and sigmoid to generate a cross-channel attention weight matrix; 3. Channel weighted fusion: The image feature map and the attention weight matrix are multiplied element-wise by channel to enhance the image channel features related to the element category; 4. Feature integration: The features are then linearly integrated by a 1×1 convolution to output a multimodal fusion feature map with unchanged size and number of channels, which is then fed into the next level module of Neck.

[0153] S308. The spliced ​​features are bidirectionally fused through the neck module to obtain multi-scale cross-modal fusion features.

[0154] Specifically, the stitched features are fused bidirectionally from top to bottom and from bottom to top by the Neck module. Since the three CSP2_X modules of the Neck correspond to three scale image features, each scale independently completes the above fusion process. Therefore, the final output is a multi-scale cross-modal fused feature, which is then fed into the head module and the pseudo-label supervision branch.

[0155] S309. Input the multi-scale cross-modal fusion features into the head module to obtain the image classification result, and obtain the image classification loss based on the image classification result and annotation information.

[0156] Specifically, multi-scale cross-modal fusion features are input into the Head module. Anchor box matching generates candidate boxes for object detection. Non-maximum suppression (NMS) is used to remove redundant boxes, outputting image classification results (including object category, bounding box coordinates, and classification confidence). This completes object detection training. Based on the image classification results and the annotation information from S301, an image classification loss is obtained. This image classification loss is the CIoU loss (balancing classification and bounding box regression, adapting to object detection scenarios) and is the core loss for image classification. Simply put: Image classification loss = f(image classification result from the Head module, ground truth image annotation information from S301).

[0157] S310. Through the pseudo-label supervision branch, the multi-scale cross-modal fusion features are mapped to the element feature space to obtain the element category prediction results. Based on the element category prediction results and the element category pseudo-labels, the pseudo-label prediction loss is obtained.

[0158] Specifically, the multi-scale cross-modal fusion features are directly input into the newly added pseudo-label supervision branch. After being processed by feature mapping and classification layers (such as fully connected + Softmax), the branch outputs the element category prediction results. The element category prediction results are combined with the element category pseudo-labels to calculate the pseudo-label prediction loss (such as cross-entropy loss).

[0159] S311. Based on the multi-scale cross-modal fusion features, obtain the fine-tuned features corresponding to the image data. Using the image classification model with pre-trained weights, generate the target features corresponding to the image data. By minimizing the difference between the fine-tuned features and the target features, obtain the contrastive learning regularization loss.

[0160] Specifically, the contrastive learning regularization loss uses the core loss from the contrastive learning pre-training stage as a regularization term in the fine-tuning stage. This constrains the feature extraction direction of the Backbone and Neck modules, preventing the loss of general features of solid waste images learned during pre-training. Its data source and computational logic are strongly correlated with the S305 pre-training stage, specifically:

[0161] (1) Core data sources: Basic data: unlabeled image datasets acquired in advance; Training generated data: positive and negative sample pairs constructed by the MoCov3 framework during the pre-training stage (positive samples: different enhanced versions of the same image; negative samples: enhanced versions of other images, stored in the momentum queue); Fine-tuning stage associated data: image features extracted by the Backbone+Neck module during fine-tuning (must be consistent with the feature dimensions and extraction methods in the pre-training stage).

[0162] (2) Calculation logic: The contrastive learning regularization loss in the fine-tuning stage is a lightweight adaptation of the contrastive loss (InfoNCE loss) in the pre-training stage. The core is to ensure that the image features extracted in the fine-tuning stage are consistent with the feature distribution in the pre-training stage. The specific process is as follows: 1. Pre-training: Construct positive and negative sample pairs through MoCov3, calculate the InfoNCE loss, and let the model learn the general features of solid waste images where the positive sample features are similar and the negative sample features are different; 2. Fine-tuning: Freeze the underlying parameters of the Backbone, and use the fine-tuned features corresponding to the currently extracted image data and the image classification model after loading the pre-training weights in the pre-training stage to generate the target features corresponding to the image data for distribution consistency constraint. The calculated loss is the contrastive learning regularization loss, which prevents the model from overfitting to the local features of pseudo-labels / image annotations in small sample fine-tuning and improves the model's generalization ability.

[0163] In an optional embodiment, the contrastive learning regularization loss in the fine-tuning phase directly uses the same formula, specifically:

[0164] .

[0165] Among them, L CL To compare the learning loss, q represents the fine-tuning feature, k+ represents the positive sample feature (i.e., the target feature), k represents the negative sample set, k+ represents the negative sample feature, and sim(a, b) represents the cosine similarity. Here is the temperature coefficient. The entire formula can be understood as follows: the model calculates the similarity score between feature q and positive sample k+, divides it by the sum of its similarity scores with all negative samples k, and then takes the negative logarithm to obtain the contrastive learning regularization loss.

[0166] S312. Based on the image classification loss, label prediction loss, and contrastive learning regularization loss, optimize the image classification model to obtain the image classification model.

[0167] Specifically, the total model loss is calculated by combining the image classification loss, label prediction loss, and contrastive learning regularization loss (k1×image classification loss + k2×pseudo-label prediction loss + k3×contrastive learning regularization loss). In practical applications, k1=0.6, k2=0.25, and k3=0.15, which are not limited here. Based on the total model loss, the image classification model is optimized to obtain the image classification model.

[0168] In an optional embodiment, the performance of the initial image classification model is verified using a test set to obtain the image classification accuracy and pseudo-label prediction accuracy of the initial image classification model. If the image classification accuracy is greater than a preset first accuracy threshold and the pseudo-label prediction accuracy is greater than a preset second accuracy threshold, such as image classification accuracy ≥ 95% and pseudo-label prediction accuracy ≥ 90%, then the trained image classification model is obtained.

[0169] In the above embodiments, the YOLOv5s model structure is specifically improved. The CBAM attention module enhances the extraction of key image features, and the cross-modal feature interaction module achieves deep fusion of elemental features and image features, making the classification of metal-based solid waste more scientific and comprehensive. In the first stage, the model backbone network is pre-trained using the MoCov3 contrastive learning framework to learn the general features of solid waste images, addressing the problem of poor model generalization in small sample scenarios. In the second stage, pre-trained weights are loaded, a pseudo-label supervision branch is added, and multi-task loss is jointly calculated to complete fine-tuning, ensuring that the model can simultaneously fit the correlation between image features and elemental attributes. This significantly improves the accuracy of the trained model.

[0170] In one exemplary embodiment, such as Figure 3 The diagram shown is the complete structure of the improved YOLOv5s model. Its complete structure and connections are as follows:

[0171] I. Overall Architecture Hierarchy: Input Focus Structure → Backbone (Feature Extraction Module CSP1_X, CBAM Attention Module, Spatial Pyramid Pooling Module SPPF) → Neck (Feature Fusion Module CSP2_X, Cross-Modal Feature Interaction Module) → Dual-Branch Output (Main Detection Branch + Pseudo-Label Supervision Branch); The model input size is 640×640×3 (RGB image). The pseudo-label feature vector is connected to the cross-modal feature interaction module of the Neck in the form of a one-dimensional feature vector, providing cross-modal information supplementation for image features.

[0172] II. Detailed structure and connection relationships of each module:

[0173] (a) Input Focus structure: The original 640×640×3 image is downsampled to reduce the amount of computation while preserving the image details and features, and a high-dimensional feature map is output to the first CSP1_X module of Backbone.

[0174] Structural details: The 640×640×3 image is transformed into a 320×320×12 feature map by slicing (stride=2), and then the number of channels is compressed to 32 by a 3×3 convolutional layer (stride=1, padding=1). The activation function is SiLU.

[0175] (ii) Backbone module: The original YOLOv5s Backbone consists of Conv convolutional layers + 4 CSP1_X modules + SPPF. After improvement, a CBAM attention module is connected in series at the output of each CSP1_X module to achieve dual weighted enhancement of image features in both channel and spatial dimensions, highlighting key features such as the shape, texture, and edges of solid waste.

[0176] The basic building blocks of Backbone are connected in the following order: Convolutional layer Conv (channel count increased from 32 to 64) → Feature extraction module CSP1_X_1 + Attention module CBAM_1 → Convolutional layer Conv (channel count increased from 64 to 128) → Feature extraction module CSP1_X_2 + Attention module CBAM_2 → Convolutional layer Conv (channel count increased from 128 to 256) → Feature extraction module CSP1_X_3 + Attention module CBAM_3 → Convolutional layer Conv (channel count increased from 256 to 512) → Feature extraction module CSP1_X_4 + Attention module CBAM_4 → Spatial pyramid pooling module SPPF.

[0177] Each Conv layer consists of a 3×3 convolution + SiLU activation + batch normalization layer, doubling the number of channels and downsampling the feature map (halving the size). CSP1_X is a classic cross-stage partial connection module in YOLOv5, which preserves shallow features through residual branches and extracts deep features through convolution in the main branch, outputting a feature map with the same number of input channels and size. The CBAM attention module outputs an H×W×C feature map (H=height, W=width, C=number of channels) from CSP1_X, which is first processed by the channel attention submodule (global average pooling + global max pooling → shared MLP (Multi-Layer)). The channel weights are obtained through a perceptron (multilayer perceptron) → sigmoid weighting, and then spatial weights are obtained through a spatial attention submodule (channel-dimensional average pooling + max pooling → 7×7 convolution → sigmoid weighting). This suppresses background, noise, and invalid region features, achieving feature enhancement. The final output is a weighted enhanced feature map. The input / output size and number of channels are completely consistent, with no additional dimensional changes. More specifically, the processing of the CBAM attention module includes:

[0178] I. Channel Attention Weighted Calculation: Focusing on the importance of different feature channels (e.g., one channel corresponds to the edge features of solid waste, while another channel corresponds to the background features), each channel is assigned a weight coefficient of 0 to 1 to strengthen effective channels and weaken ineffective channels.

[0179] 1. Global Pooling: Perform global average pooling and global max pooling on the input feature map to compress the H×W×C feature map into two 1×1×C channel feature vectors; 2. Global Average Pooling: Calculate the average of all feature values ​​in each channel to reflect the overall feature strength of the channel; 3. Global Max Pooling: Extract the maximum feature value in each channel to reflect the local key feature strength of the channel; 4. Shared MLP Mapping: Feed the two 1×1×C feature vectors into a shared two-layer perceptron (first layer has C / 4 neurons, ReLU activation function; second layer has C...). 1. **No activation:** Map the feature vectors to new 1×1×C feature vectors to achieve non-linear transformation of channel features; 2. **Feature addition and activation:** Add the two feature vectors output by the MLP element-wise, and then normalize the result to the 0~1 interval through the Sigmoid activation function to obtain the channel attention weight matrix (1×1×C); 3. **Channel weighting:** Multiply the original input feature map (H×W×C) and the channel attention weight matrix (1×1×C) by channel. All feature values ​​of each channel are multiplied by the corresponding channel weight coefficient to complete the feature enhancement of the channel dimension.

[0180] II. Spatial Attention Weighted Calculation: Based on the channel-weighted feature map, the spatial importance of the feature map is focused (e.g., the solid waste main area is an important location, and the background area is a secondary location). A weight coefficient of 0 to 1 is assigned to each spatial location of the feature map to strengthen key spatial areas and weaken ineffective areas.

[0181] 1. Channel Pooling: Perform average pooling and max pooling along the channel dimensions on the channel-weighted feature map (H×W×C) to compress the H×W×C feature map into two H×W×1 spatial feature maps. 2. Channel Average Pooling: Calculate the average of all channel feature values ​​at each spatial location, reflecting the overall feature response at that location. 3. Channel Max Pooling: Extract the maximum channel feature value at each spatial location, reflecting the local key feature response at that location. 4. Feature Concatenation and Convolution: Concatenate the two H×W×1 spatial feature maps along the channel dimension into an H×W×2 feature map, and then... 5. Convolution transformation: The convolution output feature map is normalized to the 0~1 interval using the Sigmoid activation function to obtain the spatial attention weight matrix (H×W×1); 6. Spatial weighting: The channel-weighted feature map (H×W×C) is multiplied by the spatial attention weight matrix (H×W×1) according to the spatial position. All channel feature values ​​at each spatial position are multiplied by the corresponding weight coefficient to complete the spatial dimension feature enhancement.

[0182] 3. Enhanced feature map output: The feature map after double weighting by channel attention and spatial attention is the output of the CBAM module, which is directly fed into the next level Conv layer / SPPF module to continue participating in the Backbone feature extraction process.

[0183] The SPPF module is used to perform pooling at four scales (1×1, 5×5, 9×9, 13×13) on the 512-channel feature map, and outputs a 512×2 channel feature map with a size of 8×8 (640 after multiple downsampling), realizing multi-scale feature fusion and improving the feature extraction capability for solid waste of different sizes. The Conv layer of the Backbone is alternately connected to the CSP1_X module. The output of each CSP1_X is directly connected to the corresponding CBAM module, and the output of the CBAM is used as the input of the next Conv layer. The output of the last CBAM_4 is sent to the SPPF module. The output of SPPF is divided into two paths: the main path is sent to the first CSP2_X module of the Neck, and the branch path is used as the upsampled fusion feature source of the Neck (8×8×1024). At the same time, the outputs of CSP1_X_3+CBAM_3 (16×16×512) and CSP1_X_2+CBAM_2 (32×32×256) in the Backbone are used as multi-scale feature jump connections and are sent to the corresponding upsampled modules of the Neck to achieve the fusion of shallow / deep features (the original YOLOv5s FNP (Feature Pyramid Network) + PAN (Path Aggregation Network)).

[0184] (III) Neck Feature Fusion Network: It adopts a bidirectional feature fusion structure of top-down and bottom-up. The original structure consists of 3 CSP2_X modules + upsampling layer + convolutional fusion layer. After improvement, a cross-modal feature interaction module is added to the output of each CSP2_X module. The core realizes the cross-modal fusion of multi-scale features of image and pseudo-label category features of elements, so that the image model can learn element attribute information and make up for the shortcomings of pure visual features.

[0185] The basic components of Neck are connected in the following order: Feature Fusion Module CSP2_X_1 + Cross-modal Feature Interaction Module 1 → Upsampling × 2 + Feature Concatenation → Feature Fusion Module CSP2_X_2 + Cross-modal Feature Interaction Module 2 → Upsampling × 2 + Feature Concatenation → Feature Fusion Module CSP2_X_3 + Cross-modal Feature Interaction Module 3 → Downsampling × 2 + Feature Concatenation → Feature Fusion Module CSP2_X_2 → Downsampling × 2 + Feature Concatenation → Feature Fusion Module CSP2_X_1.

[0186] Inputs to the cross-modal feature interaction module: two features ① multi-scale feature maps of the image output by Neck's CSP2_X module (e.g., CSP2_X_1 outputs 8×8×1024, CSP2_X_2 outputs 16×16×512, CSP2_X_3 outputs 32×32×256); ② one-dimensional class feature vectors obtained by mapping element pseudo-labels through a feature embedder + 1×1 convolution (the dimension matches the number of channels in the image feature map, e.g., an 8×8×1024 feature map corresponds to a 1024-dimensional element feature vector).

[0187] The cross-modal feature interaction module consists of two 1×1 convolutional layers and one cross-channel attention layer: the first 1×1 convolution maps the one-dimensional feature vector of the element pseudo-label to a feature map with the same size as the image feature map (e.g., 1024-dimensional vector → 8×8×1024 feature map), achieving dimension matching; the cross-channel attention layer performs channel-dimensional attention weighting on the image feature map and the mapped element feature map (calculating the channel correlation between the two features to generate attention weights), thus enhancing the image feature map; the second 1×1 convolution is used to compress the number of feature channels after fusion, keeping it consistent with the number of channels in the original CSP2_X output, avoiding dimensional redundancy; finally, the cross-modal feature interaction module outputs the attention fusion feature of the fused element pseudo-label features, with the same size and number of channels as the input features, and is directly fed into the next Neck module.

[0188] Neck's CSP2_X module is connected in series with the cross-modal feature interaction module. The output of the interaction module serves as the input for Neck's downsampling / upsampling / feature concatenation. The element pseudo-label features of the cross-modal feature interaction module, and the element category pseudo-labels are processed by the feature mapping layer and then integrated with the image features in real time. Finally, Neck outputs three enhanced feature maps of different scales through bidirectional fusion, which are directly fed into the Head module, while also providing feature input for the pseudo-label supervision branch.

[0189] (iv) Output end: dual-branch structure (head module Head + pseudo-label supervision branch): The improved model adopts dual-branch parallel output to adapt to the dual-label training requirements of image annotation labels + element pseudo-labels. The two branches share the multi-scale enhanced feature map output by Neck, and jointly update the model parameters after calculating the loss separately.

[0190] 1. Head Module: Implements target detection and image classification of metal-based solid waste, outputting the bounding box coordinates, confidence score, and image category label (consistent with the category labeled by LabelImg). It consists of three detection head convolutional layers, corresponding to the three scale feature maps output by the Neck. Each detection head contains a 3×3 convolution + a 1×1 convolution. The number of output channels of the 1×1 convolution is 3×(5+N) (3 is the number of anchor boxes, 5 is the bounding box coordinates + confidence score, and N is the number of solid waste categories). By performing non-maximum suppression (NMS) on the feature maps output by the detection heads, redundant bounding boxes are removed, and the final image classification result (category + confidence score) is output, corresponding to the second classification confidence matrix.

[0191] 2. Pseudo-label Supervision Branch: This branch predicts pseudo-labels for element categories, allowing the model to fit the association between image features and element attributes, thus completing cross-modal collaborative training. It follows the last convolutional fusion layer of Neck and consists of two fully connected layers (FC) and a Softmax activation layer: The first fully connected layer flattens the multi-scale feature map output by Neck into a one-dimensional feature vector, compressing it to 512 dimensions; the second fully connected layer maps the 512-dimensional features into an N-dimensional feature vector (N is the number of solid waste categories, e.g., N=11); the Softmax activation layer transforms the N-dimensional feature vector into an element category probability distribution, outputting the predicted pseudo-labels for the elements.

[0192] The output of this branch is used to calculate the cross-entropy loss with the pseudo-label of the element category. This loss is combined with the CIoU loss of the main branch and the contrastive learning regularization loss to form the total loss (e.g., total model loss = 0.6 × CIoU loss + 0.25 × pseudo-label cross-entropy loss + 0.15 × L2 regularization loss).

[0193] In one embodiment, the second training set is augmented to obtain an image augmented training set, including steps S401 to S403:

[0194] S401, in response to performing a random fluctuation enhancement operation on the element feature vector corresponding to the image data in the second training set, determine the image perturbation parameters corresponding to the image data, and perform perturbation adjustment on the image data of the metal-based solid waste sample based on the image perturbation parameters to obtain the first enhanced dataset;

[0195] S402, in response to performing a stitching and fusion operation on the image data of multiple metal-based solid waste samples, obtain the information of the elements to be enhanced corresponding to each image data, and perform weighted average fusion processing on the information of the elements to be enhanced to obtain the second enhanced dataset;

[0196] S403, add the first augmented dataset and the second augmented dataset to the second training set to obtain the image augmented training set.

[0197] Specifically, a cross-modal data augmentation strategy is adopted to expand the training set. Specifically, when performing random fluctuation augmentation on the element feature vectors corresponding to the image data in the second training set, the image perturbation parameters corresponding to the image data are determined, and the brightness and contrast of the image data are simultaneously adjusted based on these parameters to obtain the first augmented dataset, while maintaining the sample category attributes. When performing Mosaic stitching augmentation on the image data, for example, stitching four images into one, the element information corresponding to each image data is simultaneously weighted and averaged to ensure semantic consistency between the augmented elements and the image data. Finally, the first and second augmented datasets are added to the second training set to obtain the image augmented training set. It should be noted that the element feature vectors corresponding to the image data and the element information to be augmented in the image data refer to the information that is matched after spatiotemporal synchronous detection. The core principle of Mosaic stitching is that images are stitched according to spatial area proportions, and element data are weighted and averaged according to the same area proportion. The element detection device detects the average element content of a region. If the image capturing device captures a mixed region, the spectrometer measures the average element value of that region.

[0198] In the above embodiments, by using two cross-modal linkage enhancement methods, when one modality is enhanced, the other modality will make corresponding and logically consistent adjustments to maintain the semantic consistency of cross-modal data, thereby significantly improving the generalization ability of the model in real industrial mixed scenarios.

[0199] In one embodiment, the metal-based solid waste sample classification method based on multimodal fusion further includes S501~S503:

[0200] S501. Obtain the element classification weights w of the element classification model. e Image classification weights w in image classification models i ,in, .

[0201] S502, When the first category and the second category are consistent, use Calculate the fused confidence matrix and P merge The category with the highest probability was determined as the category of the metal-based solid waste sample to be tested, P. merge Let Pe be the fused confidence matrix, and Pi be the first confidence matrix and Pi be the second confidence matrix.

[0202] S503. When the class difference exceeds the preset difference threshold, extract the important element feature content in the information of the element to be tested and the important morphological feature content in the image information to be tested. Based on the important element feature content, the important morphological feature content, the first confidence matrix and the second confidence matrix, construct a conflict discrimination vector and input the conflict discrimination vector into the pre-trained conflict resolution classifier to obtain the class of the metal-based solid waste sample to be tested.

[0203] Specifically, after obtaining the first confidence matrix, the first reliability score of the first confidence matrix, the second confidence matrix, and the second reliability score of the second confidence matrix, it is necessary to determine whether there is a conflict in the category with the highest confidence in the bimodal model. This can be divided into three cases:

[0204] 1. No conflict triggered (S502): When the first category corresponding to the highest confidence level in the first confidence matrix is ​​the same as the second category corresponding to the highest confidence level in the second confidence matrix, use it directly. Calculate the fused confidence matrix and P merge The category with the highest probability was determined as the category of the metal-based solid waste sample to be tested, P. merge Let Pe be the fused confidence matrix, and Pi be the first confidence matrix and Pi be the second confidence matrix.

[0205] 2. Triggering conflict, but with little difference (S102~S103): When the first category and the second category are inconsistent, calculate the category difference between the first category and the second category; when the category difference is within a preset range (e.g., 0.3~0.6), trigger the first-level resolution mechanism, and use the reliability score weighted voting method to sum the full category confidence of each of the two modes, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, which will not be elaborated here.

[0206] 3. Triggering a conflict with a large degree of difference (S503): When the first category and the second category are inconsistent and the category difference exceeds the preset difference threshold (e.g., exceeding 0.6), a secondary resolution mechanism is triggered. The important element feature content (e.g., the element feature content of the top 3) in the information of the element to be tested and the important morphological feature content (e.g., the morphological feature content of the top 3) in the information of the image to be tested are extracted. Based on the first confidence matrix and the second confidence matrix, the confidence distribution of the bimodal model (e.g., the highest confidence, the second highest confidence, the confidence variance, etc.) is obtained. The important element feature content, the important morphological feature content and the confidence distribution of the bimodal model are concatenated to construct a 10-dimensional conflict discrimination vector. This ensures that the conflict discrimination vector can fully integrate the category judgment tendency and uncertainty of the bimodal model. The conflict discrimination vector is then input into a pre-trained conflict resolution CNN classifier (the training data is historical conflict samples) to obtain the category of the metal-based solid waste sample to be tested. In practical applications, a lightweight CNN classifier consists of 3 convolutional layers (3×3 kernel size) and 2 fully connected layers, with ReLU activation function and Softmax output layer.

[0207] For example: The sample to be tested is electroplating sludge. After testing: First confidence matrix: electroplating sludge 30%, metallized pellets 65% (highest category is metallized pellets), Re=0.70; Second confidence matrix: electroplating sludge 95%, metallized pellets 2% (highest category is electroplating sludge), Ri=0.95; The cosine distance between the two highest categories is 0.75, which is greater than 0.6, triggering secondary deep resolution; Extract the top 3 elemental features (such as copper, zinc, and chromium content) and top 3 morphological features (such as area, texture entropy, and color mean), and construct a 10-dimensional conflict discrimination vector by combining the bimodal confidence distribution. Input the vector into a pre-trained lightweight CNN classifier, and the classifier outputs the final result as electroplating sludge.

[0208] In an optional embodiment, the Top 3 element feature contents are determined as follows: based on the feature importance ranking of the element classification model (cascaded random forest), during the random forest training process, the total decrease in Gini impurity caused by each element content feature when splitting across all decision tree nodes is statistically analyzed. The larger this value, the greater the contribution of the element to solid waste classification and the stronger its discriminative ability. All element content features are arranged in descending order of the above importance values, and the top 3 elements are selected as the Top 3 key element contents for the current prediction task.

[0209] The method for determining the content of the top 3 morphological features is based on the channel attention weights or feature gradient weights of the improved YOLOv5s model. The core basis is the model's attention to visual features. Specifically, it utilizes the channel attention weight matrix output by the cross-modal feature interaction module in the Neck module (or the channel weights of the CBAM module in Backbone). The higher the weight value, the more critical the morphological feature (texture, edge, color, area-related response) of that channel is to the current classification. The high-weight feature channels are mapped back to the corresponding manually defined morphological feature set (area, perimeter, texture entropy, color mean, etc.), sorted in descending order of weight, and the top 3 morphological features are selected as the top 3 key morphological features.

[0210] In the above embodiments, differentiated resolution strategies are adopted for different conflict levels to address the problem of insufficient accuracy in traditional single resolution methods. When no conflict is triggered, the resolution is based on the element classification weight w. e Image classification weight w i Output P merge The category with the highest probability is selected. When a conflict is triggered, the result is output after hierarchical resolution.

[0211] In one embodiment, S501 includes S601~S605:

[0212] S601. Construct a reliability assessment function, and based on the reliability assessment function, calculate the historical element classification reliability score and historical image classification reliability score under different historical environmental conditions during the historical classification process of metal-based solid waste samples.

[0213] S602. Using historical environmental information under different historical environmental conditions as input features, and the normalized values ​​of historical element classification reliability scores and historical image classification reliability scores corresponding to historical environmental information as output labels, train a nonlinear mapping model between input features and output features.

[0214] S603. Input the current environmental information corresponding to the metal-based solid waste sample to be tested into the nonlinear mapping model to obtain the initial element classification weights w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0 ;

[0215] S604. Obtain the historical weight distribution information of metal-based solid waste samples, and construct a prior probability model based on the historical weight distribution information; wherein, the historical weight distribution information includes the initial weight w of the historical element classification corresponding to the historical classification process where the classification result is correct. e0历 Initial weights w for historical image classification i0历 Initial weights w for historical element classification e0历 The optimized historical element classification target weight w e历Initial weights w for historical image classification i0历 The optimized historical image classification target weights w i历 ;

[0216] S605. The optimization objective is to maximize the maximum confidence value in the fused confidence matrix. Based on the prior probability model, initial weights w are assigned to the element classification. e0 Image classification initial weights w i0 Through iterative optimization, the target weight w for element classification is obtained. e Image classification target weight w i .

[0217] Specifically, this application employs a four-level decision fusion architecture comprising dynamic modal credibility assessment, multi-factor nonlinear weight fitting, Bayesian weight adaptive update, and hierarchical conflict resolution. It constructs and trains a multimodal fusion classification model by dynamically mapping the nonlinear relationship between environmental information and the weights of the element classification model and image classification model through a machine learning model. Based on this model, the metal-based solid waste sample to be tested is classified. In the three stages of dynamic modal credibility assessment, multi-factor nonlinear weight fitting, and Bayesian weight adaptive update, the element classification weights w of the element classification model are obtained. e Image classification weights w in image classification models i If the fourth level of conflict resolution is not triggered, then the element classification weight w is used. e Image classification weights w in image classification models i This enables the classification of metal-based solid waste samples to be tested.

[0218] In the first three levels of the decision fusion architecture, the weights of the element classification model are defined as w. e The image classification model has weights w i And satisfy .

[0219] Level 1: Dynamic Evaluation of Modal Reliability: Constructing a Reliability Evaluation Function Where M is a single-modal model (such as an element classification model and an image classification model), Acc(M) is the model test set accuracy, that is, the classification accuracy of the trained single-modal model M on the independent test set, Conf(M) is the mean of the current prediction confidence matrix, and Ent(M) is the entropy value of the confidence matrix (reflecting prediction uncertainty). , , For weighting coefficients (e.g., =0.4, =0.4, =0.2), when M is an elemental classification model, the historical elemental classification reliability score under different historical environmental conditions is calculated during the historical classification process of metal-based solid waste samples; when M is an image classification model, the historical image classification reliability score under different historical environmental conditions is calculated during the historical classification process of metal-based solid waste samples. Acc(M), Conf(M) and Ent(M) achieve a positive evaluation of modal credibility with high accuracy, high confidence and low uncertainty, and finally calculate the elemental model Re and the image model Ri, providing the core basis for subsequent weight fitting.

[0220] The calculation logic of Acc(M) is as follows: The data source corresponds to the test set divided in S204 and S303. The elements / image data of the test set are input into the finally trained single-modal model M to obtain the model prediction result. The result is compared with the true label of the test set sample, and the number of correct classifications is divided by the total number of test sets, which is Acc(M). This indicator reflects the model's stable classification ability on unknown data. It is the basic static indicator for modal credibility evaluation and provides the performance benchmark of the model itself for the credibility function.

[0221] The specific calculation method of Ent(M): For the category confidence matrix of solid waste sample classification (a single sample corresponds to a 1×N dimensional matrix, where N is the number of solid waste categories), the entropy value is calculated using Shannon entropy. The specific formula and calculation steps are as follows: Core formula , where p i The predicted confidence level (0) of the i-th type of solid waste in the confidence matrix. <p i ≤1, and N represents the total number of categories of metal-based solid waste samples. If the model has extremely high predictive certainty for a certain category (e.g., a certain p...), then... i ≈1, the rest ≈0), the entropy value approaches 0, representing extremely low prediction uncertainty; if the model prediction is fuzzy and scattered (each p i (The values ​​are close), and the entropy value approaches log2(N), which means that the prediction uncertainty is extremely high. Before calculation, it is necessary to ensure that all values ​​of the confidence matrix are normalized to the range of 0 to 1 and summed to 1, which is consistent with the output form of the classification confidence matrix.

[0222] Level 2: Multi-factor nonlinear weight fitting: Using historical environmental information (light intensity, dust concentration, sample surface humidity) under different historical environmental conditions as input features, and the normalized values ​​of historical element classification reliability scores and historical image classification reliability scores corresponding to historical environmental information as output labels, a gradient boosting decision tree (GBDT) model is trained as a weight mapper to replace the original logistic regression model. The gradient boosting decision tree model is a nonlinear mapping model used to fit the nonlinear mapping relationship between environmental information, modal reliability, and weights. Then, the current environmental information corresponding to the metal-based solid waste sample to be tested is obtained, and the current environmental information is input into the nonlinear mapping model to obtain the initial element classification weights w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0 .

[0223] Before training the nonlinear mapping model, a standard regression training set needs to be constructed. The input feature matrix X has a dimension of N×3, where each row represents the environmental state of a sample, and the columns represent the light intensity, dust concentration, and sample surface humidity after normalization to the Min-Max interval [0, 1]. The output label matrix Y has a dimension of N×2, where each row corresponds to the label of the input features, and the first column Y1 is the historical element classification reliability score R of the element classification model under the Min-Max normalization in this environment. e历 The second column, Y2, represents the Min-Max normalized historical image classification reliability score R of the image classification model under this environment. i历 This process does not share a tree structure; instead, it trains two independent GBDT regressors in parallel. One GBDT regressor generates a regression tree through multiple iterations, continuously fitting the predicted values ​​to R0. e历 The residuals ultimately learn environmental information and R e历 The nonlinear function, and the same applies to another GBDT regressor, learns environmental information and R... i历 The nonlinear function is used to synthesize a nonlinear mapping model between input and output features. After training, the current environment information is input into this nonlinear mapping model, and the model outputs image classification predictions and element classification predictions. The model output is directly used as the initial weight w for the corresponding element classification. e0 Image classification initial weights w i0 In practical applications, Min-Max normalization is performed as follows: M is a single-mode model, R(M), R min (M), R max (M) represents the current reliability score, the maximum reliability score, and the minimum reliability score of the single-modal model during the historical classification process, respectively.

[0224] In an optional embodiment, the suitable threshold range for light intensity is [L]. min L max The values ​​were calibrated experimentally to be [500 lux, 5000 lux].

[0225] Level 3: Bayesian Weight Adaptive Update: Obtain historical weight distribution information for metal-based solid waste samples, construct a prior probability model based on this information, and then use a Bayesian optimization algorithm to update the weight distribution. e0 w i0 The process involves iterative optimization, with the goal of maximizing the maximum confidence value in the fused confidence matrix. The output is the optimal adaptive element classification target weight w. e Image classification target weight w i In practical applications, the number of iterations for Bayesian optimization can be set to 50, the acquisition function is the desired improvement function, and the constrained range of the optimized weights is w. e ∈[0.05, 0.95]、w i ∈[0.05, 0.95].

[0226] Among them, the historical weight distribution information refers to the initial weights of the bimodal fusion (historical element classification initial weights w) of all samples whose classification results have been verified to be correct under different industrial operating conditions. e0历 Initial weights w for historical image classification i0历 ) and the final optimized target weights (historical element classification initial weights w) e0历 The optimized historical element classification target weight w e历 Initial weights w for historical image classification i0历 The optimized historical image classification target weights w i历 The statistical distribution data of ) is a set of historical sample datasets containing environmental information, initial weights, correct optimization target weights, and classification results. The core is the numerical distribution characteristics of the weights (such as mean, variance, probability density, interval distribution, etc.).

[0227] In the above embodiments, in the four-level decision fusion architecture of dynamic modal confidence assessment, multi-factor nonlinear weight fitting, Bayesian weight adaptive update, and hierarchical conflict resolution, the first level quantifies the current prediction confidence of the bimodal model through a reliability assessment function; the second level uses the GBDT model to fit the nonlinear relationship between environmental information, modal reliability, and weights, replacing the traditional fixed threshold and linear fitting, which is more in line with complex industrial conditions; the third level uses a Bayesian optimization algorithm to adaptively update the initial weights, aiming to maximize the classification confidence after fusion, to achieve dynamic optimal solution of the weights and obtain the accurate element classification target weight w. e Image classification target weight w iThis allows for accurate classification of the metal-based solid waste samples to be tested.

[0228] In one embodiment, the method further includes a process for acquiring information about the element to be measured and information about the image to be measured, including steps S701 to S702:

[0229] S501, perform coordinate matching between the detection area of ​​the element detection device and the shooting area of ​​the image capturing device, so that the acquired image information to be tested and the data of the element to be tested correspond to the same detection area.

[0230] S502 generates a synchronized element detection trigger signal and an image acquisition trigger signal. The element detection trigger signal triggers the element detection device to detect the element information of the metal-based solid waste sample to be tested. At the same time, the image acquisition trigger signal triggers the image capturing device to acquire the image information of the metal-based solid waste sample to be tested.

[0231] Specifically, in the process of collecting elemental and image information of the metal-based solid waste sample to be tested, it is necessary to complete the spatiotemporal alignment calibration of the element-image paired sample in order to solve the problem of regional mismatch and time asynchrony between element detection and image acquisition in industrial scenarios, and ensure the semantic consistency of the paired sample.

[0232] Specifically, the spatiotemporal alignment calibration of element-image paired samples involves: using the geometric center of the metal-based solid waste sample to be tested (the geometric center of the two-dimensional projection on the detection platform) as the reference point, performing pixel-level coordinate matching between the detection area of ​​the element detection device and the imaging area of ​​the image capturing device to ensure that the element information and image information correspond to the same detection area; simultaneously, synchronizing the element detection trigger signal and the image acquisition trigger signal through timestamps to complete the temporal dimension alignment, and triggering the element detection device to detect the element information of the metal-based solid waste sample to be tested through the element detection trigger signal, while triggering the image capturing device to acquire the image information of the metal-based solid waste sample to be tested through the image acquisition trigger signal. This hardware-level trigger synchronization method ensures that the element data and image data are acquired for the same sample at the same time point, thereby eliminating the risk of sample misalignment caused by time difference.

[0233] In an optional embodiment, the elemental information includes at least the element type and elemental content. The elemental information detection process can be obtained using elemental detection equipment (such as a wavelength dispersive X-ray fluorescence spectrometer using WDXRF (Wavelength Dispersive X-ray Fluorescence Spectroscopy)). Detection parameters include six different combinations of scanning angle ranges, spectroscopic crystals, filters, detectors, tube voltage, and tube current, covering element detection from oxygen to uranium. When lower precision is required, chemical titration, gravimetric analysis, or colorimetric reaction methods can be used to determine the element type and elemental content. Image data can be obtained using image capturing equipment (such as a 1920×1080 resolution RGB (Red Green) image capture device). The camera (using the three primary colors of red, green, and blue) captures images from a top-down perspective. The imaging scene of the device covers single, independent samples, overlapping samples of the same category, samples of different categories with a small amount of overlap, and samples with different textures. More specifically, it captures single and overlapping samples for blocky solid waste samples, and samples with different textures and edge states for powdery solid waste samples, ensuring the diversity of the dataset. At the same time, this application can also collect and detect the ambient dust concentration through a dust concentration sensor and collect the sample surface humidity through a sample surface humidity sensor to obtain the current environmental information, providing multi-dimensional input for subsequent weight mapping, enabling the weights of multimodal fusion to have working condition adaptive capabilities, and completely solving the pain point of unstable classification accuracy of traditional fixed weighted fusion in complex industrial environments.

[0234] In an optional embodiment, pixel-level coordinate matching is performed between the detection area of ​​the element detection device and the shooting area of ​​the image capturing device. This can be considered as pixel-level coordinate matching between the WDXRF detection area and the camera shooting area. In this case, a two-dimensional pixel coordinate system is established with the detection stage as the reference and the geometric center of the sample's two-dimensional projection as the origin. Pixel-level matching is completed through three steps: calibration plate calibration, coordinate mapping, and error correction. The specific steps are as follows:

[0235] Step 1: Using the effective detection plane of the WDXRF inspection station as the reference plane, fix the RGB camera directly above the inspection station, keeping the top-down shooting angle, shooting height, and focal length unchanged throughout the process; establish a physical coordinate system (unit: mm) with any corner of the reference plane as the physical origin, and at the same time convert the image captured by the camera into a pixel coordinate system (unit: pixel), and record the mapping ratio k between pixels and physical size (k=pixel / mm, obtained by calibrating the camera resolution and shooting height, such as 1 pixel corresponding to 0.1mm).

[0236] Step 2: Using a high-precision crosshair calibration plate (with crosshair graduations of known physical dimensions, accuracy ±0.01mm), place the calibration plate on the reference surface of the testing platform, ensuring the center of the crosshair on the calibration plate coincides with the center of the preset standard testing position on the testing platform; start WDXRF, set its testing area to the circular testing area corresponding to the center of the crosshair on the calibration plate (e.g., 10mm in diameter), and record the center coordinates (X, Y, X) of this testing area in the physical coordinate system. w Y w The physical coordinate range of the area boundary; start the camera to capture an image of the calibration board, and extract the pixel coordinates (X, Y) of the center of the crosshair of the calibration board through image recognition. c Y c Simultaneously, the pixel coordinate range corresponding to the boundary of the WDXRF detection area in the image is identified; with the cross center of the calibration board as the reference, the physical coordinates of the WDXRF detection area are converted into the initial coordinates in the camera pixel coordinate system through the mapping ratio k, thus completing the reference calibration of the two regions.

[0237] Step 3: Extract the two-dimensional projection geometric center pixel coordinates (X) of the metal-based solid waste sample to be tested in the camera image through visual recognition. s Y s The sample is then moved to the detection stage so that its physical geometric center coincides with the detection center of the WDXRF; the sample's two-dimensional projected geometric center (X) is then used. s Y s Using the WDXRF detection area (physical coordinates) as the matching reference point, the WDXRF detection area is transformed into the target pixel area in the camera pixel coordinate system according to the mapping ratio k; the target pixel area is then aligned at the pixel level to ensure that the physical boundary of the WDXRF detection area corresponds to a unique pixel boundary in the camera image, thus achieving a one-to-one pixel correspondence between the detection area and the shooting area.

[0238] Step 4: Repeat calibration steps 5-10 times, calculate the pixel offset error between the center of the WDXRF detection area and the center of the camera shooting area, and take the average value as the system error value; during the coordinate mapping process, compensate the system error value in real time, and eliminate the random error caused by slight sample placement offset, to ensure that the final matching pixel offset error is ≤1 pixel, which meets the pixel-level matching requirements.

[0239] Step 5: After completing one calibration, the coordinate mapping relationship and error correction value obtained from the calibration can be directly reused if the camera shooting parameters, WDXRF detection parameters, and detection stage position remain unchanged; when the equipment parameters / position are adjusted, the calibration process can be repeated.

[0240] In the above embodiments, the element-image spatiotemporal alignment calibration solves the problem of regional mismatch and time asynchrony between element detection and image acquisition in industrial scenarios, ensuring the semantic consistency of paired samples; and the cross-modal linkage data augmentation strategy solves the problem of cross-modal data semantic disconnect caused by traditional single-modal augmentation, improving the quality and generalization of the dataset.

[0241] This application discloses a classification method for metal-based solid waste based on multimodal fusion, which aims to solve the problems of low accuracy and weak anti-interference ability of single-modal classification models for complex metal-based solid waste, as well as poor adaptability of existing model structures and insufficient generalization ability of training methods. This method includes: constructing an elemental information dataset and an image information dataset of metal-based solid waste, and optimizing the dataset quality through element-image spatiotemporal alignment calibration and cross-modal linkage data augmentation strategies; training a cascaded random forest elemental information single-modal classification model based on multi-scale feature fusion, and optimizing the model using a two-stage progressive training method; training an improved YOLOv5s image information single-modal classification model embedded with a CBAM attention mechanism and a cross-modal feature interaction module, and optimizing the model using a two-stage training paradigm of contrastive learning pre-training + pseudo-label supervised fine-tuning; adopting a four-level decision fusion architecture of modality credibility dynamic evaluation, multi-factor nonlinear weight fitting, Bayesian weight adaptive update, and hierarchical conflict resolution, and constructing a multimodal fusion classification model by dynamically mapping the nonlinear relationship between environmental information and the weights of the dual-modal model through a machine learning model; performing element detection and image acquisition on the metal-based solid waste to be tested, and outputting the classification results through the trained multimodal fusion classification model. This application combines the stability of elemental information with the feature discrimination advantages of image information. Through targeted improvements to the model structure and innovations in cross-modal collaborative training paradigms, it significantly enhances the accuracy, reliability, and generalization ability of metal-based solid waste classification under complex working conditions, achieving rapid classification feedback of solid waste in industrial production. To facilitate understanding of this multimodal fusion-based metal-based solid waste sample classification method by those skilled in the art, such as... Figure 4 As shown, the technical solution of this application will be described in detail below with reference to specific embodiments:

[0242] S1. Eleven types of metal-based solid waste samples were collected, including electroplating sludge, low-zinc powder, high-zinc powder, and metallized pellets, with 2 to 11 samples collected for each type. WDXRF was used to detect each sample using six sets of detection parameters, and the element types and contents of each sample were obtained. A 1920×1080 resolution RGB camera was used to capture images of each sample from above, and image data of each sample was acquired to construct element information datasets and image information datasets. Environmental information was collected using a dust concentration sensor (measurement range 0-100 mg / m³) and a sample surface humidity sensor (measurement range 0-100% RH). Using the geometric center of the sample as a reference, pixel-level coordinate matching was completed between the WDXRF detection area and the image capture area. Time stamps were synchronously acquired via hardware triggering to complete element-image spatiotemporal alignment calibration. Finally, element information datasets, image information datasets, and environmental information datasets were constructed.

[0243] S2. A cascaded random forest element classification model with multi-scale feature fusion is trained based on the element information dataset, using a two-stage progressive training method:

[0244] 2.1 The 11 solid waste samples were numbered (1~11), and a 118-dimensional input feature matrix was constructed based on the element content, and an output label array was constructed based on the number.

[0245] 2.2 The target area, perimeter, texture entropy, and mean RGB three-channel color values ​​of each sample image are extracted using OpenCV, standardized to form 8-dimensional auxiliary features, and concatenated to the input feature matrix to obtain a 126-dimensional enhanced input feature matrix;

[0246] 2.3 The enhanced input feature matrix is ​​decomposed into three layers using db4 (Daubechies 4 Wavelet) to extract low-frequency global features and high-frequency detail features, and a 126-dimensional global feature subset and a 378-dimensional detail feature subset are constructed respectively.

[0247] 2.4 For electroplating sludge (11 samples), KNN interpolation was used to generate an augmented dataset; for other samples (2 samples), the random fluctuation algorithm was used. =0.5) Generate an augmented dataset;

[0248] 2.5 Divide the augmented dataset into the first training set and the first test set in an 8:2 ratio, and initialize the hyperparameters of the element classification model: First-level global forest: number of decision trees 80, maximum depth 8, node split feature subset size 8, minimum number of leaf node samples 5; Second-level detail forest: number of decision trees 120, maximum depth 12, node split feature subset size 12, minimum number of leaf node samples 3.

[0249] 2.6 First-stage pre-training: The Bootstrap method is used to sample the first training set, and the low-frequency global features of the sampled first samples are input into the first-level global random forest of the element classification model. Based on the principle of minimizing Gini impurity, the nodes are split, and the first-level classification results of each first sample and the first-level classification confidence of each first-level classification result are output. Difficult samples with a first-level classification confidence of less than 0.7 are selected.

[0250] 2.7 Second-stage fine-tuning and optimization: For hard-to-classify samples, a secondary detail random forest is trained using high-frequency detail features. The priority weight for cross-modal auxiliary feature splitting is set to 1.2. The primary classification results and primary classification confidence of the hard-to-classify samples are converted into numerical prior features and concatenated with the high-frequency detail features of the hard-to-classify samples to obtain concatenated features. The concatenated features of the hard-to-classify samples are input into the secondary detail random forest of the element classification model to complete the cascaded joint optimization and output the secondary classification results and secondary classification confidence.

[0251] 2.8 Calculate the first classification error rate (out-of-bag error) and cross-modal feature contribution. Stop the iteration when the first classification error rate is less than 0.05 and the cross-modal feature contribution is greater than 5%, and output the trained element classification model (final out-of-bag error 0.032, cross-modal feature contribution 7.8%).

[0252] S3. Train an improved YOLOv5s image classification model that incorporates the CBAM attention mechanism and cross-modal feature interaction module. The model is trained based on an image information dataset. A two-stage training paradigm—contrastive learning pre-training + pseudo-label supervised fine-tuning—is used to optimize the MoI algorithm, resulting in a well-trained image classification model.

[0253] 3.1 Use LabelImg to annotate image data, annotate single target boxes for blocky solid waste, and annotate region boxes for powdery solid waste, and save as XML format;

[0254] 3.2 The basic random forest model (without cross-modal features and out-of-bag error of 0.08) is used to predict the element information of metal-based solid waste samples in the labeled images to obtain pseudo-labels of element categories. The image data, the labeling information of the image data and the pseudo-labels of element categories are associated to form a dual-label image dataset.

[0255] 3.3 The dual-label image dataset is divided into a second training set and a second test set at an 8:2 ratio. The training data is augmented using the Mosaic technique to obtain an augmented image training set (the matching between image texture and element category pseudo-labels is maintained during the augmentation process).

[0256] 3.4 Constructing an improved YOLOv5s model: In the original YOLOv5s Backbone, a CBAM convolutional attention module is embedded after each CSP1_X module; a cross-modal feature interaction module is added after the CSP2_X module in the Neck. This module contains two 1×1 convolutional layers and one cross-channel attention layer, which is used to fuse element pseudo-label features with image features.

[0257] 3.5 Initialize the training parameters of the improved YOLOv5s model: input image size 640×640, Epoch=300, Batchsize=8, initial learning rate 0.01, and adopt a cosine annealing learning rate decay strategy;

[0258] 3.6 First stage of contrastive learning pre-training: 5000 unlabeled images of metal-based solid waste were collected to construct a pre-training dataset. The MoCov3 contrastive learning framework was used, with the momentum coefficient set to 0.999 and the temperature coefficient set to 0.07. The improved YOLOv5s Backbone and Neck modules were pre-trained for 100 epochs to obtain the pre-trained weights.

[0259] 3.7 Second-stage pseudo-label supervised fine-tuning: Load pre-trained weights, freeze the parameters of the first 3 layers of the backbone, and perform joint training of feature extraction and object detection based on the improved YOLOv5s model; set the total loss function to: total loss = 0.6 × image classification loss CIoU + 0.25 × pseudo-label prediction loss + 0.15 × L2 contrastive learning regularization loss, and complete 200 epochs of fine-tuning training;

[0260] 3.8 Test set validation: Image classification accuracy is 97.6%, pseudo-label prediction accuracy is 93.2%, and the trained image classification model is output.

[0261] S4. A four-level decision fusion architecture is adopted, consisting of dynamic evaluation of modal credibility, multi-factor nonlinear weight fitting, Bayesian weight adaptive update, and hierarchical conflict resolution. A multimodal fusion classification model is constructed and trained by dynamically mapping the nonlinear relationships between environmental information and element classification models, and between environmental information and image classification models, through a machine learning model.

[0262] 4.1 Set the element classification weights w in the element classification model e Image classification weights w in image classification models i w e +w i =1, environmental information is normalized to [0, 1] using Min-Max;

[0263] 4.2 First-level modal reliability assessment: Setting =0.4, =0.4, =0.2, the model test set accuracy Acc(M) of the element classification model = 0.968, and the model test set accuracy Acc(M) of the image classification model = 0.976; based on the reliability evaluation function Calculate the historical element classification reliability score and historical image classification reliability score under different historical environmental conditions during the historical classification of metal-based solid waste samples;

[0264] 4.3 Second-level nonlinear weight fitting: Using light intensity, dust concentration, and sample surface humidity under different historical environmental conditions as inputs, and the normalized values ​​of historical element classification reliability scores and historical image classification reliability scores corresponding to historical environmental information as output labels, a GBDT weight mapper is trained. The tree size is set to 100, the maximum depth to 6, and the learning rate to 0.1. The trained GBDT weight mapper is then used to process the current environmental information corresponding to the metal-based solid waste sample to be tested, outputting the initial element classification weights w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0 ;

[0265] 4.4 Third-level Bayesian weight adaptive update: The number of Bayesian optimization iterations is set to 50, the acquisition function is the EI function, and the weight constraint range is we∈[0.05, 0.95]. The optimization objective is to maximize the maximum confidence value in the fused confidence matrix. The weights are updated based on the constructed prior probability model. e0 With w i0 Through iterative optimization, the target weight w for element classification is obtained. e Image classification target weight w i ;

[0266] 4.5 Fourth-level hierarchical conflict resolution: The threshold for first-level resolution is set to cosine distance of class 0.3~0.6, and the threshold for second-level resolution is cosine distance >0.6; a lightweight CNN classifier is pre-trained with 200 sets of historical conflict samples as the training set, and the test accuracy is 95.8%.

[0267] S5. Perform elemental detection, image acquisition, and environmental information acquisition (light intensity 3200 lux, dust concentration 15 mg / m³, surface humidity 12% RH) on the metal-based solid waste sample (metallized pellets) to be tested, and obtain the elemental information and image information of the sample to be tested.

[0268] S6. Input the information of the element to be tested into the trained element classification model. The trained element classification model outputs the first confidence matrix (metallized pellets 45.2%, metallized pellets sieve-passed powder 42.8%) and the first reliability score Re=0.71. Input the information of the image to be tested into the trained image classification model. The trained image classification model outputs the second confidence matrix (metallized pellets 97.3%, metallized pellets sieve-passed powder 0.5%) and the second reliability score Ri=0.74. At this time, the highest category of the two modalities is consistent, and conflict resolution is not triggered, proceeding to S7. If the highest category of the two modalities is inconsistent, the hierarchical conflict resolution mechanism is activated, proceeding to S8.

[0269] S7. Initial weights w are obtained through GBDT fitting. e0 =0.33、w i0 =0.67, and the adaptive weight w is obtained after Bayesian optimization. e =0.29、w i =0.71; Based on adaptive weights, the first confidence matrix and the second confidence matrix are weighted and fused to obtain P. merge (Metalized pellets 97.3×0.71+45.2×0.29≈82.2%, Metallized pellets sieve powder 0.5×0.71+42.8×0.29≈12.8%); The output classification result is metallized pellets.

[0270] S8. Calculate the class dissimilarity between the first category corresponding to the highest confidence level in the first confidence matrix and the second category corresponding to the highest confidence level in the second confidence matrix.

[0271] S9. When the class difference is between 0.3 and 0.6, a mild resolution mechanism is triggered, i.e., using S... i =Re×Pe i +Ri×Pi i Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0272] S10. When the class difference exceeds 0.6, a heavy resolution mechanism is triggered to extract the important element feature content in the information of the element to be tested and the important morphological feature content in the image information to be tested. Based on the important element feature content, the important morphological feature content, the first confidence matrix and the second confidence matrix, a conflict discrimination vector is constructed and input into the pre-trained conflict resolution classifier to obtain the class of the metal-based solid waste sample to be tested.

[0273] In the above embodiments, the improved multimodal fusion classification model achieves a significant improvement in classification performance through core innovations in model structure and training methods: the cascaded random forest model improves the classification accuracy of similar metallized spheres and sieved powder by 15.3%, and the recognition rate of difficult samples by 21.7%; the improved YOLOv5s model improves the detection accuracy of overlapping blocky solid waste and powdery solid waste by 9.6%, and reduces the generalization error by 12.4% in small sample scenarios; the four-level fusion strategy maintains a classification accuracy of over 97.5% under harsh working conditions such as insufficient light and high dust, which is 9.2% higher than the traditional weighted fusion scheme, and the average classification accuracy reaches 98.7% under overall complex working conditions.

[0274] In summary, the advantages of this application in the classification of metal-based solid waste samples based on multimodal fusion are as follows:

[0275] 1. Targeted innovation in model structure: Targeting the classification characteristics of metal-based solid waste, a cascaded random forest model with multi-scale feature fusion is designed. Through a two-level cascaded structure, both global element features and detailed features are taken into account, which solves the problem that traditional single-level random forests have weak ability to distinguish solid waste with similar elements. An improved YOLOv5s model with embedded CBAM attention and cross-modal feature interaction modules is designed to enhance the key feature extraction capability of overlapping solid waste and powdery solid waste, realize the deep fusion of element and image features, and significantly improve the adaptability of model structure.

[0276] 2. Breakthrough optimization of training paradigm: For element classification models, a two-stage progressive training method is proposed. Difficult samples are screened through pre-training, and targeted fine-tuning is performed to significantly improve the classification accuracy of difficult samples. For image classification models, a two-stage training paradigm of contrastive learning pre-training + pseudo-label supervised fine-tuning is proposed. This solves the problem of poor model generalization ability caused by insufficient labeled samples in industrial scenarios. At the same time, cross-modal information collaborative training is achieved through multi-task joint loss, breaking the limitation of independent training of bimodal models.

[0277] 3. Hierarchical upgrade of fusion strategy: A four-level decision fusion architecture is proposed, which adds Bayesian weight adaptive update and hierarchical conflict resolution on the original basis. Multi-factor nonlinear weight fitting is achieved through GBDT model to replace the traditional linear weighting with fixed threshold. Differential resolution strategy is used to handle modal conflicts of different degrees, which significantly improves the technical barrier and greatly enhances the anti-interference ability and classification stability under complex working conditions.

[0278] 4. Source optimization of dataset quality: The addition of elements - image spatiotemporal alignment calibration and cross-modal linkage data augmentation strategy - solves the problems of semantic mismatch between cross-modal samples and semantic disconnect of augmented data from the source, providing a high-quality dataset foundation for model training and further improving the generalization ability of the model.

[0279] 5. Maintaining the advantage of rapid classification: All model improvements are based on lightweight and efficient basic modules without significantly increasing the amount of computation. The classification time for a single sample is controlled within 200ms, which fully meets the real-time and rapid feedback requirements of industrial production.

[0280] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0281] Based on the same inventive concept, this application also provides a multimodal fusion-based metal-based solid waste sample classification device for implementing the aforementioned multimodal fusion-based metal-based solid waste sample classification method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal fusion-based metal-based solid waste sample classification device provided below can be found in the limitations of the multimodal fusion-based metal-based solid waste sample classification method described above, and will not be repeated here.

[0282] In one embodiment, such as Figure 5 As shown, a metal-based solid waste sample classification device based on multimodal fusion is provided, comprising: a single-modal analysis module 101, a multimodal analysis module 102, and a classification module 103, wherein:

[0283] The single-modal analysis module 101 is used to input the element information of the metal-based solid waste sample to be tested into a pre-trained element classification model and output a first confidence matrix and a first reliability score of the first confidence matrix. It also inputs the image information of the metal-based solid waste sample to be tested into a pre-trained image classification model and outputs a second confidence matrix and a second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample belonging to each category.

[0284] The multimodal analysis module 102 is used to calculate the class difference between the first category and the second category when the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix.

[0285] Classification module 103 is used to classify categories when the difference in category size falls within a preset range. Calculate the weighted total score of each category in the association between the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

[0286] In one embodiment, the apparatus further includes an element classification model training module, which is used to: take the elemental information of multiple metal-based solid waste samples with attached category labels as input features and initialize the hyperparameters of the element classification model, wherein the elemental information includes element type and content; extract multiple morphological features from the sample images of each metal-based solid waste sample and stitch the morphological features and input features of the metal-based solid waste sample to obtain the enhanced input features of each metal-based solid waste sample; perform multi-scale decomposition of the enhanced input features through wavelet transform to obtain the low-frequency global features and high-frequency detail features of each metal-based solid waste sample; when the total number of low-frequency global features and high-frequency detail features of the metal-based solid waste sample is greater than a first threshold or less than a second threshold, perform data augmentation processing based on the low-frequency global features and high-frequency detail features to obtain the enhanced dataset of each metal-based solid waste sample, and select a first training set from each enhanced dataset, wherein the first threshold is greater than the second threshold; sample the first training set and input the low-frequency global features of the sampled first samples into the first-level global random forest of the element classification model, perform node splitting based on the Gini impurity minimization principle, and output the first-level classification results of each first sample and each first-level classification result. The system uses the primary classification confidence score of the class results to filter out difficult-to-classify samples whose primary classification confidence score is lower than the third threshold. The primary classification results and confidence scores of these difficult-to-classify samples are converted into numerical prior features and concatenated with the high-frequency detail features of the difficult-to-classify samples to obtain concatenated features. These concatenated features are then input into the secondary detail random forest of the elemental classification model, outputting the secondary classification results and secondary classification confidence scores. The high-frequency detail features of the second samples not sampled in the first training set are input into the elemental classification model during training, outputting the predicted classification results for each second sample. The first classification error rate is obtained by comparing with the true classification results. The morphological features of each metal-based solid waste sample are exchanged to obtain the exchanged morphological features of each metal-based solid waste sample. Based on the exchanged morphological features of the metal-based solid waste samples and the enhanced input features after splicing the input features, the step of multi-scale decomposition by wavelet transform is repeated to obtain the second classification error rate. The difference between the first classification error rate and the second classification error rate is calculated to obtain the cross-modal feature contribution. When the first classification error rate is less than the fourth threshold and the cross-modal feature contribution is greater than the fifth threshold, the trained element classification model is obtained.

[0287] In one embodiment, the apparatus further includes an image classification model training module, which is used to: collect image data of each metal-based solid waste sample and label the metal-based solid waste samples in the image data to obtain labeled images; use a pre-trained initial element prediction model to predict the element information of the metal-based solid waste samples in the labeled images to obtain element category pseudo-labels, and associate the image data, the labeling information of the image data, and the element category pseudo-labels to form a dual-label image dataset; divide the dual-label image dataset into a second training set and a test set, and perform data augmentation processing on the second training set to obtain an image augmentation training set; initialize the model training parameters of the image classification model; and based on the pre-trained initial element prediction model, predict the element information of the metal-based solid waste samples in the labeled images to obtain element category pseudo-labels, and associate the image data, the labeling information of the image data, and the element category pseudo-labels to form a dual-label image dataset; divide the dual-label image dataset into a second training set and a test set, and perform data augmentation processing on the second training set to obtain an image augmentation training set; initialize the model training parameters of the image classification model; and perform pre-training processing on the image classification model. First, an unlabeled image dataset is acquired. The initialized image classification model is then pre-trained to obtain pre-trained weights, which are then loaded onto the model. The image classification model with pre-trained weights includes at least a backbone module, a neck module, a head module, and a pseudo-label supervision branch. The backbone module includes at least a feature extraction module, an attention module, and a spatial pyramid pooling module. The neck module includes at least a feature fusion module and a cross-modal feature interaction module. The image augmentation training set is then input into the feature extraction module to obtain the mid-level semantic features of the image. The attention module then performs channel- and spatial-dimensional weighted enhancement on these mid-level semantic features to obtain image semantic enhancement features. The image semantic enhancement features are input into the spatial pyramid pooling module to obtain global deep features of the image. These global deep features are then dimensionality-reduced to obtain a backbone feature map. The feature fusion module performs internal residual fusion on the backbone feature map to obtain deep semantic features of the image. Through the cross-modal feature interaction module, the pseudo-label feature vectors corresponding to the element category pseudo-labels are fused with the deep semantic features of the image through cross-channel attention to obtain attention fusion features. These attention fusion features are then upsampled and concatenated with the mid-level semantic features of the image. The concatenated features are then bidirectionally fused through the neck module to obtain multi-scale cross-modal fusion features. Finally, the multi-scale cross-modal fusion features are input into the head module to obtain image classification. As a result, based on the image classification results and annotation information, the image classification loss is obtained; through the pseudo-label supervision branch, the multi-scale cross-modal fusion features are mapped to the element feature space to obtain the element category prediction results, and based on the element category prediction results and element category pseudo-labels, the pseudo-label prediction loss is obtained; based on the multi-scale cross-modal fusion features, the fine-tuned features corresponding to the image data are obtained, and the target features corresponding to the image data are generated using the image classification model with pre-trained weights loaded, and the contrastive learning regularization loss is obtained by minimizing the difference between the fine-tuned features and the target features; based on the image classification loss, label prediction loss and contrastive learning regularization loss, the image classification model is optimized to obtain the image classification model.

[0288] In one embodiment, the apparatus further includes a data augmentation module, which is configured to: respond to performing a random fluctuation enhancement operation on the element feature vectors corresponding to the image data in the second training set, determine the image perturbation parameters corresponding to the image data, and perform perturbation adjustment on the image data of the metal-based solid waste sample based on the image perturbation parameters to obtain a first augmented dataset; respond to performing a stitching and fusion operation on the image data of multiple metal-based solid waste samples, obtain the information of the elements to be augmented corresponding to each image data, and perform weighted average fusion processing on the information of the elements to be augmented to obtain a second augmented dataset; and add the first augmented dataset and the second augmented dataset to the second training set to obtain an image augmentation training set.

[0289] In one embodiment, the apparatus further includes a sample category classification module, which is used to obtain the element classification weights w of the element classification model. e Image classification weights w in image classification models i ,in, When the first category and the second category are consistent, use Calculate the fused confidence matrix and P merge The category with the highest probability was determined as the category of the metal-based solid waste sample to be tested, P. merge Pe is the fused confidence matrix, Pi is the first confidence matrix, and Pi is the second confidence matrix. When the class difference exceeds the preset difference threshold, the important element feature content in the information of the element to be tested and the important morphological feature content in the image information to be tested are extracted. Based on the important element feature content, the important morphological feature content, the first confidence matrix and the second confidence matrix, a conflict discrimination vector is constructed. The conflict discrimination vector is then input into a pre-trained conflict resolution classifier to obtain the class of the metal-based solid waste sample to be tested.

[0290] In one embodiment, the apparatus further includes a weight acquisition module, which is used to construct a reliability evaluation function and, based on the reliability evaluation function, calculate the historical element classification reliability score and historical image classification reliability score under different historical environmental conditions during the historical classification process of the metal-based solid waste sample; using historical environmental information under different historical environmental conditions as input features, and using the normalized values ​​of the historical element classification reliability score and historical image classification reliability score corresponding to the historical environmental information as output labels, train a nonlinear mapping model between input features and output features; input the current environmental information corresponding to the metal-based solid waste sample to be tested into the nonlinear mapping model to obtain the initial element classification weight w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0The historical weight distribution information of metal-based solid waste samples was obtained, and a prior probability model was constructed based on the historical weight distribution information. The historical weight distribution information includes the initial weights w of historical element classifications corresponding to historical classification processes where the classification result was correct. e0历 Initial weights w for historical image classification i0历 Initial weights w for historical element classification e0历 The optimized historical element classification target weight w e历 Initial weights w for historical image classification i0历 The optimized historical image classification target weights w i历 The optimization objective is to maximize the maximum confidence value in the fused confidence matrix. Initial weights w for element classification are assigned based on the prior probability model. e0 Image classification initial weights w i0 Through iterative optimization, the target weight w for element classification is obtained. e Image classification target weight w i .

[0291] In one embodiment, the device further includes an information acquisition module, which is used to perform coordinate matching between the detection area of ​​the element detection device and the shooting area of ​​the image capturing device, so that the acquired image information to be tested and the data of the element to be tested correspond to the same detection area; generate synchronous element detection trigger signal and image acquisition trigger signal, and trigger the element detection device to detect the information of the element to be tested in the metal-based solid waste sample through the element detection trigger signal, and trigger the image capturing device to acquire the image information of the metal-based solid waste sample through the image acquisition trigger signal.

[0292] Each module in the aforementioned multimodal fusion-based metal-based solid waste sample classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0293] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores pre-trained data such as element classification models and image classification models. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a multimodal fusion-based method for classifying metal-based solid waste samples.

[0294] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0295] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0296] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0297] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0298] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0299] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0300] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for classifying metal-based solid waste samples based on multimodal fusion, characterized in that, The method includes: S101. Input the element information of the metal-based solid waste sample to be tested into a pre-trained element classification model, and output a first confidence matrix and a first reliability score of the first confidence matrix. Input the image information of the metal-based solid waste sample to be tested into a pre-trained image classification model, and output a second confidence matrix and a second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample to be tested belonging to each category. S102. When the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix, calculate the category difference degree between the first category and the second category. S103, when the category difference is within a preset range, use Calculate the weighted total score for each category associated with the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

2. The method according to claim 1, characterized in that, The method also includes a training process for the element classification model, which includes: S201. Using the elemental information of multiple metal-based solid waste samples with attached category labels as input features, the hyperparameters of the elemental classification model are initialized. The elemental information includes element type and content. S202. Extract multiple morphological features from the sample images of each of the metal-based solid waste samples, and stitch the morphological features and the input features of the metal-based solid waste samples together to obtain the enhanced input features of each of the metal-based solid waste samples. S203. The enhanced input features are decomposed into low-frequency global features and high-frequency detail features of each metal-based solid waste sample by wavelet transform. S204. When the total number of low-frequency global features and high-frequency detail features of the metal-based solid waste sample is greater than a first threshold or less than a second threshold, data augmentation processing is performed based on the low-frequency global features and high-frequency detail features to obtain augmented datasets for each metal-based solid waste sample, and a first training set is selected from each augmented dataset, wherein the first threshold is greater than the second threshold. S205. Sample the first training set and input the low-frequency global features of the sampled first samples into the first-level global random forest of the element classification model. Perform node splitting based on the principle of minimizing Gini impurity, output the first-level classification result of each first sample and the first-level classification confidence of each first-level classification result, and filter out the difficult samples whose first-level classification confidence is lower than the third threshold. S206. The first-level classification result and the first-level classification confidence of the difficult-to-classify sample are converted into numerical prior features, and then concatenated with the high-frequency detail features of the difficult-to-classify sample to obtain concatenated features. S207. Input the splicing features of the difficult-to-distinguish samples into the secondary detail random forest of the element classification model, and output the secondary classification result and the secondary classification confidence. S208. Input the high-frequency detail features of the second samples not sampled in the first training set into the element classification model in training, output the predicted classification results of each second sample, and compare the predicted classification results of each second sample with the true classification results to obtain the first classification error rate. S209. Exchange the morphological features between the metal-based solid waste samples to obtain the exchanged morphological features of each metal-based solid waste sample. Based on the exchanged morphological features of the metal-based solid waste samples and the enhanced input features obtained by splicing the input features, execute S203, S204, S205, S206, S207 and S208 to obtain the second classification error rate. Calculate the difference between the first classification error rate and the second classification error rate to obtain the cross-modal feature contribution. S210. When the first classification error rate is less than the fourth threshold and the cross-modal feature contribution is greater than the fifth threshold, the trained element classification model is obtained.

3. The method according to claim 1, characterized in that, The method also includes a training process for an image classification model, which includes: S301. Collect image data of each metal-based solid waste sample, and label the metal-based solid waste samples in the image data to obtain labeled images; S302. Using a pre-trained initial element prediction model, the element information of the metal-based solid waste sample in the labeled image is predicted to obtain element category pseudo-labels. The image data, the labeling information of the image data and the element category pseudo-labels are associated to form a dual-label image dataset. S303. Extract a second training set from the dual-label image dataset, and perform data augmentation processing on the second training set to obtain an image augmentation training set; S304. Initialize the model training parameters of the image classification model. Based on the pre-acquired unlabeled image dataset, pre-train the initialized image classification model to obtain pre-trained weights, and load the pre-trained weights into the image classification model. The image classification model after loading the pre-trained weights includes at least a backbone module, a neck module, a head module, and a pseudo-label supervision branch. The backbone module includes at least a feature extraction module, an attention module, and a spatial pyramid pooling module. The neck module includes at least a feature fusion module and a cross-modal feature interaction module. S305. Input the image augmentation training set into the feature extraction module to obtain the image mid-level semantic features of the image augmentation training set, and perform weighted enhancement of the image mid-level semantic features by channel dimension and spatial dimension through the attention module to obtain image semantic enhancement features; S306. Input the image semantic enhancement features into the spatial pyramid pooling module to obtain the global deep features of the image, and reduce the dimensionality of the global deep features of the image to obtain the backbone feature map. S307. The feature fusion module performs internal residual fusion on the backbone feature map to obtain deep semantic features of the image. The cross-modal feature interaction module performs cross-channel attention fusion on the pseudo-label feature vector corresponding to the element category pseudo-label and the deep semantic features of the image to obtain attention fusion features. The attention fusion features are then upsampled and concatenated with the mid-level semantic features of the image. S308. The spliced ​​features are bidirectionally fused through the neck module to obtain multi-scale cross-modal fused features; S309. Input the multi-scale cross-modal fusion features into the head module to obtain the image classification result, and obtain the image classification loss based on the image classification result and the annotation information; S310. Through the pseudo-label supervision branch, the multi-scale cross-modal fusion features are mapped to the element feature space to obtain the element category prediction result, and the pseudo-label prediction loss is obtained based on the element category prediction result and the element category pseudo-label. S311. Based on the multi-scale cross-modal fusion features, obtain the fine-tuned features corresponding to the image data, use the image classification model after loading the pre-trained weights to generate the target features corresponding to the image data, and obtain the contrastive learning regularization loss by minimizing the difference between the fine-tuned features and the target features. S312. Based on the image classification loss, the label prediction loss, and the contrastive learning regularization loss, the image classification model is optimized to obtain an image classification model.

4. The method according to claim 3, characterized in that, The step of performing data augmentation processing on the second training set to obtain an image augmented training set includes: S401, in response to performing a random fluctuation enhancement operation on the element feature vector corresponding to the image data in the second training set, determining the image perturbation parameters corresponding to the image data, and performing perturbation adjustment on the image data of the metal-based solid waste sample based on the image perturbation parameters to obtain the first enhanced dataset; S402, in response to performing a stitching and fusion operation on the image data of multiple metal-based solid waste samples, obtain the information of the element to be enhanced corresponding to each image data, and perform a weighted average fusion process on the information of the element to be enhanced to obtain a second enhanced dataset; S403, add the first augmented dataset and the second augmented dataset to the second training set to obtain the image augmented training set.

5. The method according to any one of claims 1-4, characterized in that, The metal-based solid waste sample classification method based on multimodal fusion also includes: S501. Obtain the element classification weights w of the element classification model. e Image classification weights w of the image classification model i ,in, ; S502, When the first category and the second category are the same, use Calculate the fused confidence matrix and P merge The category with the highest probability is determined as the category of the metal-based solid waste sample to be tested, P. merge Let Pe be the fused confidence matrix, and Pi be the first confidence matrix and Pi be the second confidence matrix. S503. When the category difference exceeds a preset difference threshold, extract the important element feature content in the element information to be tested and the important morphological feature content in the image information to be tested. Based on the important element feature content, the important morphological feature content, the first confidence matrix and the second confidence matrix, construct a conflict discrimination vector and input the conflict discrimination vector into a pre-trained conflict resolution classifier to obtain the category of the metal-based solid waste sample to be tested.

6. The method according to claim 5, characterized in that, S501 includes: S601. Construct a reliability assessment function, and calculate the historical element classification reliability score and historical image classification reliability score under different historical environmental conditions during the historical classification process of metal-based solid waste samples according to the reliability assessment function. S602. Using historical environmental information under different historical environmental conditions as input features, and using the normalized values ​​of the historical element classification reliability score and the historical image classification reliability score corresponding to the historical environmental information as output labels, train a nonlinear mapping model between the input features and the output features. S603. Input the current environmental information corresponding to the metal-based solid waste sample to be tested into the nonlinear mapping model to obtain the initial element classification weight w corresponding to the metal-based solid waste sample to be tested. e0 Image classification initial weights w i0 ; S604. Obtain historical weight distribution information of metal-based solid waste samples, and construct a prior probability model based on the historical weight distribution information; wherein, the historical weight distribution information includes the initial weight w of historical element classification corresponding to the historical classification process where the classification result is correct. e0历 Initial weights w for historical image classification i0历 The initial weight w for the classification of historical elements e0历 The optimized historical element classification target weight w e历 The initial weights w for the historical image classification i0历 The optimized historical image classification target weights w i历 ; S605. The optimization objective is to maximize the maximum confidence value in the fused confidence matrix. Based on the prior probability model, the initial weights w for element classification are... e0 Image classification initial weights w i0 Through iterative optimization, the target weight w for element classification is obtained. e Image classification target weight w i .

7. The method according to claim 1, characterized in that, The method further includes a process for acquiring the information of the element to be tested and the information of the image to be tested, wherein the process for acquiring the information of the element to be tested and the information of the image to be tested includes: S701, perform coordinate matching between the detection area of ​​the element detection device and the shooting area of ​​the image capturing device, so that the acquired image information to be tested and the element data to be tested correspond to the same detection area; S702, generate a synchronized element detection trigger signal and an image acquisition trigger signal, and trigger the element detection device to detect the element information to be tested in the metal-based solid waste sample through the element detection trigger signal, and trigger the image acquisition device to acquire the image information to be tested in the metal-based solid waste sample through the image acquisition trigger signal.

8. A metal-based solid waste sample classification device based on multimodal fusion, characterized in that, The device includes: The single-modal analysis module is used to input the element information of the metal-based solid waste sample to be tested into a pre-trained element classification model and output a first confidence matrix and a first reliability score of the first confidence matrix. It also inputs the image information of the metal-based solid waste sample to be tested into a pre-trained image classification model and outputs a second confidence matrix and a second reliability score of the second confidence matrix. The elements in the first confidence matrix and the second confidence matrix represent the probability confidence of the metal-based solid waste sample to be tested belonging to each category. The multimodal analysis module is used to calculate the class difference between the first category and the second category when the first category corresponding to the highest confidence level in the first confidence matrix is ​​inconsistent with the second category corresponding to the highest confidence level in the second confidence matrix. The classification module is used to, when the category difference is within a preset range, use Calculate the weighted total score for each category associated with the first confidence matrix and the second matrix, and determine the category with the highest weighted total score as the category of the metal-based solid waste sample to be tested, S. i Let Re be the weighted total score for category i, Ri be the first reliability score, and Pe be the second reliability score. i Pi represents the probability confidence level of the tested metal-based solid waste sample belonging to category i in the first confidence matrix. i denoted as the probability confidence level of the metal-based solid waste sample to be tested belonging to category i in the second confidence matrix.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images