Classification method and device for regenerated materials, medium and electronic equipment

Abstract

The embodiment of the invention discloses a classification method and device for regenerated materials, a medium and electronic equipment. A specific embodiment of the method comprises the following steps: acquiring a multi-modal image set for a regenerated material by using a multi-modal sensor; performing pollution monitoring and positioning on the image of the first target mode in the image set to obtain a pollution rough area sub-image set; performing fine segmentation processing on the polluted rough area sub-image set to obtain a global pollution boundary mask; determining a pollutant area proportion according to the global pollution boundary mask; inputting the multi-modal fusion feature information corresponding to the image set into a pre-trained classification model to generate prediction feature information; according to the pollutant area proportion and the category sensitivity parameter set, adjusting the prediction feature information to obtain adjusted feature information; and inputting the adjusted feature information into a classification layer to obtain an impurity classification result. According to the embodiment, the classification accuracy can be improved, manual intervention is reduced, and the waste recovery treatment efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to a classification method, apparatus, medium, and electronic device for recycled materials. Background Art

[0002] In the process of waste classification and recycling, accurate classification is the key to improving recycling efficiency, reducing pollution, and increasing resource reuse rate.

[0003] However, existing waste classification systems mostly rely on manual or traditional image recognition technologies, and often have the following technical problems: low classification accuracy, large errors, and inability to adapt to factors such as waste surface contamination and morphological changes. Therefore, developing an intelligent classification method that can adapt to environmental changes such as waste morphological changes and surface contamination has become a key technology to improve waste treatment efficiency. Summary of the Invention

[0004] The content part of the present disclosure is used to briefly introduce concepts, which will be described in detail in the subsequent detailed implementation part. The content part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Some embodiments of the present disclosure provide a classification method, apparatus, medium, and electronic device for recycled materials to solve one or more of the technical problems mentioned in the above background art part.

[0006] In a first aspect, some embodiments of the present disclosure provide a classification method for recycled materials, including: using a multi-modal sensor to obtain an image set of recycled materials in multiple modalities; performing pollution monitoring and localization on images of a first target modality in the above image set to obtain a sub-image set of roughly polluted regions; performing fine segmentation processing on the above sub-image set of roughly polluted regions to obtain a globally polluted boundary mask after fine segmentation; determining the proportion of the pollutant area according to the above globally polluted boundary mask; inputting the multi-modal fusion feature information corresponding to the above image set into a pre-trained classification model to generate prediction feature information, where the above prediction feature information is a vector composed of impurity categories and impurity proportions, and where the above prediction feature information is the output result of the layer above the classification layer in the above classification model; adaptively adjusting the impurity proportion corresponding to each impurity category in the above prediction feature information according to the above pollutant area proportion and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information; inputting the above adjusted feature information into the above classification layer to obtain an impurity classification result.

[0007] In a second aspect, some embodiments of the present disclosure provide a classification device for recycled materials, including: an acquisition unit configured to acquire an image set of recycled materials in multiple modalities by using a multi-modal sensor; a rough segmentation unit configured to perform pollution monitoring and localization on the images of the first target modality in the above image set to obtain a sub-image set of rough pollution regions; a fine segmentation unit configured to perform fine segmentation processing on the above sub-image set of rough pollution regions to obtain a globally polluted boundary mask after fine segmentation; a determination unit configured to determine the proportion of the pollutant area according to the above globally polluted boundary mask; a generation unit configured to input the multi-modal fusion feature information corresponding to the above image set into a pre-trained classification model to generate prediction feature information, where the above prediction feature information is a vector composed of impurity categories and impurity proportions, and the above prediction feature information is the output result of the layer above the classification layer in the above classification model; an adjustment unit configured to adaptively adjust the impurity proportion corresponding to each impurity category in the above prediction feature information according to the above pollutant area proportion and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information; and a result output unit configured to input the above adjusted feature information into the above classification layer to obtain an impurity classification result.

[0008] In a third aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device having stored thereon one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method described in any implementation manner of the first aspect.

[0009] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having stored thereon a computer program, where the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0010] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the classification method for recycled materials in some embodiments of the present disclosure, the classification accuracy can be improved, manual intervention can be reduced, and the waste recycling and treatment efficiency can be enhanced. Specifically, the reasons for inaccurate classification of recycled materials are as follows: Existing waste classification systems mostly rely on manual or traditional image recognition technologies, suffering from low classification accuracy, large errors, and being unable to adapt to factors such as surface contamination and morphological changes of waste materials. Based on this, in some embodiments of the present disclosure, for the classification method of recycled materials, first, a multi-modal sensor is used to obtain an image set of recycled materials in multiple modalities. By obtaining the multi-modal image set and fusing different spectral characteristics, the ability to characterize the surface contamination of waste materials can be enhanced through complementary information. Then, pollution monitoring and localization are performed on the images of the first target modality in the above-mentioned image set to obtain a sub-image set of the roughly polluted area. Processing the first target modality images alone can quickly locate the polluted area, reduce the computational overhead of multi-modal data fusion, and improve the processing efficiency. By specifically analyzing the target modality features, the pollution range is preliminarily screened to reduce the complexity of subsequent refined processing. Then, the sub-image set of the roughly polluted area is subjected to fine segmentation processing to obtain a globally polluted boundary mask after fine segmentation. The fine segmentation processing can eliminate the misjudgment of rough detection, accurately locate the pollutant boundary, and avoid local omission by combining global image information to ensure the integrity of the polluted area contour. Secondly, the proportion of the pollutant area is determined according to the above-mentioned globally polluted boundary mask. It can provide a quantitative basis for dynamically adjusting the classification model, accurately reflect the influence weight of pollutants on the classification result, and enhance the sensitivity of the model to the pollution degree. Thirdly, the multi-modal fusion feature information corresponding to the above-mentioned image set is input into a pre-trained classification model to generate prediction feature information, where the above-mentioned prediction feature information is a vector composed of impurity categories and impurity proportions, and the above-mentioned prediction feature information is the output result of the layer above the classification layer in the above-mentioned classification model. The multi-modal fusion features are input into the classification model to enhance the ability to characterize the material of recycled materials, the thermal radiation and fluorescence characteristics of pollutants. Using the intermediate feature output of the layer before the classification layer not only retains the spatial position information for facilitating the location of the polluted area but also avoids the global computational redundancy of the fully connected layer, significantly improving the model inference efficiency and providing high-resolution feature support for subsequent dynamic confidence adjustment. Thirdly, according to the above-mentioned proportion of the pollutant area and a predefined set of category sensitivity parameters for each impurity category, the impurity proportion corresponding to each impurity category in the above-mentioned prediction feature information is adaptively adjusted to obtain adjusted feature information. This step dynamically corrects the model output deviation by quantifying the influence weight of pollutants on different categories, making the classification result more in line with the actual pollution scenario. Combining the predefined sensitivity parameters to avoid the global interference of a single pollution on the classification decision significantly improves the robustness and classification decision reliability of this method under complex pollution conditions. Finally, the above-mentioned adjusted feature information is input into the above-mentioned classification layer to obtain the impurity classification result.Inputting the adjusted feature information into the classification layer can make the impurity classification result more accurate and better fit the actual pollution scenario. Description of the Drawings

[0011] In combination with the drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the elements and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flowchart of some embodiments of a classification method for recycled materials according to the present disclosure; Figure 2 is a schematic structural diagram of some embodiments of a classification device for recycled materials according to the present disclosure; Figure 3 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Description of the Embodiments

[0013] Embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0014] It should also be noted that for the sake of convenience of description, only parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0015] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0016] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0018] The present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0019] Reference Figure 1 , which shows a flow 100 of some embodiments of a classification method for recycled materials according to the present disclosure. The classification method for recycled materials includes the following steps: Step 101, using a multimodal sensor, obtain an image set of the recycled materials in multiple modalities.

[0020] In some embodiments, the execution subject of the above classification of recycled materials can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the above-listed hardware devices. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0021] In some other embodiments, the above execution subject can use a multimodal sensor to obtain an image set of the recycled materials in multiple modalities. Among them, the above multimodal sensor can be an RGB camera, an infrared camera, and a ultraviolet camera. The recycled materials can be waste resources that can be recycled after recovery, such as waste metals, plastics, papers, etc. The image set in multiple modalities is a set of image data collected synchronously by multiple types of sensors such as RGB cameras, infrared cameras, and ultraviolet cameras.

[0022] Step 102, perform pollution monitoring and localization on the images of the first target modality in the image set to obtain a sub-image set of the roughly polluted area.

[0023] In some embodiments, the above execution subject can perform pollution monitoring and localization on the images of the first target modality in the above image set to obtain a sub-image set of the roughly polluted area. Among them, the above images of the first target modality can be RGB images captured by an RGB camera. The above sub-images of the roughly polluted area can be image segments that are suspected to have polluted areas obtained through preliminary screening. Among them, pollution monitoring and localization can be a process of quickly identifying surface pollutants (such as oil stains and rust) on the waste materials and locating their roughly areas through image processing techniques (such as HSV color space conversion, adaptive threshold segmentation, and morphological optimization).

[0024] In some optional implementation manners of some embodiments, the above execution subject can perform pollution monitoring and localization on the images of the first target modality in the above image set to obtain a sub-image set of the roughly polluted area, which can include the following steps: First step, perform color space conversion on the image in the above-mentioned first target modality to obtain the converted image. Among them, the image in the above-mentioned first target modality can be an RGB image, and the converted image can be an HSV image. In practice, through an open-source library (for example, cv2.cvtColor(src, cv2.COLOR_BGR2HSV) in OpenCV), map the RGB values of each pixel of the RGB image to the HSV space to obtain the HSV image.

[0025] Second step, perform adaptive threshold segmentation on the brightness channel of the above-mentioned converted image to obtain a binary mask image. In practice, first, traverse the V-channel image with a sliding window (for example, 5×5 pixels). Then, determine the local mean ( ) and standard deviation ( ) within each window. Dynamically generate the threshold according to the formula . Secondly, compare each pixel value with the threshold. If the pixel value is greater than the threshold, it is determined as an uncontaminated area and can be set to 0. If the pixel value is less than or equal to the threshold, it is determined as a contaminated area and can be set to 1. Finally, obtain the binary mask image.

[0026] Third step, use the target denoising algorithm to denoise the above-mentioned binary mask image to obtain the denoised binary mask image. In practice, first, perform erosion and then dilation operations on the binary mask image using a structuring element (such as a 3×3 rectangular kernel) to eliminate isolated noise points and burrs and obtain the first denoising result. Then, perform dilation and then erosion operations on the above-mentioned first denoising result to fill small holes and connect adjacent regions to avoid fragmentation of the contaminated area. Finally, obtain the denoised binary mask image.

[0027] Fourth step, use the target algorithm to label the connected regions of the above-mentioned denoised binary mask image to obtain a set of connected region images. In practice, the 8-neighborhood or 4-neighborhood rule can be used to traverse each pixel of the binary image to determine whether it is connected to the surrounding pixels, assign a unique label to each newly discovered connected region, and assign the same label to the pixels in the same region, thereby obtaining a set of connected region images. Among them, each connected region image represents an independent contaminated area, and finally outputs a set containing multiple sub-images, and each sub-image only retains a complete contaminated area and its boundary information. The above-mentioned target algorithm can be the cv2.connectedComponents() function in OpenCV.

[0028] Step 5: Use the bounding box coordinates corresponding to each connected region image in the above-mentioned connected region image set to crop the image in the above-mentioned first target modality to obtain a sub-image set of the roughly contaminated regions. First, for each sub-image in the connected region image set, determine its minimum bounding rectangle. Then, map the bounding box coordinates back to the original RGB image (the first target modality) to ensure that the cropping range accurately corresponds to the contaminated region. Finally, crop the original RGB image according to the bounding box coordinates. For example, the cv2.rectangle() in OpenCV can be used to perform the cropping operation on the original RGB image.

[0029] Step 103: Perform fine segmentation on the sub-image set of the roughly contaminated regions to obtain a globally contaminated boundary mask after fine segmentation.

[0030] In some embodiments, the above-mentioned execution entity can perform fine segmentation on the sub-image set of the roughly contaminated regions to obtain a globally contaminated boundary mask after fine segmentation. Among them, the globally contaminated boundary mask can be a binary image generated by performing pixel-level fine segmentation on the sub-image set of the roughly contaminated regions.

[0031] In some optional implementation manners of some embodiments, the above-mentioned execution entity can perform fine segmentation on the sub-image set of the roughly contaminated regions to obtain a globally contaminated boundary mask after fine segmentation, which may include the following steps: Step 1: Preprocess each sub-image of the sub-image set of the roughly contaminated regions to obtain a preprocessed sub-image set. In practice, normalization processing can be performed on each sub-image of the sub-image set of the roughly contaminated regions. For example, each contaminated sub-image is scaled to a fixed size (e.g., 256×256 pixels).

[0032] Step 2: Input the above-mentioned preprocessed sub-image set into a pre-trained segmentation model to obtain a segmentation probability map. Among them, each pixel value in the above-mentioned segmentation probability map represents the probability that the corresponding image region content is pollutant content. Among them, the above-mentioned pre-trained segmentation model can be a U-Net model or an FCN model, which includes an encoder (extracting multi-scale features) and a decoder (restoring spatial resolution), and fuses context information through skip connections. The encoder uses ResNet-50 to extract features, and the decoder generates a segmentation probability map through upsampling and convolution.

[0033] Step 3: Use a target conversion algorithm to convert the above-mentioned segmentation probability map into a globally binary mask image. Among them, the target conversion algorithm can be an Otsu adaptive threshold segmentation method, or it can be to convert the pollution probability (0~1) of each pixel in the probability map into a binary value (0 or 1) by setting a threshold or an algorithm.

[0034] Step 4: Optimize the above-mentioned global binary mask image to obtain an optimized global binary mask image. For the specific implementation method, refer to the generation of the binary mask image after denoising, which will not be elaborated here.

[0035] Step 5: Perform boundary optimization and spatial alignment on the above-mentioned optimized global binary mask image to obtain a global pollution boundary mask.

[0036] As an example, first, perform dilation and erosion operations on the above-mentioned optimized global binary mask image to extract the edge contour of the pollution area and obtain a global binary image with a continuous edge contour. Then, use the Zhang-Suen thinning algorithm to delete the non-skeleton pixels on the boundary of the global binary image with a continuous edge contour and retain the single-pixel-wide continuous skeleton to obtain a thinned single-pixel boundary. Then, apply median filtering along the boundary direction to smooth the boundary serrations to preserve geometric continuity and obtain a global binary mask image with optimized boundaries. Finally, correct the geometric distortion through feature matching and perspective transformation, and combine bilinear interpolation resampling to completely align the spatial positions of the optimized global binary mask image and the original image to obtain a global pollution boundary mask.

[0037] Step 104: Determine the proportion of the pollutant area according to the global pollution boundary mask.

[0038] In some embodiments, the above-mentioned execution subject may determine the proportion of the pollutant area according to the global pollution boundary mask. Among them, the proportion of the pollutant area may be the proportion of the pixels in the pollution area in the global pollution boundary mask (binary image) to the total image area (for example, the oil stain accounts for 28% of the total image area), which is used to quantify the pollution degree.

[0039] In some optional implementation manners of some embodiments, the above-mentioned execution subject may determine the proportion of the pollutant area according to the global pollution boundary mask, which may include the following steps: First step: Determine the total number of pixels with a fixed mask value in the above-mentioned global pollution boundary mask. In practice, the binary array corresponding to the global pollution boundary mask image may be traversed to count the number of pollution pixels (pixel value is 1).

[0040] Second step: Determine the proportion of the pollutant area according to the total number of pixels with a fixed mask value and the total number of pixels corresponding to the image in the above-mentioned first target modality. In practice, first, the total number of pixels of the image in the first target modality may be determined by the size (height × width) corresponding to the image in the first target modality. Then, determine the proportion of the number of pollution pixels to the total number of pixels of the image in the first target modality. Finally, determine the above proportion as the proportion of the pollutant area.

[0041] Step 105: Input the multi-modal fusion feature information corresponding to the image set into a pre-trained classification model to generate prediction feature information.

[0042] In some embodiments, the above-mentioned execution subject may input the multi-modal fusion feature information corresponding to the above-mentioned image set into a pre-trained classification model to generate prediction feature information. Among them, the above-mentioned prediction feature information is a vector composed of impurity categories and impurity ratios, and the above-mentioned prediction feature information is the output result of the layer above the classification layer in the above-mentioned classification model. Using the intermediate feature output of the layer above the classification layer not only retains the spatial position information for facilitating the positioning of the contaminated area but also avoids the global calculation redundancy of the fully connected layer. The above-mentioned impurity categories may refer to the types of pollutants or impurities that may exist in the recycled materials predicted by the classification model (such as oil stains, rust, plastics, etc.), usually represented in the form of discrete labels. The above-mentioned impurity ratios may refer to the quantitative estimation of the coverage area or volume ratio of various impurities in the recycled materials by the classification model.

[0043] As an example, the above-mentioned pre-trained classification model may be a pre-trained Convolutional Neural Network (CNN), such as ResNet-50 and VGGNet.

[0044] Step 106: According to the pollutant area ratio and a predefined set of category sensitivity parameters for each impurity category, adaptively adjust the impurity ratio corresponding to each impurity category in the prediction feature information to obtain adjusted feature information.

[0045] In some embodiments, the above-mentioned execution subject may adaptively adjust the impurity ratio corresponding to each impurity category in the above-mentioned prediction feature information according to the above-mentioned pollutant area ratio and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information.

[0046] In some optional implementation manners of some embodiments, the above-mentioned execution subject may adaptively adjust the impurity ratio corresponding to each impurity category in the above-mentioned prediction feature information according to the above-mentioned pollutant area ratio and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information, which may include the following steps: The first step: Determine the pollutant impact factor according to the above-mentioned pollutant area ratio.

[0047] In practice, the pollutant impact factor can be obtained through the following formula: .

[0048] Among them, is the pollutant impact factor. is the pollutant area ratio.

[0049] Step 2: Determine the standard deviation of the vector corresponding to the above-mentioned predicted feature information. Among them, the vector corresponding to the above-mentioned predicted feature information can be , where N is the number of categories. First, according to determine the mean of the vector corresponding to the predicted feature information , and then determine the standard deviation according to the mean and the logits vector.

[0050] In practice, the standard deviation of the vector corresponding to the above-mentioned predicted feature information can be obtained through the following formula: .

[0051] Among them, is the standard deviation of the vector corresponding to the predicted feature information, represents the confidence score of the i-th impurity category in the vector corresponding to the predicted feature information. For each , represents the mean of the vector corresponding to the predicted feature information.

[0052] Step 3: Use the above-mentioned category sensitivity parameter set to determine the sensitivity parameter corresponding to the above-mentioned predicted feature information, and obtain the sensitivity parameter vector. Among them, the above-mentioned category sensitivity parameter set can be based on the existing classification labels, and the basic sensitivity parameters of each recycled material for impurity categories are preset. For example, "clean high-grade aluminum sheet" is inversely proportional to the pollutant, that is, the more pollutants, the lower the probability of being "clean high-grade aluminum sheet"; "oiled aluminum casting" is directly proportional to the pollutant, that is, the more pollutants, the greater the probability of being "oiled aluminum casting". For categories inversely proportional to pollutants, set the basic sensitivity parameter to -1. For categories directly proportional to pollutants, set the basic sensitivity parameter to 1. For categories irrelevant to pollutants, set the basic sensitivity parameter to 0. In practice, the parameter α i represents the basic sensitivity parameter of the i-th category, and the sensitivity parameter vector can be [α1, α2, α3]. For example, the sensitivity parameter vector can be [1, -1, 0], corresponding to the sensitivity parameters corresponding to oiled aluminum castings, clean high-grade aluminum sheets, and rusty aluminum plates respectively.

[0053] Step 4: Use the above-mentioned pollution impact factor, the sensitivity parameter corresponding to each predicted feature information in the above-mentioned sensitivity parameter vector, and the above-mentioned standard deviation to adjust the impurity ratio corresponding to the corresponding impurity category, so as to obtain the adjusted impurity ratio corresponding to each impurity category, and obtain the adjusted feature information.

[0054] In practice, the adjusted feature information can be obtained through the following formula: .

[0055] Among them, Represents the confidence score of the i-th impurity category in the vector corresponding to the predicted feature information before adjustment. Represents the basic sensitivity parameter of the i-th impurity category, σ is the standard deviation of the vector corresponding to the predicted feature information. F is the influence factor of the pollutant. Is the confidence score of the i-th impurity category in the vector corresponding to the predicted feature information after adjustment.

[0056] When the above scheme is used to adaptively adjust the predicted feature information corresponding to the above-mentioned recycled materials, if the recycled materials are high-value samples, the following technical problem is often faced: "How to ensure the accuracy of the adjusted feature information corresponding to the recycled materials". The factors leading to the above technical problems are often as follows: Using a linear model to adjust the predicted feature information corresponding to the recycled materials lacks an understanding of the correlation between feature information. Therefore, the following scheme can be decided: First step, construct a category sensitivity matrix according to the predefined set of category sensitivity parameters for each impurity category. Among them, the above category sensitivity matrix can be a parameter matrix quantifying the correlation strength between impurity categories and pollutants, and can be .

[0057] As an example, the vector corresponding to the set of sensitivity parameters can be [1, -1, 0], corresponding respectively to: oiled aluminum castings ( ): The more oil stains, the more likely it is an oiled aluminum casting.

[0058] Clean aluminum ( ): The more oil stains, the less likely it is clean aluminum.

[0059] Rusted aluminum plate ( ): Oil stains have no effect on it.

[0060] The constructed category sensitivity matrix can be: .

[0061] Second step, use the above pollutant area ratio, the above category sensitivity matrix, and the feature vector corresponding to the above predicted feature information to construct a pollution correlation graph. Among them, the above pollution correlation graph is graph structure data, including nodes (impurity categories) and edges (correlations between impurity categories). In practice, the edge weights can be jointly determined by the sensitivity matrix, the pollutant area ratio, and the feature vector corresponding to the predicted feature information.

[0062] In practice, the following formula can be used to determine the edge weights: .

[0063] Among them, Is the pollutant area ratio, Represents the -th predicted score in the vector corresponding to the predicted feature information before adjustment, and the sensitivity matrix represents the quantification parameter of the influence direction (positive / inverse / irrelevant) of the th category (such as "clean aluminum plate", "oiled casting") on the pollutant. The edge weight represents the weight value of the edge connecting the " th category node" and the "pollutant node" in the pollution correlation graph.

[0064] As an example, the node feature can be the feature vector corresponding to the predicted feature information. For example, it can be [0.1, 0.8, 0.3], and the edge weight can be [-0.02, 0.16, 0]. Among them, the negative sign represents inverse proportion, and the stronger the inverse proportion, the larger the value; the positive sign represents direct proportion, and the stronger the direct proportion, the larger the value.

[0065] In the third step, input the above pollution correlation graph into a pre-trained graph embedding feature generation model to generate a graph embedding feature vector. Among them, the above pre-trained graph embedding feature generation model can be a GNN (Graph Neural Network), which can convert discrete impurity categories into continuous low-dimensional vector representations. The above graph embedding feature vector can be a vector that captures the global correlation information of the graph. In practice, the above graph embedding feature vector can be a vector with a shape of (1, D) (D is the embedding dimension. For example, the embedding dimension is 8). For example, the graph embedding feature vector can be [0.2, -0.1, 0.3, 0.0, 0.1, 0.2, -0.1, 0.0].

[0066] In the fourth step, perform a linear transformation on the above graph embedding feature vector to obtain a linearly transformed feature vector. Through the linear transformation, the graph embedding feature vector can be mapped to a task-related feature space (such as a classification confidence adjustment space). In practice, the dimension mapping can be performed through a fully connected layer (linear layer). Among them, the linearly transformed feature vector can be in the shape of (1, M) (M is the target dimension. For example, M = 3 corresponds to 3 types of impurities). In practice, the linearly transformed feature vector can be [0.5, -0.2, 0.3].

[0067] In the fifth step, perform feature screening on the above linearly transformed feature vector through a target activation function to obtain an activated feature vector. Among them, the above activation function can be ReLU activation. Through the non-linear activation function, important features can be screened and noise or irrelevant features can be suppressed. For example, if the linearly transformed feature vector is [0.5, -0.2, 0.3], the ReLU activation function can be used to retain non-negative features, and the activated feature vector can be [0.5, 0.0, 0.3].

[0068] Step 6: Normalize the above-activated feature vectors to obtain a weight adjustment vector. In practice, the Softmax function can be used to normalize the activated features into a probability distribution. For example, the activated feature vector [0.5, 0.0, 0.3] can be normalized to obtain a weight feature vector [0.625, 0, 0.375].

[0069] Step 7: Use the above weight adjustment vector to adaptively adjust the impurity ratio corresponding to each impurity category in the above prediction feature information to obtain adjusted feature information. The adjusted feature information includes the impurity ratio corresponding to each impurity category.

[0070] As an example, the vector corresponding to the feature information before adjustment is [0.1, 0.8, 0.3], the weight feature vector is [0.625, 0, 0.375], and the vector corresponding to the adjusted feature information is [0.0625, 0, 0.1125].

[0071] The above optional steps and their related content are an inventive point of the embodiments of the present disclosure, which solves the above technical problem of "how to ensure the accuracy of the adjusted feature information corresponding to the recycled materials". The factors leading to the above technical problem are often as follows: Using a linear model to adjust the prediction feature information corresponding to the recycled materials lacks an understanding of the correlation between feature information. And this inventive point can significantly improve the accuracy of the adjusted feature information of the recycled materials by constructing a pollution correlation graph.

[0072] Step 107: Input the adjusted feature information into the above classification layer to obtain an impurity classification result.

[0073] In some embodiments, the above execution subject may input the above adjusted feature information into the above classification layer to obtain an impurity classification result.

[0074] In some optional implementation manners of some embodiments, the above execution subject may input the above adjusted feature information into the above classification layer to obtain an impurity classification result, which may include the following steps: First step: For the impurity ratio corresponding to each impurity category in the above adjusted feature information, use the objective function corresponding to the above classification layer to convert it into a probability value to obtain an impurity classification probability vector. The objective function corresponding to the above classification layer may be the Softmax function, which can convert the corresponding impurity ratio into a probability value. The impurity classification probability vector may represent the confidence of each impurity category. For example, the impurity classification probability vector may be [0.6, 0.1, 0.3] (the sum of probabilities is 1), the probability of impurity category 1 is 0.6, the probability of impurity category 2 is 0.1, and the probability of impurity category 3 is 0.3.

[0075] Step 2: Determine the maximum impurity classification probability in the above impurity classification probability vector. For example, in the impurity classification probability vector [0.6, 0.1, 0.3], the maximum impurity classification probability is 0.6.

[0076] Step 3: Map the above maximum impurity classification probability to the corresponding impurity category label to obtain the impurity category with the maximum probability. Among them, the above impurity category label can be a classification label identifying the impurity category in the recycled material. For example, if the impurity category label list is ["oil-bearing aluminum castings", "clean high-grade aluminum sheets", "rusty aluminum plates"], then the impurity category with the maximum probability is "oil-bearing aluminum castings".

[0077] Step 4: Determine the above impurity category with the maximum probability as the impurity classification result. For example, the impurity category classification result is "oil-bearing aluminum castings".

[0078] In some optional implementation manners of some embodiments, after the above execution subject inputs the above adjusted feature information into the above classification layer to obtain the impurity classification result, the following steps may be included: Step 1: Convert the above global pollution boundary mask into a color mask with the same size as the image of the above first target modality. In practice, by traversing the mask pixel values, each pixel can be mapped to a predefined color. For example, pixels with a value of 0 are marked in green, and pixels with a value of 1 are marked in red.

[0079] Step 2: Superimpose the above color mask and the image of the above first target modality through the target image blending technique to obtain the superimposed image. In practice, an image blending formula can be used (for example, result = original image corresponding matrix × (1 - β) + mask × β), where β is the transparency (usually taking values from 0.3 to 0.7).

[0080] Step 3: Generate a classification probability visualization chart according to the above impurity classification probability vector. Among them, in practice, the above visualization chart can be used to draw a bar chart or a pie chart of the impurity classification probability vector by using Matplotlib or Seaborn.

[0081] Step 4: Generate a pollution index visualization component according to the above pollutant area ratio and the above impurity classification result. Among them, the above pollution index visualization component can be a text box or a graphical label. In practice, OpenCV can be used to draw a text box on the image.

[0082] Step 5: Integrate the above superimposed image, the above classification probability visualization chart, and the above pollution index visualization component to obtain a complete classification visualization report. In practice, PIL (Python Imaging Library) or OpenCV can be used to splice the image and the component.

[0083] In the sixth step, based on the above classification visualization report, use the pre-created database to generate recycling suggestions. Among them, the above pre-created database can be a structured data set storing the characteristics of pollutants and corresponding treatment rules in the recycling scenario. In practice, each piece of structured data can include: pollutant type (for example, such as oil stain, rust, mixed pollution, etc.), pollution degree index (for example, such as the proportion of pollutant area, pollution intensity level), material type (for example, such as aluminum, steel, plastic, etc.), and treatment suggestions (for example, cleaning method, repair process, recycling process).

[0084] When training a classification model using traditional methods, it often relies on single-modal images (for example, only using RGB images). When training a classification model with single-modal images, there are often the following technical problems: "Single-modal images cannot comprehensively represent the characteristics of impurities." The reasons for the above technical problems are often as follows: RGB images are vulnerable to interference from light and reflection, infrared images can only reflect thermal radiation characteristics, and ultraviolet images can only capture the fluorescence characteristics of materials. Therefore, the following scheme can be decided to train the classification model: In the first step, obtain a multi-modal image sample set of recycled materials and the corresponding label set. Among them, the above multi-modal image sample set includes: the first target modal sample set, the second target modal sample set, and the third target modal sample set. Among them, the above multi-modal image sample set can refer to a set of multiple types of images obtained by different modal sensors for the same batch of recycled materials. The above corresponding label set can include: types of recycled materials, types of impurities, and coordinate information of the polluted area. In practice, the above first target modal sample set can be an RGB image set, the second target modal sample set can be an infrared image set, and the third target modal sample set can be an ultraviolet image set.

[0085] In the second step, convert the first target modal sample set in the above multi-modal image sample set to obtain a converted sample set. Among them, the above first target modal sample set can be an RGB image sample set, and the converted image can be an HSV image sample set. In practice, through an open-source library (for example, cv2.cvtColor(src, cv2.COLOR_BGR2HSV) of OpenCV) or a custom algorithm, map the RGB values corresponding to each pixel of each RGB image sample to the HSV space to obtain an HSV image sample, and obtain an HSV image sample set.

[0086] In the third step, traverse the brightness channels of each sample in the above converted sample set according to a window of the target size to obtain a window set of the target size. In practice, the window of the target size can be a 5×5 window.

[0087] Step 4: Determine the target adaptive threshold according to the pixel mean and standard deviation in the corresponding pixel value set within each window in the above window set. For the specific steps, refer to the steps for generating the adaptive threshold when generating the binary mask image above, which will not be elaborated here.

[0088] Step 5: Perform adaptive threshold segmentation on the luminance channel of the above-converted image to obtain a binary mask image. In practice, first, a sliding window (e.g., 5×5 pixels) can be used to traverse the V-channel image. Then, determine the local mean ( ), and the standard deviation ( ) within each window. Dynamically generate the threshold according to the formula . Secondly, compare each pixel value with the threshold. If the pixel value is greater than the threshold, it is determined as an uncontaminated area and can be set to 0. If the pixel value is less than or equal to the threshold, it is determined as a contaminated area and can be set to 1. Finally, obtain the binary mask image.

[0089] Step 6: In response to the pixel value in the above pixel value set being greater than the target adaptive threshold, replace the above pixel value with the corresponding pixel mean in the pixel value set to generate a denoised sample, and obtain a denoised sample set. In response to the pixel value in the above pixel value set not being greater than the target adaptive threshold, retain the corresponding pixel value in the above pixel set.

[0090] Step 7: Fuse the second target modality sample set, the third target modality sample set, and the above denoised image set to obtain a fused image set. Among them, the above fused image set is an image set after multi-modal information fusion.

[0091] Step 8: Input the above fused image set into a deep feature extraction model for deep feature extraction to obtain a deep feature information set. Among them, the above deep feature extraction model can be a pre-trained convolutional neural network (CNN), such as ResNet-50, to extract the global pooling layer features of the image. In practice, input each fused image in the fused image set into ResNet-50. After convolution and pooling, a 2048-dimensional feature vector is obtained, and after global average pooling, it is reduced to 512 dimensions. The feature vector corresponding to the deep feature information in the deep feature information set can be a 512-dimensional feature vector.

[0092] In the ninth step, according to the above-mentioned depth feature information set and the corresponding label set, using the target loss function and the target optimizer, the pre-constructed initial classification model is trained through cross-validation to obtain a classification model. Among them, the above-mentioned target loss function can be a cross-entropy loss function. The above-mentioned target optimizer can be an Adam optimizer. The above-mentioned cross-validation method can be a five-fold cross-validation method. For example, 1000 fused images can be evenly divided into 5 parts, with 200 images in each part. Among them, 4 parts (800 fused images) are used to train the classification model, and 1 part (200 fused images) is used to verify the classification model.

[0093] In the tenth step, in response to the impurity classification results corresponding to the above-mentioned classification model being manually marked as wrong results for the cumulative target number of times, the classification model is fine-tuned using the image set corresponding to the wrong results of the above-mentioned target number of times, and the fine-tuned model is determined as the classification model and the classification model before fine-tuning is backed up. In practice, the above-mentioned cumulative target number of times can be 50 times. In practice, when fine-tuning the classification model, the first few convolutional layers can be frozen (retaining general features), and only the last few fully connected layers are trained (adapting to the image set corresponding to the misclassified results).

[0094] In the eleventh step, in response to reaching the preset time interval, the classification model is retrained according to the image set corresponding to the impurity classification result set corresponding to the above-mentioned classification model and the multi-modal image sample set, and the retrained model is determined as the classification model. In practice, the above-mentioned preset time interval can be three months.

[0095] When training the classification model using the above steps, there are often the following technical problems: "difficulty in aligning multi-modal images and low fusion quality". The reasons for the above technical problems are often as follows: Different modal images (such as RGB images, infrared images, ultraviolet images) have geometric misalignments (such as the position offset of the same impurity in RGB and infrared images) and radiation inconsistencies due to differences in sensor perspectives, shooting angles, and radiation characteristics. Direct fusion will lead to chaotic features (such as overlapping misalignments of impurity regions in the fused image). Therefore, the following solutions can be decided: Optionally, the above-mentioned fusion of the second target modal sample set, the third target modal sample set, and the above-mentioned denoised image set to obtain a fused image set may include the following steps: In the first step, use the target feature extraction algorithm to extract feature points from the above-mentioned second target modality sample set and third target modality sample set, obtaining the second feature point set and second descriptor set corresponding to the second target modality sample set, and the third feature point set and third descriptor set corresponding to the third target modality sample set. Among them, the above-mentioned target feature extraction algorithm can be the SIFT algorithm. The second descriptor set can be a set composed of descriptors corresponding to the feature points in the second target modality sample set. The third descriptor set can be a set composed of descriptors corresponding to the feature points in the third target modality sample set. The descriptor can be the "feature fingerprint" of the image, used for subsequent feature matching and multimodal fusion. The above-mentioned descriptor can be a quantitative representation of the local features in the image, capturing the unique attributes (such as texture, shape, color) of the region through a set of numerical values (usually high-dimensional vectors). For example, the vector corresponding to the descriptor can be [0.9, 0.1,..., 0.7].

[0096] In the second step, use the target feature point matching method to match the denoised image set and the second descriptor set, obtaining the second matching pair set. Among them, the matching pairs in the above-mentioned second matching pair set include: the feature point coordinates and descriptors corresponding to the second matching pair set. Among them, the above-mentioned target feature point matching method can be the ORB lightweight feature point matching method. For example, the point (x1, y1) in the denoised image matches the point (x2, y2) in the infrared image.

[0097] In the third step, use the target feature point matching method to match the denoised image set and the third descriptor set, obtaining the third matching pair set. Among them, the matching pairs in the above-mentioned third matching pair set include: the feature point coordinates and descriptors corresponding to the third matching pair set. Among them, the above-mentioned target feature point matching method can be the ORB lightweight feature point matching method. For example, the point (x1, y1) in the denoised image matches the point (x3, y3) in the infrared image.

[0098] In the fourth step, use the target screening algorithm to screen the above-mentioned second matching pair set and the above-mentioned third matching pair set, obtaining the screened second matching pair set and the screened third matching pair set. Among them, the above-mentioned target screening algorithm can be the RANSAC algorithm. By using the RANSAC algorithm to estimate the homography matrix, the mismatched point pairs that do not conform to the matrix transformation are removed. For example, in the matching pairs between the denoised image and the infrared image, most point pairs satisfy the homography transformation (i.e., geometric consistency), and a small number of point pairs that do not satisfy are removed.

[0099] Step 5: Perform perspective transformation on the second matched pair set after the above screening, the third matched pair set after the above screening, and the denoised image set to obtain an aligned second target modality sample set and an aligned third target modality sample set. In practice, a perspective transformation matrix (such as a homography matrix H) can be estimated based on the matched pairs, and the second target modality sample images and the third target modality sample images are transformed to the coordinate system corresponding to the first target modality image through H.

[0100] Step 6: Perform radiometric calibration on the aligned second target modality sample set and the aligned third target modality sample set to obtain a calibrated second target modality sample set and a calibrated third target modality sample set. In practice, through a radiometric transfer function (RTF) or histogram matching, the pixel values of the second modality images and the third target modality images are mapped to the radiometric range of the first target modality image. For example, the pixel value range [0, 255] (from low temperature to high temperature) of an infrared image can be calibrated to [0, 255] (from dark to bright) of an RGB image, making the two comparable on the same scale.

[0101] Step 7: Fuse the calibrated second target modality sample set, the calibrated third target modality sample set, and the denoised image set to obtain a fused image set. In practice, weights can be assigned to each modality (for example, the weight of the RGB image is 0.4, the weight of the infrared image is 0.3, and the weight of the ultraviolet image is 0.3). Then, the grayscale images of each target modality are used as different channels to generate a multi-channel fused image.

[0102] The above optional steps and their related content are an inventive point of the embodiments of the present disclosure, which solve the above technical problems of "single-modal images cannot comprehensively represent impurity characteristics" and "difficult alignment of multi-modal images and low fusion quality". The factors leading to the above technical problems are often as follows: RGB images are vulnerable to light and reflection interference, infrared images can only reflect thermal radiation characteristics, and ultraviolet images can only capture material fluorescence characteristics. Due to differences in sensor perspective, shooting angle, and radiation characteristics among different modality images, there are geometric misalignments and radiometric inconsistencies, and direct fusion will result in chaotic features. However, this inventive point trains a classification model using multi-modal images. During training, the multi-modal images are precisely aligned, which can significantly improve the accuracy and robustness of the classification model.

[0103] The above-described various embodiments of the present disclosure have the following beneficial effects: Through the classification method for recycled materials in some embodiments of the present disclosure, the classification accuracy can be improved, manual intervention can be reduced, and the efficiency of waste recycling and treatment can be enhanced. Specifically, the reasons for inaccurate classification of recycled materials are as follows: Existing waste classification systems mostly rely on manual or traditional image recognition technologies, suffering from low classification accuracy, large errors, and being unable to adapt to factors such as surface contamination and morphological changes of waste materials. Based on this, in some embodiments of the present disclosure, the classification method for recycled materials first uses a multi-modal sensor to obtain an image set of recycled materials in multiple modalities. By obtaining the multi-modal image set, different spectral characteristics can be fused, and the ability to characterize surface contamination of waste materials can be enhanced through complementary information. Then, pollution monitoring and localization are performed on the images of the first target modality in the above image set to obtain a sub-image set of roughly polluted regions. Processing the first-modal images alone can quickly locate the polluted regions, reduce the computational overhead of multi-modal data fusion, and improve the processing efficiency; by specifically analyzing the characteristics of the target modality, the pollution range is preliminarily screened, reducing the complexity of subsequent refined processing. Then, the sub-image set of roughly polluted regions is subjected to refined segmentation processing to obtain a globally polluted boundary mask after refined segmentation. The refined segmentation processing can eliminate misjudgments in rough detection, accurately locate the boundaries of pollutants; combining global image information avoids local omissions and ensures the integrity of the contours of polluted regions. Secondly, the proportion of the area of pollutants is determined according to the above globally polluted boundary mask. This can provide a quantitative basis for dynamically adjusting the classification model, accurately reflecting the influence weight of pollutants on the classification result, and enhancing the sensitivity of the model to the degree of pollution. Thirdly, the multi-modal fusion feature information corresponding to the above image set is input into a pre-trained classification model to generate predicted feature information, where the above predicted feature information is a vector composed of impurity categories and impurity proportions, and the above predicted feature information is the output result of the layer above the classification layer in the above classification model. The multi-modal fusion features are input into the classification model to enhance the ability to characterize the material of waste materials, the thermal radiation and fluorescence characteristics of pollutants. Using the intermediate feature output of the layer before the classification layer not only retains the spatial position information for facilitating the localization of polluted regions but also avoids the global computational redundancy of the fully connected layer, significantly improving the model inference efficiency and providing high-resolution feature support for subsequent dynamic confidence adjustment. Thirdly, according to the proportion of the area of pollutants and a predefined set of category sensitivity parameters for each impurity category, the impurity proportion corresponding to each impurity category in the above predicted feature information is adaptively adjusted to obtain adjusted feature information. This step dynamically corrects the output deviation of the model by quantifying the influence weight of pollutants on different categories, making the classification result more suitable for the actual pollution scenario. Combining the predefined sensitivity parameters to avoid the global interference of a single pollution on classification decisions significantly enhances the robustness and classification decision reliability of this method under complex pollution conditions. Finally, the above adjusted feature information is input into the above classification layer to obtain the impurity classification result.Input the adjusted feature information into the classification layer, which can accurately fuse the weights corresponding to the pollution adjustment feature information, making the classification result more in line with the actual pollution scenario.

[0104] Further refer to Figure 2 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a classification device for recycled materials, and these device embodiments correspond to Figure 1 the method embodiments shown, and the classification device for recycled materials can be specifically applied to various electronic devices.

[0105] As Figure 2 shown, a classification device 200 for recycled materials includes: an acquisition unit 201, a rough segmentation unit 202, a fine segmentation unit 203, a determination unit 204, a generation unit 205, an adjustment unit 206, and a result output unit 207. Among them, the acquisition unit 201 is configured to: use a multi-modal sensor to acquire an image set of recycled materials in multiple modalities. The rough segmentation unit 202 is configured to: perform pollution monitoring and localization on the images of the first target modality in the above image set to obtain a sub-image set of roughly polluted areas. The fine segmentation unit 203 is configured to: perform fine segmentation processing on the above sub-image set of roughly polluted areas to obtain a globally polluted boundary mask after fine segmentation. The determination unit 204 is configured to: determine the proportion of the pollutant area according to the above globally polluted boundary mask. The generation unit 205 is configured to: input the multi-modal fusion feature information corresponding to the above image set into a pre-trained classification model to generate predicted feature information, where the above predicted feature information is a vector composed of impurity categories and impurity proportions, and the above predicted feature information is the output result of the layer above the classification layer in the above classification model. The adjustment unit 206 is configured to: adaptively adjust the impurity proportion corresponding to each impurity category in the above predicted feature information according to the above pollutant area proportion and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information. The result output unit 207 is configured to: input the above adjusted feature information into the above classification layer to obtain an impurity classification result.

[0106] It can be understood that the units described in the classification device 200 for recycled materials correspond to the respective steps in the method described with reference to Figure 1 . Therefore, the operations, features, and beneficial effects described above for the method also apply to the classification device 200 for recycled materials and the units included therein, and will not be repeated here.

[0107] Next, refer to Figure 3 , which shows a schematic structural diagram of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3The electronic device shown is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0108] As Figure 3 shown, the electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage device 308 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0109] Generally, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 the electronic device 300 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had. Figure 3 Each block shown in

[0110] particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such some embodiments, the computer program may be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the method of some embodiments of the present disclosure are performed.

[0111] It should be noted that, in some embodiments of the present disclosure, the above-mentioned computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0113] The above computer-readable medium may be included in the above electronic device; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device is caused to: obtain an image set of the recycled material in multiple modalities by using a multimodal sensor; perform pollution monitoring and localization on the images of the first target modality in the above image set to obtain a sub-image set of the roughly polluted area; perform fine segmentation processing on the sub-image set of the roughly polluted area to obtain a globally polluted boundary mask after fine segmentation; determine the proportion of the pollutant area according to the above globally polluted boundary mask; input the multimodal fusion feature information corresponding to the above image set into a pre-trained classification model to generate prediction feature information, where the above prediction feature information is a vector composed of an impurity category and an impurity ratio, and the above prediction feature information is the output result of the layer above the classification layer in the above classification model; adaptively adjust the impurity ratio corresponding to each impurity category in the above prediction feature information according to the above pollutant area proportion and a predefined category sensitivity parameter set for each impurity category to obtain adjusted feature information; input the above adjusted feature information into the above classification layer to obtain an impurity classification result.

[0114] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++; and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0116] The units described in some embodiments of the present disclosure can be implemented in software or in hardware. The described units can also be provided in a processor. For example, it can be described as: a processor includes an acquisition unit, a rough segmentation unit, a fine segmentation unit, a determination unit, a generation unit, an adjustment unit, and a result output unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases. For example, the acquisition unit can also be described as "the unit that uses a multi-modal sensor to acquire an image set of a recycled material under multi-modal conditions".

[0117] The functions described above can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0118] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, technical solutions formed by mutually replacing the above features with technical features (but not limited to) having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A classification method for recycled materials, comprising: Using a multi-modal sensor to obtain an image set of the recycled materials in multiple modalities; Performing pollution monitoring and localization on the images of the first target modality in the image set to obtain a sub-image set of roughly polluted areas; Performing fine segmentation processing on the sub-image set of roughly polluted areas to obtain a globally polluted boundary mask after fine segmentation; Determining the proportion of the pollutant area according to the globally polluted boundary mask; Inputting the multi-modal fusion feature information corresponding to the image set into a pre-trained classification model to generate predicted feature information, where the predicted feature information is a vector composed of impurity categories and impurity proportions, and the predicted feature information is the output result of the layer above the classification layer in the classification model; According to the proportion of the pollutant area and a predefined set of category sensitivity parameters for each impurity category, adaptively adjusting the impurity proportion corresponding to each impurity category in the predicted feature information to obtain adjusted feature information; Inputting the adjusted feature information into the classification layer to obtain an impurity classification result.

2. The method according to claim 1, wherein The performing pollution monitoring and localization on the images of the first target modality in the image set to obtain a sub-image set of roughly polluted areas includes: Performing color space conversion on the images in the first target modality to obtain converted images; Performing adaptive threshold segmentation on the luminance channel of the converted images to obtain binary mask images; Performing denoising processing on the binary mask images using a target denoising algorithm to obtain denoised binary mask images; Using a target algorithm to mark the connected regions of the denoised binary mask images to obtain a set of connected region images; Using the bounding box coordinates corresponding to each connected region image in the set of connected region images to crop the images in the first target modality to obtain a sub-image set of roughly polluted areas.

3. The method according to claim 1, wherein, The performing fine segmentation processing on the sub-image set of roughly polluted areas to obtain a globally polluted boundary mask after fine segmentation includes: Performing preprocessing on each sub-image of the sub-image set of roughly polluted areas to obtain a preprocessed sub-image set; Inputting the preprocessed sub-image set into a pre-trained segmentation model to obtain a segmentation probability map, where each pixel value in the segmentation probability map represents the probability that the corresponding image region content is pollutant content; Using a target conversion algorithm to convert the segmentation probability map into a global binary mask image; Optimizing the global binary mask image to obtain an optimized global binary mask image; Performing boundary optimization and spatial alignment on the optimized global binary mask image to obtain a globally polluted boundary mask.

4. The method according to claim 1, wherein The determining the proportion of the pollutant area according to the globally polluted boundary mask includes: Determining the total number of pixels with a fixed mask value in the globally polluted boundary mask; Determining the proportion of the pollutant area according to the total number of pixels with a fixed mask value and the total number of pixels corresponding to the images in the first target modality.

5. The method according to claim 1, wherein Adaptively adjusting the impurity proportion corresponding to each impurity category in the predicted feature information according to the pollutant area ratio and a predefined set of category sensitivity parameters for each impurity category to obtain adjusted feature information, including: Determining a pollution impact factor according to the pollutant area ratio; Determining the standard deviation of the vector corresponding to the predicted feature information; Using the set of category sensitivity parameters to determine the sensitivity parameters corresponding to the predicted feature information to obtain a sensitivity parameter vector; Adjusting the impurity proportion corresponding to the corresponding impurity category by using the pollution impact factor, the sensitivity parameter corresponding to each predicted feature information in the sensitivity parameter vector, and the standard deviation to obtain the adjusted impurity proportion corresponding to each impurity category, so as to obtain the adjusted feature information.

6. The method according to claim 1, wherein Inputting the adjusted feature information into the classification layer to obtain an impurity classification result, including: Converting the impurity proportion corresponding to each impurity category in the adjusted feature information into a probability value by using the objective function corresponding to the classification layer to obtain an impurity classification probability vector; Determining the maximum impurity classification probability in the impurity classification probability vector; Mapping the maximum impurity classification probability to the corresponding impurity category label to obtain the maximum probability impurity category; Determining the maximum probability impurity category as the impurity classification result.

7. The method according to claim 6, wherein After inputting the adjusted feature information into the classification layer to obtain the impurity classification result, the method further includes: Converting the global pollution boundary mask into a color mask with the same size as the image of the first target modality; Overlaying the color mask and the image of the first target modality through a target image blending technique to obtain an overlaid image; Generating a classification probability visualization chart according to the impurity classification probability vector; Generating a pollution index visualization component according to the pollutant area ratio and the impurity classification result; Integrating the overlaid image, the classification probability visualization chart, and the pollution index visualization component to obtain a complete classification visualization report; Generating a recycling suggestion according to the classification visualization report by using a pre-created database.

8. A classification device for recycled materials, including: An acquisition unit configured to acquire an image set of recycled materials in multiple modalities by using a multi-modal sensor; A rough segmentation unit configured to perform pollution monitoring and positioning on the image of the first target modality in the image set to obtain a sub-image set of a roughly polluted area; A fine segmentation unit configured to perform fine segmentation processing on the sub-image set of the roughly polluted area to obtain a globally polluted boundary mask after fine segmentation; A determination unit configured to determine the pollutant area ratio according to the global pollution boundary mask; A generation unit configured to input the multi-modal fusion feature information corresponding to the image set into a pre-trained classification model to generate predicted feature information, where the predicted feature information is a vector composed of impurity categories and impurity proportions, and the predicted feature information is the output result of the layer above the classification layer in the classification model. An adjustment unit, configured to adaptively adjust the impurity ratio corresponding to each impurity category in the predicted feature information according to the pollutant area ratio and a predefined set of category sensitivity parameters for each impurity category, to obtain adjusted feature information; A result output unit, configured to input the adjusted feature information into the classification layer to obtain an impurity classification result.

9. An electronic device, comprising: One or more processors; A storage device having stored thereon one or more programs, When the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the method according to any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, The program, when executed by the processor, implements the method according to any one of claims 1-7.

Images