A solid waste intelligent storage and management system

By optimizing the deep convolutional pulse neural network and the border collie optimization algorithm, combined with semi-supervised fusion edge detection and joint cross-modal learning methods, the problems of low classification efficiency and insufficient recognition accuracy of solid waste are solved, and efficient classification is achieved Identification and storage management.

CN118840739BActive Publication Date: 2024-11-26TIANJIN HONGLING INTELLIGENT IOT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411317239.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-26
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The prior art is slow in the recycling, processing and storage of solid waste, has low processing efficiency, and has low identification accuracy for various types of waste when facing a large number of stacked waste, which affects subsequent targeted storage.

Method used

Feature extraction and classification are adopted with an optimized deep convolutional pulse neural network, combined with the Border Collie optimization algorithm and Archimedes algorithm to optimize neural network parameters, using semi-supervised fusion edge detection automatic labeling model and migratory joint cross-modal Learning method, generate pseudo-features for joint learning of visual and text, and realize the optimization of multi-label classification annotation.

Benefits of technology

It improves the classification efficiency and accuracy of solid waste, can efficiently identify large amounts of stacked waste, facilitates subsequent targeted storage, and reduces network computing load and storage needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118840739B_ABST
    Figure CN118840739B_ABST
Patent Text Reader

Abstract

The invention relates to the field of intelligent storage, and in particular to an intelligent storage management system for solid waste, comprising a crawler conveying module, an image acquisition module, an automatic labeling and classification module, a classification sorting and storage module and a central control module; the invention adopts an optimized deep convolutional pulse neural network to dynamically capture the association between different parts of solid waste, efficiently process large-scale solid waste images, and realize accurate classification; the invention adopts an automatic labeling model based on semi-supervised fusion edge detection to perform more accurate and high-quality classification and identification for a large amount of stacked waste, so as to facilitate subsequent targeted storage; the invention adopts a transferable joint cross-modal learning method, utilizes automatic labeling to generate pseudo features of solid waste, performs multi-modal joint learning according to real features and pseudo features of solid waste, enhances the model expression ability through adapter fine-tuning, and finally utilizes comparative learning to realize the tuning of multi-label classification labeling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of intelligent storage, and in particular relates to an intelligent storage management system for solid waste. Background Art

[0002] Solid waste storage is the process of storing and stacking the generated solid waste. It is an important part of waste management. Reasonable storage measures can be taken to ensure that the waste does not cause harm to the environment and human health during temporary or permanent storage. Solid waste storage can prevent waste from polluting the environment. Proper storage can also facilitate the recycling and reuse of these resources, reducing the demand for natural resources. However, at this stage, when recycling and storing solid waste, the classification is slow and the processing efficiency is low. When faced with a large amount of stacked waste, the accuracy of identifying various types of waste is low, affecting the subsequent targeted storage. Summary of the invention

[0003] In view of the above situation, in order to overcome the defects of the prior art, the prior art has the problem of slow classification and low processing efficiency when recycling and storing solid waste. The present invention adopts an optimized deep convolutional pulse neural network to perform more efficient feature extraction, dynamically capture the association between different parts of solid waste, reduce the load and storage requirements of network computing, and more efficiently process large-scale solid waste images. At the same time, the Border Collie optimization algorithm and the Archimedes algorithm are used to optimize the neural network parameters to make the classification efficiency and accuracy higher; in view of the problem that the prior art has low recognition accuracy for various types of waste when facing a large amount of stacked waste, which affects the subsequent targeted storage, the present invention An automatic labeling model based on semi-supervised fusion edge detection is creatively adopted, which can perform more accurate classification and identification of stacked waste. At the same time, based on the high classification accuracy of the optimized deep convolutional pulse neural network, this automatic labeling method can also efficiently and accurately classify and identify solid waste when facing a large number of stacks, which is convenient for subsequent targeted storage; the present invention creatively adopts a transferable joint cross-modal learning method, uses automatic labeling to generate pseudo-features of solid waste, conducts joint learning of vision and text based on the real features and pseudo-features of solid waste, and enhances the model expression ability through adapter fine-tuning, and finally uses comparative learning to achieve the tuning of multi-label classification labeling.

[0004] The technical solution adopted by the present invention is as follows: a solid waste intelligent storage and management system, including a crawler conveying module, an image acquisition module, an automatic marking and classification module, a classification sorting and storage module and a central control module;

[0005] The crawler conveying module uses crawlers to convey the solid waste and finally conveys it to the classification, sorting and storage module;

[0006] The image acquisition module uses a linear color camera to acquire an RGB image of the solid waste on the crawler, and uses a depth image camera to acquire a depth image of the solid waste on the crawler, and transmits the RGB image and the depth image to the automatic labeling and classification module, wherein the linear color camera and the depth image camera are placed above the crawler;

[0007] The automatic labeling and classification module classifies solid wastes according to RGB images and depth images using an optimized deep convolutional spike neural network, and classifies and labels solid wastes using a semi-supervised fusion edge detection automatic labeling model, and uses a transferable joint cross-modal learning method to tune the parameters of the automatic labeling model to obtain optimized classification labeling results, and transmits the classification labeling results to the classification sorting and storage module;

[0008] The classification, sorting and storage module classifies and stores the solid waste according to the classification and labeling results;

[0009] The central control module directly controls the crawler conveying module, the image acquisition module, the automatic labeling and classification module, and the classification, sorting and storage module.

[0010] Furthermore, in the automatic labeling and classification module, an optimized deep convolutional spike neural network is used to classify solid waste, which specifically includes the following steps:

[0011] Step S1: data preprocessing, preprocessing the RGB image of solid waste to obtain a preprocessed RGB image;

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: image loading, loading the RGB image of the solid waste into the computer;

[0014] Step S12: image size adjustment, adjusting the size of the RGB image of the solid waste to a fixed and uniform size;

[0015] Step S13: pixel normalization, normalizing all pixels in the RGB image to obtain a normalized RGB image;

[0016] Step S14: image enhancement, using a random rotation method to process the normalized RGB image to obtain a preprocessed RGB image;

[0017] Step S2: data division, dividing the preprocessed RGB images into a training set and a test set;

[0018] Step S3: Initialize the deep convolutional spike neural network, initialize the spike encoding layer, flattening layer, convolution layer, maximum pooling layer, dropout layer and fully connected layer, set the maximum pooling layer parameters, pulse conditions, neuron parameters and synaptic weights;

[0019] Step S4: Use a convolutional layer to extract features from the training set to obtain a feature map, use a maximum pooling layer to downsample the feature map to obtain n non-overlapping rectangular area feature maps, and flatten the n rectangular area feature maps to obtain n one-dimensional feature vectors and the modulus of the n one-dimensional feature vectors;

[0020] Step S5: converting the n one-dimensional feature vectors into n current amplitudes according to the magnitude of the vector modulus;

[0021] Step S6: Arrange the n current amplitudes to obtain a pulse sequence diagram, and classify the pulse sequence diagram using a pulse neural network according to neuron parameters and synaptic weights to obtain a preliminary classification result of the solid waste;

[0022] Step S7: Calculate the accuracy of the preliminary classification result, and optimize the maximum pooling layer parameters using the Border Collie optimization algorithm according to the accuracy of the preliminary classification result, and optimize the neuron parameters and synaptic weights using the Archimedes algorithm to obtain an optimized deep convolutional spike neural network and preliminary classification results;

[0023] Further, step S7 specifically includes the following steps:

[0024] Step S71: define the objective function and determine the accuracy of the classification result as the target;

[0025] Step S72: Initialize the population individuals, parameters and iteration times of the Border Collie optimization algorithm, where each individual represents a set of maximum pooling layer parameters, and initialize the parameters and fitness threshold of the Archimedean algorithm, where the parameters include position density, volume and acceleration;

[0026] Step S73: performing optimization operation on the population individuals according to the border collie optimization algorithm to obtain the preliminary optimal individual;

[0027] Step S74: optimizing the neuron parameters and synaptic weights according to the Archimedean algorithm to obtain preliminarily optimized neuron parameters and synaptic weights;

[0028] Step S75: repeating steps S73 and S74 until the number of iterations and the fitness threshold are reached, and obtaining the optimal maximum pooling layer parameters and optimized neuron parameters and synaptic weights;

[0029] Step S76: Input the optimal maximum pooling layer parameters and the optimized neuron parameters and synaptic weights into the deep convolutional spiking neural network to obtain the optimized deep convolutional spiking neural network and preliminary classification results.

[0030] Furthermore, in the automatic labeling and classification module, a semi-supervised automatic labeling model fused with edge detection is used to classify and label the solid waste, which specifically includes the following steps:

[0031] Step Q1: Segment the depth image, and perform dilation, erosion and contour search operations to obtain a segmented depth map and an outer contour map, and grayscale the RGB image to obtain a grayscale image;

[0032] Step Q2: Use median filtering to filter the grayscale image to obtain a median filter image, multiply the median filter image with the segmentation depth map to obtain a filtered segmentation image, and multiply the depth image with the segmentation depth map to obtain a fused depth map;

[0033] Step Q3: Perform edge detection on the filtered segmentation image to obtain an RGB edge image, and perform edge detection on the fused depth image to obtain a depth edge image;

[0034] Step Q4: Use the outer contour map to crop the RGB edge map and the depth edge map and perform fixed threshold segmentation to obtain an enhanced RGB contour map and an enhanced depth contour map;

[0035] Step Q5: Simplify the enhanced RGB contour map and the enhanced depth contour map using a skeletonization algorithm to obtain a simplified RGB contour map and a simplified depth contour map;

[0036] Step Q6: Fuse the simplified RGB contour map with the simplified depth contour map to obtain a fused RGB image, and use the optimized deep convolutional spike neural network to annotate the fused RGB image to obtain a preliminary classification and annotation result.

[0037] Furthermore, in the automatic labeling and classification module, a transferable joint cross-modal learning method is used to tune the parameters of the automatic labeling model to obtain optimized classification labeling results, which specifically includes the following steps:

[0038] Step K1: Pseudo feature extraction, specifically including the following steps:

[0039] Step K11: using a text encoder to perform text feature encoding on the classification labels of each classification of solid waste;

[0040] Step K12: using a visual encoder to perform visual feature encoding on the pre-processed RGB images of each type of classified solid waste;

[0041] Step K13: Pseudo-feature design: the text feature code and visual feature code of each classification of solid waste are fused to obtain the pseudo-feature of each classification of solid waste, which is defined as:

[0042] P=[p1 , p2 , p3 , ... , pi , ... , pn ] ;

[0043] In the formula, Representative Pseudo features of classes, is the number of categories, is a pseudo feature set;

[0044] Step K2: embedding feature extraction and adapter fine-tuning, specifically including the following steps:

[0045] Step K21: Use the visual encoder to encode the pseudo features into global visual features to obtain a global visual embedding;

[0046] Step K22: Use the text encoder to encode the pseudo features into global text features to obtain global text embedding;

[0047] Step K23: real category labeling, collecting solid waste images with real classification labels in the network to train the visual neural network, obtaining the real classification categories of all solid wastes and the trained visual neural network, using the trained visual neural network to identify the pre-processed RGB images of solid waste, and obtaining the real classification results of all solid wastes;

[0048] Step K24: Use a text encoder to encode the real text features of all the real classification results of solid waste to obtain real text embedding;

[0049] Step K25: Adapter fine-tuning. Use the adapter module to perform adapter fine-tuning on the global visual embedding, global text embedding, and real text embedding. The formula used is as follows:

[0050] ;

[0051] ;

[0052] ;

[0053] In the formula, and Represents the adapter fine-tuning functions for visual and text modes respectively, is the preset weight, , and Represent global visual embedding, global text embedding and real text embedding respectively, , and They represent the global visual embedding, global text embedding, and real text embedding after fine-tuning of the adapter respectively;

[0054] Step K3: Multi-mode similarity calculation, specifically including the following steps:

[0055] Step K31: Calculate the pseudo-similarity of the same modality. The formula used is as follows:

[0056] ;

[0057] In the formula, and Represents the first Class global text embedding and Real-like text embeddings, Represents similarity calculation, Representative Homomodal pseudo-similarity of classes;

[0058] Step K32: Calculate cross-modal pseudo similarity, using the following formula:

[0059] ;

[0060] In the formula, and Represents the first Class global text embedding and Class global visual embedding, Represents similarity calculation, Representative Cross-modal pseudo-similarity of classes;

[0061] Step K33: Calculate the cross-modal true similarity. The formula used is as follows:

[0062] ;

[0063] In the formula, and Represents the first Class-real text embedding and Class global visual embedding, Represents similarity calculation, Representative Cross-modal true similarity of classes;

[0064] Step K4: Pseudo contrast loss calculation, the formula used is as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] In the formula, To preset adjustment parameters, and Represent the cross-modal pseudo similarity and the cross-modal true similarity respectively, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the pseudo-ratio loss;

[0069] Step K5: Calculation of true contrast loss, the formula used is as follows:

[0070] ;

[0071] ;

[0072] In the formula, To preset adjustment parameters, and Represent the pseudo-similarity of the same modality and the true similarity of cross-modality, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the true contrast loss;

[0073] Step K6: Calculate the total training loss by adding the true contrast loss to the pseudo contrast loss to get the total loss;

[0074] Step K7: Automatic labeling model tuning: According to the total loss, a semi-supervised fusion edge detection automatic labeling model is parameter-tuned, and the optimized automatic labeling model is used to automatically label the solid waste to obtain an optimized classification labeling result.

[0075] The beneficial results achieved by the present invention using the above scheme are as follows:

[0076] (1) In view of the problem of slow classification and low processing efficiency in the existing technology when recycling and storing solid waste, the present invention adopts an optimized deep convolutional spike neural network to perform more efficient feature extraction, dynamically capture the association between different parts of solid waste, reduce the network computing load and storage requirements, and more efficiently process large-scale solid waste images. At the same time, the Border Collie optimization algorithm and Archimedes algorithm are used to optimize the neural network parameters to achieve higher classification efficiency and accuracy.

[0077] (2) In view of the problem that the existing technology has low accuracy in identifying various types of waste when facing a large amount of stacked waste, which affects the subsequent targeted storage, the present invention adopts an automatic labeling model based on semi-supervised fusion edge detection, which can perform more accurate classification and identification of stacked waste. At the same time, based on the high classification accuracy of the optimized deep convolutional pulse neural network, the automatic labeling method can also efficiently and accurately classify and identify solid waste when facing a large amount of stacked waste, which is convenient for subsequent targeted storage;

[0078] (3) The present invention creatively adopts a transferable joint cross-modal learning method, which uses automatic annotation to generate pseudo-features of solid waste, conducts joint visual and textual learning based on the real features and pseudo-features of solid waste, and enhances the model's expression ability through adapter fine-tuning. Finally, contrastive learning is used to achieve the optimization of multi-label classification annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A module diagram of a solid waste intelligent storage and management system provided by the present invention;

[0080] Figure 2 A flowchart of a semi-supervised fusion edge detection automatic labeling model;

[0081] Figure 3 Schematic diagram of the process of a transferable joint cross-modal learning method.

[0082] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0084] Example 1, see Figure 1 , the present invention provides a solid waste intelligent storage and management system, including a crawler conveying module, an image acquisition module, an automatic labeling and classification module and a classification, sorting and storage module;

[0085] The crawler conveying module uses crawlers to convey the solid waste and finally conveys it to the classification, sorting and storage module;

[0086] The image acquisition module uses a linear color camera to acquire an RGB image of the solid waste on the crawler, and uses a depth image camera to acquire a depth image of the solid waste on the crawler, and transmits the RGB image and the depth image to the automatic labeling and classification module, wherein the linear color camera and the depth image camera are placed above the crawler;

[0087] The automatic labeling and classification module classifies the solid waste according to the RGB image and the depth image using an optimized deep convolutional spike neural network, and at the same time uses a semi-supervised fusion edge detection automatic labeling model to classify and label the solid waste, obtain the classification labeling results, and transmit the classification labeling results to the classification sorting and storage module;

[0088] The classification, sorting and storage module classifies, sorts and stores the solid waste according to the classification and labeling results.

[0089] Example 2, see Figure 2 ,In the automatic labeling and classification module, an optimized deep convolutional spike neural network is used to classify solid waste, which specifically includes the following steps:

[0090] Step S1: data preprocessing, preprocessing the RGB image of solid waste to obtain a preprocessed RGB image;

[0091] Furthermore, step S1 specifically includes the following steps:

[0092] Step S11: image loading, loading the RGB image of the solid waste into the computer;

[0093] Step S12: image size adjustment, adjusting the size of the RGB image of the solid waste to a fixed and uniform size;

[0094] Step S13: pixel normalization, normalizing all pixels in the RGB image to obtain a normalized RGB image;

[0095] Step S14: image enhancement, using a random rotation method to process the normalized RGB image to obtain a preprocessed RGB image;

[0096] Step S2: data division, dividing the preprocessed RGB images into a training set and a test set;

[0097] Step S3: Initialize the deep convolutional spike neural network, initialize the spike encoding layer, flattening layer, convolution layer, maximum pooling layer, dropout layer and fully connected layer, set the maximum pooling layer parameters, pulse conditions, neuron parameters and synaptic weights;

[0098] Step S4: Use a convolutional layer to extract features from the training set to obtain a feature map, use a maximum pooling layer to downsample the feature map to obtain n non-overlapping rectangular area feature maps, and flatten the n rectangular area feature maps to obtain n one-dimensional feature vectors and the modulus of the n one-dimensional feature vectors;

[0099] Step S5: converting the n one-dimensional feature vectors into n current amplitudes according to the magnitude of the vector modulus;

[0100] Step S6: Arrange the n current amplitudes to obtain a pulse sequence diagram, and classify the pulse sequence diagram using a pulse neural network according to neuron parameters and synaptic weights to obtain a preliminary classification result of the solid waste;

[0101] Step S7: Calculate the accuracy of the preliminary classification result, and optimize the maximum pooling layer parameters using the Border Collie optimization algorithm according to the accuracy of the preliminary classification result, and optimize the neuron parameters and synaptic weights using the Archimedes algorithm to obtain an optimized deep convolutional spike neural network and preliminary classification results;

[0102] Further, step S7 specifically includes the following steps:

[0103] Step S71: define the objective function and determine the accuracy of the classification result as the target;

[0104] Step S72: Initialize the population individuals, parameters and iteration times of the Border Collie optimization algorithm, where each individual represents a set of maximum pooling layer parameters, and initialize the parameters and fitness threshold of the Archimedean algorithm, where the parameters include position density, volume and acceleration;

[0105] Step S73: performing optimization operation on the population individuals according to the border collie optimization algorithm to obtain the preliminary optimal individual;

[0106] Step S74: optimizing the neuron parameters and synaptic weights according to the Archimedean algorithm to obtain preliminarily optimized neuron parameters and synaptic weights;

[0107] Step S75: repeating steps S73 and S74 until the number of iterations and the fitness threshold are reached, and obtaining the optimal maximum pooling layer parameters and optimized neuron parameters and synaptic weights;

[0108] Step S76: Input the optimal maximum pooling layer parameters and the optimized neuron parameters and synaptic weights into the deep convolutional spiking neural network to obtain the optimized deep convolutional spiking neural network and preliminary classification results.

[0109] By performing the above operations, in view of the problems of slow classification and low processing efficiency in the prior art when recycling and storing solid waste, the present invention adopts an optimized deep convolutional pulse neural network to perform more efficient feature extraction, dynamically capture the association between different parts of solid waste, reduce the network computing load and storage requirements, and more efficiently process large-scale solid waste images. At the same time, the Border Collie optimization algorithm and the Archimedes algorithm are used to optimize the neural network parameters to achieve higher classification efficiency and accuracy.

[0110] Example 3, see Figure 2 In the automatic labeling and classification module, a semi-supervised automatic labeling model with fusion edge detection is used to classify and label solid waste, which specifically includes the following steps:

[0111] Step Q1: Segment the depth image, and perform dilation, erosion and contour search operations to obtain a segmented depth map and an outer contour map, and grayscale the RGB image to obtain a grayscale image;

[0112] Step Q2: Use median filtering to filter the grayscale image to obtain a median filter image, multiply the median filter image with the segmentation depth map to obtain a filtered segmentation image, and multiply the depth image with the segmentation depth map to obtain a fused depth map;

[0113] Step Q3: Perform edge detection on the filtered segmentation image to obtain an RGB edge image, and perform edge detection on the fused depth image to obtain a depth edge image;

[0114] Step Q4: Use the outer contour map to crop the RGB edge map and the depth edge map and perform fixed threshold segmentation to obtain an enhanced RGB contour map and an enhanced depth contour map;

[0115] Step Q5: Simplify the enhanced RGB contour map and the enhanced depth contour map using a skeletonization algorithm to obtain a simplified RGB contour map and a simplified depth contour map;

[0116] Step Q6: Fuse the simplified RGB contour map with the simplified depth contour map to obtain a fused RGB image, and use the optimized deep convolutional spike neural network to annotate the fused RGB image to obtain a classification annotation result.

[0117] By performing the above operations, in order to solve the problem that the prior art has low accuracy in identifying various types of waste when facing a large amount of stacked waste, which affects the subsequent targeted storage, the present invention adopts an automatic labeling model based on semi-supervised fusion edge detection, which can perform more accurate classification and identification of stacked waste. At the same time, based on the high classification accuracy of the optimized deep convolutional pulse neural network, the automatic labeling method can also efficiently and accurately classify and identify solid waste when facing a large amount of stacking, which is convenient for subsequent targeted storage.

[0118] Example 4, see Figure 3 In the automatic labeling and classification module, the transferable joint cross-modal learning method is used to tune the parameters of the automatic labeling model to obtain the optimized classification labeling results, which specifically includes the following steps:

[0119] Step K1: Pseudo feature extraction, specifically including the following steps:

[0120] Step K11: using a text encoder to perform text feature encoding on the classification labels of each classification of solid waste;

[0121] Step K12: using a visual encoder to perform visual feature encoding on the pre-processed RGB images of each type of classified solid waste;

[0122] Step K13: Pseudo-feature design: the text feature code and visual feature code of each classification of solid waste are fused to obtain the pseudo-feature of each classification of solid waste, which is defined as:

[0123] P=[p1 , p2 , p3 , ... , pn ] ;

[0124] In the formula, Representative Pseudo features of classes, is the number of categories, is a pseudo feature set;

[0125] Step K2: embedding feature extraction and adapter fine-tuning, specifically including the following steps:

[0126] Step K21: Use the visual encoder to encode the pseudo features into global visual features to obtain a global visual embedding;

[0127] Step K22: Use the text encoder to encode the pseudo features into global text features to obtain global text embedding;

[0128] Step K23: real category labeling, collecting solid waste images with real classification labels in the network to train the visual neural network, obtaining the real classification categories of all solid wastes and the trained visual neural network, using the trained visual neural network to identify the pre-processed RGB images of solid waste, and obtaining the real classification results of all solid wastes;

[0129] Step K24: Use a text encoder to encode the real text features of all the real classification results of solid waste to obtain real text embedding;

[0130] Step K25: Adapter fine-tuning. Use the adapter module to perform adapter fine-tuning on the global visual embedding, global text embedding, and real text embedding. The formula used is as follows:

[0131] ;

[0132] ;

[0133] ;

[0134] In the formula, and Represents the adapter fine-tuning functions for visual and text modes respectively, is the preset weight, , and Represent global visual embedding, global text embedding and real text embedding respectively, , and They represent the global visual embedding, global text embedding, and real text embedding after fine-tuning of the adapter respectively;

[0135] Step K3: Multi-mode similarity calculation, specifically including the following steps:

[0136] Step K31: Calculate the pseudo-similarity of the same modality. The formula used is as follows:

[0137] ;

[0138] In the formula, and Represents the first Class global text embedding and Real-like text embeddings, Represents similarity calculation, Representative Homomodal pseudo-similarity of classes;

[0139] Step K32: Calculate cross-modal pseudo similarity, using the following formula:

[0140] ;

[0141] In the formula, and Represents the first Class global text embedding and Class global visual embedding, Represents similarity calculation, Representative Cross-modal pseudo-similarity of classes;

[0142] Step K33: Calculate the cross-modal true similarity. The formula used is as follows:

[0143] ;

[0144] In the formula, and Represents the first Class-real text embedding and Class global visual embedding, Represents similarity calculation, Representative Cross-modal true similarity of classes;

[0145] Step K4: Pseudo contrast loss calculation, the formula used is as follows:

[0146] ;

[0147] ;

[0148] ;

[0149] In the formula, To preset adjustment parameters, and Represent the cross-modal pseudo similarity and the cross-modal true similarity respectively, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the pseudo-ratio loss;

[0150] Step K5: Calculation of true contrast loss, the formula used is as follows:

[0151] ;

[0152] ;

[0153] In the formula, To preset adjustment parameters, and Represent the pseudo-similarity of the same modality and the true similarity of cross-modality, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the true contrast loss;

[0154] Step K6: Calculate the total training loss by adding the true contrast loss to the pseudo contrast loss to get the total loss;

[0155] Step K7: Automatic labeling model tuning: According to the total loss, a semi-supervised fusion edge detection automatic labeling model is parameter-tuned, and the optimized automatic labeling model is used to automatically label the solid waste to obtain an optimized classification labeling result.

[0156] By performing the above operations, pseudo features of solid waste are generated by automatic labeling, visual and textual joint learning is performed based on the real features and pseudo features of solid waste, and the model expression ability is enhanced through adapter fine-tuning. Finally, contrastive learning is used to achieve the tuning of multi-label classification annotation.

[0157] Embodiment 5: This embodiment is based on the above embodiment, and the selection of solid waste is as follows:

[0158] Solid waste is collected from the city for system training. Irregular cartons, glass, bricks, transparent plastic water bottles, chemical reagent bottles and fruit cores are collected as solid waste and the quantity is increased to a certain extent. At the same time, all solid waste is stacked to better simulate the real solid waste treatment situation.

[0159] Embodiment 6: This embodiment is based on the above embodiment, and the camera device selection in the image acquisition module is as follows:

[0160] Linear color camera: Nikon z8;

[0161] Sensor type: CMOS;

[0162] Effective pixels: 4571W;

[0163] RAW photo output: 14bit;

[0164] Applicable to: Professional level;

[0165] Depth camera: Intel;

[0166] Standard ISO sensitivity: ISO 100-12800;

[0167] Pixel: 0-2000w.

[0168] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0169] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0170] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A solid waste intelligent storage and management system, Features: It includes crawler conveying module, image acquisition module, automatic labeling and classification module, classification sorting and storage module and central control module; The crawler conveying module uses crawlers to convey the solid waste and finally conveys it to the classification, sorting and storage module; The image acquisition module uses a linear color camera to acquire an RGB image of the solid waste on the crawler, and uses a depth image camera to acquire a depth image of the solid waste on the crawler, and transmits the RGB image and the depth image to the automatic labeling and classification module, wherein the linear color camera and the depth image camera are placed above the crawler; The automatic labeling and classification module classifies solid wastes according to RGB images and depth images using an optimized deep convolutional spike neural network, and classifies and labels solid wastes using a semi-supervised fusion edge detection automatic labeling model, and uses a transferable joint cross-modal learning method to tune the parameters of the automatic labeling model to obtain optimized classification labeling results, and transmits the optimized classification labeling results to the classification sorting and storage module; The method of using an optimized deep convolutional pulse neural network to classify solid waste specifically includes the following steps: Step S1: data preprocessing, preprocessing the RGB image of solid waste to obtain a preprocessed RGB image; Step S2: data division, dividing the preprocessed RGB images into a training set and a test set; Step S3: Initialize the deep convolutional spike neural network, initialize the spike encoding layer, flattening layer, convolution layer, maximum pooling layer, dropout layer and fully connected layer, set the maximum pooling layer parameters, pulse conditions, neuron parameters and synaptic weights; Step S4: Use a convolutional layer to extract features from the training set to obtain a feature map, use a maximum pooling layer to downsample the feature map to obtain n non-overlapping rectangular area feature maps, and flatten the n rectangular area feature maps to obtain n one-dimensional feature vectors and the modulus of the n one-dimensional feature vectors; Step S5: converting the n one-dimensional feature vectors into n current amplitudes according to the magnitude of the vector modulus; Step S6: Arrange the n current amplitudes to obtain a pulse sequence diagram, and classify the pulse sequence diagram using a pulse neural network according to neuron parameters and synaptic weights to obtain a preliminary classification result of the solid waste; Step S7: Calculate the accuracy of the preliminary classification result, and optimize the maximum pooling layer parameters using the Border Collie optimization algorithm according to the accuracy of the preliminary classification result, and optimize the neuron parameters and synaptic weights using the Archimedes algorithm to obtain an optimized deep convolutional spike neural network and preliminary classification results; The method of using a semi-supervised fusion edge detection automatic labeling model to classify and label solid waste specifically includes the following steps: Step Q1: Segment the depth image, and perform dilation, erosion and contour search operations to obtain a segmented depth map and an outer contour map, and grayscale the RGB image to obtain a grayscale image; Step Q2: Use median filtering to filter the grayscale image to obtain a median filter image, multiply the median filter image with the segmentation depth map to obtain a filtered segmentation image, and multiply the depth image with the segmentation depth map to obtain a fused depth map; Step Q3: Perform edge detection on the filtered segmentation image to obtain an RGB edge image, and perform edge detection on the fused depth image to obtain a depth edge image; Step Q4: Use the outer contour map to crop the RGB edge map and the depth edge map and perform fixed threshold segmentation to obtain an enhanced RGB contour map and an enhanced depth contour map; Step Q5: Simplify the enhanced RGB contour map and the enhanced depth contour map using a skeletonization algorithm to obtain a simplified RGB contour map and a simplified depth contour map; Step Q6: Fuse the simplified RGB contour map with the simplified depth contour map to obtain a fused RGB image, and use the optimized deep convolutional spike neural network to annotate the fused RGB image to obtain a classification annotation result; The classification, sorting and storage module classifies and stores the solid waste according to the optimized classification and labeling results; The central control module directly controls the crawler conveying module, the image acquisition module, the automatic labeling and classification module, and the classification, sorting and storage module.

2. The intelligent storage and management system for solid waste according to claim 1, Features: In the automatic labeling and classification module, a transferable joint cross-modal learning method is used to tune the parameters of the automatic labeling model to obtain optimized classification labeling results, which specifically includes the following steps: Step K1: Pseudo feature extraction: using a text encoder to perform text feature encoding on the classification labels of each classification of solid waste, using a visual encoder to perform visual feature encoding on the preprocessed RGB image of each classification of solid waste, and fusing the text feature encoding and visual feature encoding of each classification of solid waste to obtain pseudo features of each classification of solid waste; Step K2: embedding feature extraction and adapter fine-tuning processing, processing the pseudo features to obtain the global visual embedding, global text embedding and real text embedding after adapter fine-tuning; Step K3: Multimodal similarity calculation: multimodal similarity calculation is performed on the global visual embedding, global text embedding and real text embedding after fine-tuning of the adapter to obtain cross-modal pseudo-similarity, cross-modal real similarity and same-modal pseudo-similarity; Step K4: Pseudo contrast loss calculation, the formula used is as follows: ; ; ; In the formula, To preset adjustment parameters, and Represent the cross-modal pseudo similarity and the cross-modal true similarity respectively, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the pseudo-to-spurious ratio loss, The number of categories representing the total classification of solid waste; Step K5: Calculation of true contrast loss, the formula used is as follows: ; ; In the formula, To preset adjustment parameters, and Represent the pseudo-similarity of the same modality and the true similarity of cross-modality, represents the natural number exponential function, and is the intermediate quantity, represents the cross entropy loss function, represents the true contrast loss, The number of categories representing the total classification of solid waste; Step K6: Calculate the total training loss by adding the true contrast loss to the pseudo contrast loss to get the total loss; Step K7: Automatic labeling model tuning: According to the total loss, a semi-supervised fusion edge detection automatic labeling model is parameter-tuned, and the optimized automatic labeling model is used to automatically label the solid waste to obtain an optimized classification labeling result.