Composite material waste sorting method based on hyperspectral technology and AI classification model

Abstract

The present invention relates to a composite material waste sorting method based on hyperspectral technology and an AI classification model, comprising the following steps: S1: collecting multimodal data; S2: preprocessing and fusing the collected multimodal data to obtain joint features of the multimodal data; S3: constructing a classification model based on a multimodal neural network model and performing classification based on the joint features; S4: based on the classification results, using the SegFormer deep segmentation network to identify the components of the waste at the pixel level and achieve accurate segmentation of objects composed of multiple materials; S5: maintaining target tracking of the waste trajectory through a target tracking algorithm; S6: based on the target tracking results in S5, predicting the position of the waste on the conveyor belt in real time, and accurately performing sorting actions based on the classification and segmentation results. The present invention can efficiently and accurately process complex composite material waste.

Description

Technical Field

[0001] The present invention relates to the field of intelligent sorting, and in particular to a composite material waste sorting method based on hyperspectral technology and an AI classification model. Background Art

[0002] Composite waste is composed of a variety of materials, typically metals, plastics, and fibers. Traditional manual sorting or RGB-based sorting methods struggle to accurately distinguish these materials, leading to low sorting efficiency, resource waste, and environmental pollution. Traditional sorting technologies (manual, mechanical, or simple image recognition) suffer from low accuracy and inefficiency when dealing with complex composite waste. Summary of the Invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide a composite material waste sorting method based on hyperspectral technology and AI classification model, which can efficiently and accurately process complex composite material waste.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A composite waste sorting method based on hyperspectral technology and AI classification model includes the following steps:

[0006] S1: Deploy several hyperspectral and RGB cameras at different nodes along the conveyor belt, with each node focusing on multimodal data collection of garbage in a certain section;

[0007] S2: Preprocess the collected multimodal data and fuse the RGB and hyperspectral data according to the timestamp and spatial position to obtain the joint features of the multimodal data;

[0008] S3: Build a classification model based on the multimodal neural network model and perform classification based on joint features;

[0009] S4: Based on the classification results, the SegFormer deep segmentation network is used to identify the components of garbage at the pixel level and segment objects composed of multiple materials;

[0010] S5: Maintain target tracking of the garbage trajectory through target tracking algorithm;

[0011] S6: Based on the target tracking results in S5, the position of the garbage on the conveyor belt is predicted in real time, and sorting actions are performed according to the classification and segmentation results.

[0012] Furthermore, S1 is specifically:

[0013] Pre-measure the length L and width W of the sections at different locations along the conveyor belt;

[0014] According to the garbage flow rate v, combined with the density of garbage R density , determine the sampling frequency f of each node sampling ;

[0015]

[0016] Among them, d min is the minimum distance between adjacent garbage;

[0017] According to the sampling field F view =(θ H ,θ V ), where θ H is the horizontal field of view, θ V It is the vertical field of view, which determines the overlapping area, avoids the blind area of garbage detection, and the deployment spacing of adjacent nodes. S nodes for:

[0018] ;

[0019] Each node includes an RGB camera, a hyperspectral imager, a trigger device, and a controller;

[0020] Assume that the garbage on the conveyor belt moves at a speed v, and the sensor dynamically captures the multimodal information of the object: the length of the conveyor belt covered by each sampling is calculated based on the resolution and FOV of the sensor. L capture and width W capture :

[0021]

[0022] Where h is the vertical height from the sensor to the conveyor belt, where:

[0023] L capture ≥L,W capture ≥W;

[0024] Garbage is collected between two sampling time intervals T capture Displacement Δ x Calculated using the following formula:

[0025] ;

[0026] Ensure Δx≤d min , ensuring continuous sampling and no missed detection.

[0027] Furthermore, S2 is specifically:

[0028] Assume the collected RGB data is R t(x,y,c), corresponding to timestamp t R ;Hyperspectral data is H t (x,y,λ), corresponding to timestamp t H ;where x,y represent the spatial dimensions, λ represents the wavelength of light in the hyperspectral image; c is the number of channels;

[0029] For each frame of RGB data R t , find the value with R t The hyperspectral data H with the closest timestamp t :

[0030]

[0031] in, It’s the time difference; Indicates that in the set of candidate time points, find the time difference Minimum t H value;

[0032] After time alignment, the synchronized multimodal data pairs (R t ,H t );

[0033] Spatial alignment is achieved by calibrating the transformation matrix T align Completed, the formula is:

[0034]

[0035] Among them, [u,v] Hyper are the pixel coordinates of the hyperspectral data; [u,v] RGB are the pixel coordinates of the RGB data;

[0036] Align the pixel coordinates of the hyperspectral data with the pixel coordinates of the RGB data to obtain spatially consistent multimodal data;

[0037] After time and space alignment, for each pixel, the RGB data and hyperspectral data are directly spliced to form a joint feature F t (x,y).

[0038] Furthermore, the multimodal neural network model includes a hyperspectral branch, an RGB branch, a multi-layer fusion strategy layer, and a classification head, as follows:

[0039] The hyperspectral branch inputs the pre-processed hyperspectral data , extract spectral features and capture material information at different wavelengths:

[0040] Extract local spectral-spatial features through 3D convolutional layers :

[0041] ;

[0042] Among them, kernel is the convolution kernel size; Channels is the number of convolution output channels; Con3D represents the 3D convolution operation;

[0043] Spectral Transformer module, capturing global spectral dependencies :

[0044] ;

[0045] Among them, Q, K, and V are query vector, key vector, and value vector respectively; W Q 、W K 、W V are the projection parameters of query vector, key vector and value vector respectively; d k is the dimension of the attention space; T represents transposition; Attention represents the attention mechanism; LayerNorm is the normalization operation;

[0046] Finally, the spectral feature vector is generated through the global average pooling operation f H :

[0047] ;

[0048] Among them, GlobalAvgPool represents the global average pooling operation;

[0049] The RGB branch inputs the preprocessed RGB data R′ and extracts shape and texture features:

[0050] Through the ResNet-50 backbone network, multi-level spatial features are extracted:

[0051] ;

[0052] Among them, ResNet gradually extracts multi-level features through 4 residual blocks ResNet-Block1, ResNet-Block2, ResNet-Block3, and ResNet-Block4, including 、 、 、 ;

[0053] Spatial attention module to enhance key area features:

[0054] ;

[0055] Among them, Conv2D represents the convolution operation; ⊙ is the element-level weighted operation; σ is the activation function;

[0056] w R (x,y) is the attention weight; is the output of the spatial attention module;

[0057] Global average pooling to generate spatial feature vectors:

[0058] ;

[0059] Multi-layer fusion strategy layer, fusing hyperspectral and RGB features at multiple levels to enhance modality complementarity:

[0060] Directly splice the feature maps of the two modes in the middle layer to obtain the spliced features :

[0061] ;

[0062] After splicing, the convolutional layer is fused to obtain the fused features. :

[0063] ;

[0064] The classification head passes the fused features through the fully connected layer to output the classification results:

[0065] ;

[0066] in, is the category probability; Softmax is the activation function; W c and b c are the weights and biases of the fully connected layer.

[0067] Furthermore, for the classification model, meta-learning is introduced to optimize the classification model for the few-sample composite material, as follows:

[0068] Different material classification tasks are used as the meta-learning task set τ i ;

[0069] Each task consists of a support set S=(X,Y) and a query set Q'=(X′,Y′);

[0070] Model training process:

[0071] In the task set τ i Update the model parameters on the support set:

[0072] ;

[0073] in, are model parameters, is the learning rate; is the meta-task τ i The corresponding loss function; For the task set τ i The updated model parameters on the support set; Represents the gradient of the loss function with respect to the parameter θ;

[0074] Calculate the loss of the query set and find the total gradient across all tasks to optimize the global parameters:

[0075] ;

[0076] in, is the meta-learning rate;

[0077] When new types of garbage data are input, the support set is quickly updated:

[0078] .

[0079] Furthermore, based on the classification results, we used the SegFormer deep segmentation network to identify the components of garbage at the pixel level and segment objects composed of multiple materials, as follows:

[0080] The class probability output by the classification model As a priori condition for the segmentation network, the joint features and class probabilities of the input multimodal data ;

[0081] Through conditional splicing, the classification label is broadcast to each pixel to form a conditional feature :

[0082] ;

[0083] in, is the expanded condition information;

[0084] The SegFormer deep segmentation network consists of an encoder and a decoder.

[0085] The encoder uses the pre-trained SegFormer-B5 model, with input Fcond(x,y) and output multi-scale features :

[0086] ;

[0087] Among them, SegFormerEncoder is the encoder of the SegFormer-B5 model;

[0088] The decoder fuses multi-scale features and generates a segmentation mask M seg :

[0089] ;

[0090] Wherein, MLP represents MLP decoder;

[0091] Using cross entropy loss Processing pixel-level output:

[0092] ;

[0093] in, One-Hot representation of the pixel's true material label; is the predicted segmentation mask; k is the material category index.

[0094] Furthermore, S5 is specifically:

[0095] Obtain the target's location, category, and segmentation mask information from the classification and segmentation modules: The segmentation module outputs the pixel-level mask of the a-th object to generate the minimum bounding rectangle b boxa :

[0096] b boxa =[x min ,y min ,x max ,y max ];

[0097] Among them, [x min ,y min ] is the coordinate of the upper left corner of the rectangle [x max ,y max ] is the coordinate of the lower right corner of the rectangular box; the classification module outputs the category probability vector of the i-th item ; Pixel-level material segmentation result M output by SegFormer i (x,y);

[0098] Use Kalman filtering to predict the position of the target in the next frame; and use the Hungarian algorithm to associate the detection box with the tracked target;

[0099] The following rules are used to handle the appearance of new targets, disappearance or occlusion of old targets:

[0100] New target appears: the unmatched detection box is initialized as a new tracking target, assigned a unique ID, and the Kalman filter state is initialized;

[0101] The old target disappears: If the tracked target does not match the detection box for Nmiss consecutive frames, its trajectory is terminated;

[0102] Occlusion handling: For short-term occlusion, the position is predicted and tracked using Kalman filtering; for long-term occlusion, the object is re-identified by combining the material information of the segmentation mask.

[0103] Furthermore, the Kalman filter is used to predict the position of the target in the next frame, as follows:

[0104] Assume that the conveyor belt motion is a linear uniform velocity model, and the current state vector x t Including the position and velocity of the target:

[0105] ;

[0106] Among them, x, y are the coordinates of the target center; v x ,v y is the target speed in the conveyor coordinate system; the superscript T indicates transposition;

[0107] Kalman filter prediction equation:

[0108] ;

[0109] Among them, x t-1 is the state vector at the previous moment; F is the state transfer matrix; is the process noise covariance matrix; is the predicted state of the current frame; w t is the weight coefficient; is the prediction covariance matrix; P t-1 is the error covariance matrix of the previous moment;

[0110] State transition matrix F:

[0111] ;

[0112] Where, Δt is the time interval between two frames;

[0113] When the detection box matches the tracking target, update the status:

[0114] ;

[0115] Among them, H is the observation matrix; z t is the observed value; R is the observation noise covariance; K t is the Kalman gain; is the prediction covariance matrix; is the updated covariance matrix; x t is the updated state, and I is the identity matrix.

[0116] Furthermore, the Hungarian algorithm is used to associate the detection box with the tracked target, as follows:

[0117] Match the detection box of the current frame with the existing tracking target, and construct the cost matrix using Mahalanobis distance and appearance feature distance :

[0118] ;

[0119] Among them, λ1,λ 2， λ3 is the weighting coefficient; Target and goals Movement distance; Target and goals The apparent distance; Target and goals The category distance;

[0120] Mahalanobis distance:

[0121] ;

[0122] in, is the covariance matrix of the Kalman filter; Target The predicted status; Target Observed values of

[0123] If DeepSORT is used, the depth features of the detection box are extracted and the cosine distance is calculated;

[0124] Calculate the category matching penalty item based on the category label output by the classification module;

[0125] The total cost is minimized by the Hungarian algorithm to obtain the optimal match between the detection box and the tracked target.

[0126] Furthermore, based on the motion state of S5 target tracking, the time and space coordinates of the garbage arriving at the sorting port are predicted, and sorting instructions are generated according to the classification and segmentation results. Through time-space mapping and delay compensation, the sorting device is triggered to act at the right time.

[0127] The present invention has the following beneficial effects:

[0128] 1. This invention effectively solves the problem of mixed material identification and automatic sorting by combining garbage movement tracking technology in a conveyor belt environment with a real-time classification system. This not only improves sorting efficiency but also significantly increases resource recycling and utilization, providing important technical support for achieving a green circular economy and sustainable development.

[0129] 2. The garbage classification model based on a multimodal neural network can efficiently process complex multi-material garbage, accurately classify garbage through joint features, and use the SegFormer deep segmentation network to accurately segment the material composition of garbage at the pixel level, supporting fine-grained recognition of multi-material mixed objects (such as plastic-wrapped metal and composite materials).

[0130] 3. The present invention uses a target tracking algorithm to continuously track the trajectory of garbage, ensuring that the target is not lost in the dynamic environment on the conveyor belt. Based on the target tracking results, the position of garbage on the conveyor belt is predicted in real time, providing accurate position information for the robotic arm or sorting equipment, ensuring the accuracy and timeliness of the sorting action. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0132] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0133] refer to Figure 1 In this embodiment, a composite waste sorting method based on hyperspectral technology and AI classification model is provided, comprising the following steps:

[0134] S1: Deploy several hyperspectral and RGB cameras at different nodes along the conveyor belt, with each node focusing on multimodal data collection of garbage in a certain section;

[0135] S2: Preprocess the collected multimodal data and fuse the RGB and hyperspectral data according to the timestamp and spatial position to obtain the joint features of the multimodal data;

[0136] S3: Build a classification model based on the multimodal neural network model and perform classification based on joint features;

[0137] S4: Based on the classification results, the SegFormer deep segmentation network is used to identify the components of garbage at the pixel level and achieve accurate segmentation of objects composed of multiple materials;

[0138] S5: Maintain target tracking of the garbage trajectory through the target tracking algorithm;

[0139] S6: Based on the target tracking results in S5, the position of garbage on the conveyor belt is predicted in real time, and the sorting action is accurately performed based on the classification and segmentation results.

[0140] In this embodiment, S1 is specifically:

[0141] Pre-measure the length L and width W of the sections at different locations along the conveyor belt;

[0142] According to the garbage flow velocity v (the speed of the conveyor belt), combined with the density of the garbage R density (the amount of garbage detected per unit time), determine the sampling frequency f of each node sampling ;

[0143]

[0144] Among them, d min is the minimum distance between adjacent garbage;

[0145] According to the sampling field of view (FOV) F view =(θ H ,θ V ), where θ H is the horizontal field of view, θ V It is the vertical field of view, which determines the overlapping area, avoids the blind area of garbage detection, and the deployment spacing of adjacent nodes. S nodes for:

[0146] ;

[0147] Each node includes an RGB camera, a hyperspectral imager, a trigger device, and a controller;

[0148] Assume that the garbage on the conveyor belt moves at a speed v, and the sensor dynamically captures the multimodal information of the object: the length of the conveyor belt covered by each sampling is calculated based on the resolution and FOV of the sensor. L capture and width W capture :

[0149]

[0150] Where h is the vertical height from the sensor to the conveyor belt, where:

[0151] L capture ≥L,W capture ≥W;

[0152] Garbage is collected between two sampling time intervals T capture Displacement Δ x Calculated using the following formula:

[0153] ;

[0154] Ensure Δx≤d min , ensuring continuous sampling and no missed detection.

[0155] In this embodiment, S2 is specifically:

[0156] Assume the collected RGB data is R t (x,y,c), corresponding to timestamp t R ;Hyperspectral data is H t (x,y,λ), corresponding to timestamp t H ;where x,y represent the spatial dimensions, λ represents the wavelength of light in the hyperspectral image; c is the number of channels;

[0157] For each frame of RGB data R t , find the value with R t The hyperspectral data H with the closest timestamp t :

[0158]

[0159] in, It’s the time difference; Indicates that in the set of candidate time points, find the time difference Minimum t H value;

[0160] After time alignment, the synchronized multimodal data pairs (R t ,H t );

[0161] Spatial alignment is achieved by calibrating the transformation matrix T align Completed, the formula is:

[0162]

[0163] Among them, [u,v] Hyper are the pixel coordinates of the hyperspectral data; [u,v] RGB are the pixel coordinates of the RGB data;

[0164] Align the pixel coordinates of the hyperspectral data with the pixel coordinates of the RGB data to obtain spatially consistent multimodal data;

[0165] After time and space alignment, for each pixel, the RGB data and hyperspectral data are directly spliced to form a joint feature F t (x,y).

[0166] In this embodiment, the multimodal neural network model includes a hyperspectral branch, an RGB branch, a multi-layer fusion strategy layer, and a classification head, as follows:

[0167] The hyperspectral branch inputs the pre-processed hyperspectral data , extract spectral features and capture material information at different wavelengths:

[0168] Extract local spectral-spatial features through 3D convolutional layers :

[0169] ;

[0170] Among them, kernel is the convolution kernel size; Channels is the number of convolution output channels; Con3D represents the 3D convolution operation;

[0171] Spectral Transformer module, capturing global spectral dependencies :

[0172] ;

[0173] Among them, Q, K, and V are query vector, key vector, and value vector respectively; W Q 、W K 、W V are the projection parameters of query vector, key vector and value vector respectively; d k is the dimension of the attention space; T represents transposition; Attention represents the attention mechanism; LayerNorm is the normalization operation;

[0174] Finally, the spectral feature vector is generated through the global average pooling operation f H :

[0175] ;

[0176] Among them, GlobalAvgPool represents the global average pooling operation;

[0177] The RGB branch inputs the preprocessed RGB data R′ and extracts shape and texture features:

[0178] Through the ResNet-50 backbone network, multi-level spatial features are extracted:

[0179] ;

[0180] Among them, ResNet gradually extracts multi-level features through 4 residual blocks ResNet-Block1, ResNet-Block2, ResNet-Block3, and ResNet-Block4, including 、 、 、 ;

[0181] Spatial attention module to enhance key area features:

[0182] ;

[0183] Among them, Conv2D represents the convolution operation; ⊙ is the element-level weighted operation; σ is the activation function;

[0184] w R (x,y) is the attention weight; is the output of the spatial attention module;

[0185] Global average pooling to generate spatial feature vectors:

[0186] ;

[0187] Multi-layer fusion strategy layer, fusing hyperspectral and RGB features at multiple levels to enhance modality complementarity:

[0188] Directly splice the feature maps of the two modes in the middle layer to obtain the spliced features :

[0189] ;

[0190] After splicing, the convolutional layer is fused to obtain the fused features. :

[0191] ;

[0192] The classification head passes the fused features through the fully connected layer to output the classification results:

[0193] ;

[0194] in, is the category probability; Softmax is the activation function; W c and b c are the weights and biases of the fully connected layer.

[0195] In this embodiment, a classification model is introduced to optimize the classification model by meta-learning for composite materials with a small number of samples (such as glass fiber composite and carbon fiber composite), as follows:

[0196] Use different material classification tasks (such as glass fiber composite vs. carbon fiber composite) as the meta-learning task set τ i ;

[0197] Each task consists of a support set Q' and a query set S;

[0198] Model training process:

[0199] In the task set τ i Update the model parameters on the support set:

[0200] ;

[0201] in, are model parameters, is the learning rate; is the meta-task τ i The corresponding loss function is, For the task τ i The updated model parameters on the support set; Represents the gradient of the loss function with respect to the parameter θ;

[0202] Calculate the loss of the query set and find the total gradient across all tasks to optimize the global parameters:

[0203] ;

[0204] in, is the meta-learning rate;

[0205] When new types of junk data (such as hybrid composites) are input, they are quickly updated using the support set:

[0206] .

[0207] In this example, based on the classification results, the SegFormer deep segmentation network is used to identify the components of garbage at the pixel level and achieve accurate segmentation of objects composed of multiple materials, as follows:

[0208] The class probability output by the classification model As a priori condition for the segmentation network, the joint features and class probabilities of the input multimodal data ;

[0209] Through conditional splicing, the classification label is broadcast to each pixel to form a conditional feature :

[0210] ;

[0211] in, is the expanded condition information;

[0212] The SegFormer deep segmentation network consists of an encoder and a decoder.

[0213] The encoder uses the pre-trained SegFormer-B5 model, with input Fcond(x,y) and output multi-scale features :

[0214] ;

[0215] Among them, SegFormerEncoder is the encoder of the SegFormer-B5 model;

[0216] The decoder fuses multi-scale features and generates a segmentation mask M seg :

[0217] ;

[0218] Wherein, MLP represents MLP decoder;

[0219] Using cross entropy loss Processing pixel-level output:

[0220] ;

[0221] in, One-Hot representation of the pixel's true material label; is the predicted segmentation mask; k is the material category index.

[0222] In this embodiment, S5 is specifically:

[0223] Obtain the target's location, category, and segmentation mask information from the classification and segmentation modules: The segmentation module outputs the pixel-level mask of the a-th object to generate the minimum bounding rectangle b boxa :

[0224] b boxa =[x min ,y min ,x max ,y max ];

[0225] Among them, [x min ,y min ] is the coordinate of the upper left corner of the rectangle [x max ,y max ] is the coordinate of the lower right corner of the rectangular box; the classification module outputs the category probability vector of the i-th item ; Pixel-level material segmentation result M output by SegFormer i (x,y);

[0226] Use Kalman filtering to predict the position of the target in the next frame; and use the Hungarian algorithm to associate the detection box with the tracked target;

[0227] The following rules are used to handle the appearance of new targets, disappearance or occlusion of old targets:

[0228] New target appears: the unmatched detection box is initialized as a new tracking target, assigned a unique ID, and the Kalman filter state is initialized;

[0229] The old target disappears: If the tracked target does not match the detection box for Nmiss consecutive frames, its trajectory is terminated;

[0230] Occlusion handling: For short-term occlusion, the position is predicted and tracked using Kalman filtering; for long-term occlusion, the object is re-identified by combining the material information of the segmentation mask.

[0231] In this embodiment, the Kalman filter is used to predict the position of the target in the next frame, as follows:

[0232] Assume that the conveyor belt motion is a linear uniform velocity model, and the current state vector x t Including the position and velocity of the target:

[0233] ;

[0234] Among them, x, y are the coordinates of the target center; v x ,v y is the target speed in the conveyor coordinate system; the superscript T indicates transposition;

[0235] Kalman filter prediction equation:

[0236] ;

[0237] Among them, x t-1 is the state vector at the previous moment; F is the state transfer matrix; is the process noise covariance matrix; is the predicted state of the current frame; w t is the weight coefficient; is the prediction covariance matrix; P t-1 is the error covariance matrix of the previous moment;

[0238] State transition matrix F:

[0239] ;

[0240] Where, Δt is the time interval between two frames;

[0241] When the detection box matches the tracking target, update the status:

[0242] ;

[0243] Among them, H is the observation matrix; z t is the observed value; R is the observation noise covariance; K t is the Kalman gain; is the prediction covariance matrix; is the updated covariance matrix; x t is the updated state, and I is the identity matrix.

[0244] In this embodiment, the detection box and the tracking target are associated using the Hungarian algorithm, as follows:

[0245] Match the detection box of the current frame with the existing tracking target, and construct the cost matrix using Mahalanobis distance (motion similarity) and appearance feature distance (category and segmentation consistency) :

[0246] ;

[0247] Among them, λ1,λ 2， λ3 is the weighting coefficient; Target and goals Movement distance; Target and goals The apparent distance; Target and goals The category distance;

[0248] Mahalanobis distance (motion similarity):

[0249] ;

[0250] in, is the covariance matrix of the Kalman filter; Target The predicted status; Target Observed values of

[0251] If DeepSORT is used, extract the deep features of the detection box (such as the embedding vector of ResNet-50) and calculate the cosine distance;

[0252] Calculate the category matching penalty item based on the category label output by the classification module;

[0253] The total cost is minimized by the Hungarian algorithm to obtain the optimal match between the detection box and the tracked target.

[0254] In this embodiment, S6 specifically includes: predicting the spatiotemporal coordinates of the garbage arriving at the sorting port based on the motion state of the target tracking in S5, generating sorting instructions based on the classification and segmentation results (material, category), and triggering the sorting device (such as air valve, robotic arm) to act at the right time through time-space mapping and delay compensation.

[0255] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0256] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0257] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0258] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0259] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A composite waste sorting method based on hyperspectral technology and AI classification model, characterized in that: The following steps are involved: S1: Deploy several hyperspectral and RGB cameras at different nodes along the conveyor belt, with each node focusing on multimodal data collection of garbage in a certain section; S2: Preprocess the collected multimodal data and fuse the RGB and hyperspectral data according to the timestamp and spatial position to obtain the joint features of the multimodal data; S3: Constructing a classification model based on a multimodal neural network model to perform classification based on joint features; the multimodal neural network model includes a hyperspectral branch, an RGB branch, a multi-layer fusion strategy layer, and a classification head; the classification model introduces meta-learning to optimize the classification model for small-sample composite materials; S4: Based on the classification results, the SegFormer deep segmentation network is used to identify the components of garbage at the pixel level and segment objects composed of multiple materials; S5: Maintain target tracking of the garbage trajectory through target tracking algorithm; S6: Based on the target tracking results in S5, the location of the garbage on the conveyor belt is predicted in real time, and sorting actions are performed according to the classification and segmentation results; S1 is specifically: Pre-measure the length L and width W of the sections at different locations along the conveyor belt; According to the garbage flow rate v, combined with the density of garbage R density , determine the sampling frequency f of each node sampling ; ； Among them, d min is the minimum distance between adjacent garbage; According to the sampling field F view =(θ H ,θ V ), where θ H is the horizontal field of view, θ V It is the vertical field of view, which determines the overlapping area, avoids the blind area of garbage detection, and the deployment spacing of adjacent nodes. S nodes for: ； Each node includes an RGB camera, a hyperspectral imager, a trigger device, and a controller; Assume that the garbage on the conveyor belt moves at a speed v, and the sensor dynamically captures the multimodal information of the object: the length of the conveyor belt covered by each sampling is calculated based on the resolution and FOV of the sensor. L capture and width W capture : ； Where h is the vertical height from the sensor to the conveyor belt, where: L capture ≥L,W capture ≥W; Garbage is collected between two sampling time intervals T capture Displacement Δ x Calculated using the following formula: ； Ensure Δx≤d min , ensuring continuous sampling and no missed detection; S2 is specifically: Assume the collected RGB data is R t (x,y,c), corresponding to timestamp t R ;Hyperspectral data is H t (x,y,λ), corresponding to timestamp t H ;where x,y represent the spatial dimensions, λ represents the wavelength of light in the hyperspectral image; c is the number of channels; For each frame of RGB data R t , find the value with R t The hyperspectral data H with the closest timestamp t : ； in, It’s the time difference; Indicates that in the set of candidate time points, find the time difference Minimum t H value; After time alignment, the synchronized multimodal data pairs (R t ,H t ); Spatial alignment is achieved by calibrating the transformation matrix T align Completed, the formula is: ； Among them, [u,v] Hyper are the pixel coordinates of the hyperspectral data; [u,v] RGB are the pixel coordinates of the RGB data; Align the pixel coordinates of the hyperspectral data with the pixel coordinates of the RGB data to obtain spatially consistent multimodal data; After time and space alignment, for each pixel, the RGB data and hyperspectral data are directly spliced to form a joint feature F t (x,y).

2. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 1 is characterized in that: The multimodal neural network model includes a hyperspectral branch, an RGB branch, a multi-layer fusion strategy layer, and a classification head, as follows: The hyperspectral branch inputs the pre-processed hyperspectral data , extract spectral features and capture material information at different wavelengths: Extract local spectral-spatial features through 3D convolutional layers : ； Among them, kernel is the convolution kernel size; Channels is the number of convolution output channels; Con3D represents the 3D convolution operation; Spectral Transformer module, capturing global spectral dependencies : ； Among them, Q, K, and V are query vector, key vector, and value vector respectively; W Q 、W K 、W V are the projection parameters of query vector, key vector and value vector respectively; d k is the dimension of the attention space; T represents transposition; Attention represents the attention mechanism; LayerNorm is the normalization operation; Finally, the spectral feature vector is generated through the global average pooling operation f H : ； Among them, GlobalAvgPool represents the global average pooling operation; The RGB branch inputs the preprocessed RGB data R′ and extracts shape and texture features: Through the ResNet-50 backbone network, multi-level spatial features are extracted: ； Among them, ResNet gradually extracts multi-level features through 4 residual blocks ResNet-Block1, ResNet-Block2, ResNet-Block3, and ResNet-Block4, including 、 、 、 ; Spatial attention module to enhance key area features: ； Among them, Conv2D represents the convolution operation; ⊙ is the element-level weighted operation; σ is the activation function; w R (x,y) is the attention weight; is the output of the spatial attention module; Global average pooling to generate spatial feature vectors: ； Multi-layer fusion strategy layer, fusing hyperspectral and RGB features at multiple levels to enhance modality complementarity: Directly splice the feature maps of the two modes in the middle layer to obtain the spliced features : ； After splicing, the convolutional layer is fused to obtain the fused features. : ； The classification head passes the fused features through the fully connected layer to output the classification results: ； in, is the category probability; Softmax is the activation function; W c and b c are the weights and biases of the fully connected layer.

3. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 2 is characterized in that: The classification model introduces a meta-learning optimization classification model for small sample composite materials, as follows: Different material classification tasks are used as the meta-learning task set τ i ; Each task consists of a support set S = (X, Y) and a query set Q' = (X', Y'); Model training process: In the task set τ i Update the model parameters on the support set: ； in, are model parameters, is the learning rate; is the meta-task τ i The corresponding loss function; For the task set τ i The updated model parameters on the support set; Represents the gradient of the loss function with respect to the parameter θ; Calculate the loss of the query set and find the total gradient across all tasks to optimize the global parameters: ； in, is the meta-learning rate; When new types of garbage data are input, the support set is quickly updated: 。 4. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 1 is characterized in that: Based on the classification results, the SegFormer deep segmentation network is used to identify the components of garbage at the pixel level and segment objects composed of multiple materials. The details are as follows: The class probability output by the classification model As a priori condition for the segmentation network, the joint features and class probabilities of the input multimodal data ; Through conditional splicing, the classification label is broadcast to each pixel to form a conditional feature : ； in, is the expanded condition information; The SegFormer deep segmentation network consists of an encoder and a decoder. The encoder uses the pre-trained SegFormer-B5 model, with input Fcond(x,y) and output multi-scale features : ； Among them, SegFormerEncoder is the encoder of the SegFormer-B5 model; The decoder fuses multi-scale features and generates a segmentation mask M seg : ； Wherein, MLP represents MLP decoder; Using cross entropy loss Processing pixel-level output: ； in, One-Hot representation of the pixel's true material label; is the predicted segmentation mask; k is the material category index.

5. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 1 is characterized in that: The S5 is specifically: Obtain the target's location, category, and segmentation mask information from the classification and segmentation modules: The segmentation module outputs the pixel-level mask of the a-th object to generate the minimum bounding rectangle b boxa : b boxa =[x min ,y min ,x max ,y max ]; Among them, [x min ,y min ] is the coordinate of the upper left corner of the rectangle [x max ,y max ] is the coordinate of the lower right corner of the rectangular box; the classification module outputs the category probability vector of the ath item ; Pixel-level material segmentation result M output by SegFormer a (x,y); Use Kalman filtering to predict the position of the target in the next frame; and use the Hungarian algorithm to associate the detection box with the tracked target; The following rules are used to handle the appearance of new targets, disappearance or occlusion of old targets: New target appears: the unmatched detection box is initialized as a new tracking target, assigned a unique ID, and the Kalman filter state is initialized; The old target disappears: If the tracked target does not match the detection box for Nmiss consecutive frames, its trajectory is terminated; Occlusion handling: For short-term occlusion, the position is predicted and tracked using Kalman filtering; for long-term occlusion, the object is re-identified by combining the material information of the segmentation mask.

6. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 5 is characterized in that: The Kalman filter is used to predict the position of the target in the next frame, as follows: Assume that the conveyor belt motion is a linear uniform velocity model, and the current state vector x t Including the position and velocity of the target: ； Among them, x, y are the coordinates of the target center; v x ,v y is the target speed in the conveyor coordinate system; the superscript T indicates transposition; Kalman filter prediction equation: ； Among them, x t-1 is the state vector at the previous moment; F is the state transfer matrix; is the process noise covariance matrix; is the predicted state of the current frame; w t is the weight coefficient; is the prediction covariance matrix; P t-1 is the error covariance matrix of the previous moment; State transition matrix F: ； Where, Δt is the time interval between two frames; When the detection box matches the tracking target, update the status: ； Among them, H is the observation matrix; z t is the observed value; R is the observation noise covariance; K t is the Kalman gain; is the prediction covariance matrix; is the updated covariance matrix; is the updated state, and I is the identity matrix.

7. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 5 is characterized in that: The Hungarian algorithm is used to associate the detection box with the tracked target, as follows: Match the detection box of the current frame with the existing tracking target, and construct the cost matrix using Mahalanobis distance and appearance feature distance : ； Among them, λ1,λ 2， λ3 is the weighting coefficient; Target and goals Movement distance; Target and goals The apparent distance; Target and goals The category distance; Mahalanobis distance: ； in, is the covariance matrix of the Kalman filter; Target The predicted status; Target Observed values of If DeepSORT is used, the depth features of the detection box are extracted and the cosine distance is calculated; Calculate the category matching penalty item based on the category label output by the classification module; The total cost is minimized by the Hungarian algorithm to obtain the optimal match between the detection box and the tracked target.

8. The composite material waste sorting method based on hyperspectral technology and AI classification model according to claim 1 is characterized in that: The S6 specifically includes: predicting the spatiotemporal coordinates of the garbage arriving at the sorting port based on the motion state of the target tracking in S5, generating sorting instructions according to the classification and segmentation results, and triggering the sorting device to act at the right time through time-space mapping and delay compensation.

Images