A method, medium and system for rapid identification of solid waste in port cargo

Abstract

The present invention provides a method, medium and system for rapid identification of solid waste of port cargo, belonging to the technical field of multimodal electronic digital data processing model identification, including: automatic identification of solid waste, collecting multimodal data of different resolutions, including images, infrared spectra and laser reflections, etc. Preprocessing and feature extraction of these data, and obtaining fusion features by using a fusion method with adjustable weights. Using the highest resolution data and its manual annotation as a training set, a teacher model is trained. Then, in order of resolution from high to low, a distillation training method is adopted, using the recognition model of the previous resolution as a teacher, to train a student model of each resolution. For the port cargo to be identified, its multimodal data is collected, and the recognition model closest to the resolution is selected for prediction to obtain the identification result of the solid waste. The technical problem of large consumption of computing resources and low recognition efficiency in the prior art is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-modal electrical digital data processing model recognition, and in particular, relates to a method, medium and system for rapid recognition of solid waste in port cargo. Background Art

[0002] Traditional solid waste identification methods mainly rely on manual visual inspection and simple mechanical sorting. This method is not only time-consuming and labor-intensive, but also easily affected by human factors, making it difficult to ensure the accuracy and consistency of the identification results. With the advancement of science and technology, some automated identification technologies have gradually been applied to the field of solid waste management.

[0003] Image recognition technology is one of the most widely used methods. By collecting high-resolution images of solid waste and combining them with computer vision algorithms, preliminary identification of waste types can be achieved. However, in order to obtain accurate identification results, only high-resolution images can be collected. The acquisition and processing of high-resolution images requires a lot of computing resources, which may lead to low recognition efficiency in practical applications. Summary of the invention

[0004] In view of this, the present invention provides a method, medium and system for rapid identification of solid waste in port cargo, which can solve the technical problems in the prior art that identification can only be performed based on high-resolution images, consumes a lot of computing resources and has low identification efficiency.

[0005] The present invention is achieved in that:

[0006] The first aspect of the present invention provides a method for quickly identifying solid waste in port cargo, comprising the following steps:

[0007] S10, collecting multiple groups of multimodal data of different resolution groups of solid waste in a large area and corresponding manual annotations of the solid waste, wherein the multimodal data includes image data, infrared spectrum data, and reflection data of lasers of different colors;

[0008] S20, preprocessing and feature extraction are performed on the image data, infrared spectrum data and reflection data of different color lasers to obtain image features, infrared spectrum features and reflection features;

[0009] S30, fusing the image features, infrared spectrum features, and reflection features corresponding to the multimodal data of each resolution group in a fusion method with adjustable weights to obtain fused features;

[0010] S40, using the fusion features corresponding to the multimodal data with the highest resolution group and the corresponding manual annotations of solid waste as a training set, training a teacher model, recorded as the highest resolution solid waste recognition model;

[0011] S50, in descending order of resolution groups, starting from the multimodal data of the second highest resolution group and ending with the multimodal data of the lowest resolution group, distillation training is performed in sequence, specifically, using the solid waste recognition model corresponding to the previous resolution group as the teacher model, using the fusion features corresponding to the current resolution group and the corresponding solid waste manual annotations as the training set, training a student model, and finally obtaining a solid waste recognition model for each resolution;

[0012] S60, for the port cargo to be identified, collect its multimodal data, and preprocess and extract features of the collected multimodal data to obtain image features, infrared spectrum features, and reflection features;

[0013] S70. According to the multimodal data resolution group of the port cargo to be identified, select a solid waste identification model with a resolution group closest to the solid waste identification model, input the extracted features into the selected solid waste identification model, and obtain the identification result of the solid waste.

[0014] The large area refers specifically to an area exceeding 1 hectare.

[0015] The method for sorting the resolution groups is specifically to sort the image data resolutions from high to low.

[0016] On the basis of the above technical solution, the method for rapid identification of solid waste in port cargo of the present invention can also be improved as follows:

[0017] The step S20 specifically includes:

[0018] Step 201, performing pre-processing operations such as color correction, denoising, and histogram equalization on the image data;

[0019] Step 202: extracting color features, texture features and shape features from the preprocessed image data;

[0020] Step 203, performing preprocessing such as band correction, baseline correction, and spectrum normalization on the infrared spectrum data;

[0021] Step 204, extracting features such as spectral reflectance, absorption peak position and intensity from the pre-processed infrared spectrum data;

[0022] Step 205, pre-processing the reflection data such as noise filtering and signal correction;

[0023] Step 206: extract reflection intensity and reflection time characteristics of the red, green and blue lasers from the pre-processed reflection data.

[0024] Among them, the different resolution groups, for different modalities, specifically have image data resolutions of 10 meters to 0.1 meters, infrared spectrum data resolutions of 100 bands to 1000 bands, and reflection data resolutions of different color lasers of 5 nanometers to 0.5 nanometers. Each resolution group is represented by a vector, specifically [image resolution, infrared resolution, laser resolution].

[0025] Furthermore, the different resolution groups include at least 10 sequence resolution groups, specifically [0.1 meter, 1000 band, 0.5 nanometers], [0.2 meter, 900 band, 1 nanometer], [0.5 meter, 800 band, 1.5 nanometers], [1 meter, 700 band, 2 nanometers], [2 meters, 600 band, 2.5 nanometers], [3 meters, 500 band, 3 nanometers], [4 meters, 400 band, 3.5 nanometers], [5 meters, 300 band, 4 nanometers], [7 meters, 200 band, 4.5 nanometers], and [10 meters, 100 band, 5 nanometers].

[0026] Furthermore, it also includes the step of cross-validating solid waste identification models of different resolutions. Specifically, the data of one resolution group is used to test the models trained by other resolution groups to evaluate the generalization ability of the model on data of different resolutions.

[0027] Furthermore, the image data specifically refers to a multispectral image including visible light bands and near-infrared bands, and is obtained by shooting with a multispectral camera; the infrared spectral data specifically refers to spectral data including short-wave infrared, medium-wave infrared and long-wave infrared bands, and is obtained by scanning with a hyperspectral imager; the reflection data of lasers of different colors specifically refers to reflection intensity data after irradiation with red, green and blue lasers, and is obtained by emitting lasers through a lidar system and receiving reflected signals.

[0028] Furthermore, the image features include color features, texture features and shape features; the infrared spectrum features include spectral reflectance, absorption peak position and intensity; and the reflection features include reflection intensity and reflection time of lasers of different colors.

[0029] Furthermore, the teacher model adopts a deep convolutional neural network structure, including multiple convolutional layers, pooling layers, feature extraction modules, attention mechanism modules and classification modules; the feature extraction module uses residual connections to improve feature extraction capabilities; the attention mechanism module uses a combination of spatial and channel attention; the classification module uses a fully connected layer to achieve final classification;

[0030] The student model specifically includes an embedded equation group module, a feature extraction module, an attention mechanism module, and a classification module. The embedded equation group module includes a scale equation, a smoothing equation, an edge enhancement equation, and a feature fusion equation. The feature extraction module adopts a lightweight convolutional neural network structure. The attention mechanism module uses a self-attention mechanism. The classification module uses a fully connected layer with dropout.

[0031] The feature extraction module is used to extract deep features, the input is the fused features, and the output is the deep features;

[0032] The attention mechanism module is used to focus on important features, the input is deep features, and the output is weighted features;

[0033] The classification module is used to classify solid waste, the input is the weighted features, and the output is the classification result.

[0034] The distillation training specifically realizes knowledge transfer by minimizing the cross entropy loss of the output of the teacher model and the student model and the L2 distance of the feature representation.

[0035] Furthermore, the scaling equation is used to adjust the scale of the feature, the input is the original feature, and the output is the scaled feature;

[0036] The smoothing equation is used to reduce the influence of noise, the input is the scaled feature, and the output is the smoothed feature;

[0037] The edge enhancement equation is used to highlight the edge of the target, the input is the smoothed feature, and the output is the edge enhanced feature;

[0038] The feature fusion equation is used to fuse multimodal features, the input is the edge-enhanced feature, and the output is the fused feature.

[0039] The formula of each equation in the embedded equations module is expressed as follows:

[0040] 1. The scale equation is specifically expressed as follows:

[0041] ;

[0042] In the formula, is the feature map; is the scale parameter; is the diffusion coefficient; is the bias term coefficient; is the original feature map; and is the coefficient of the higher-order term; is the noise term.

[0043] The parameter acquisition method is:

[0044] Optimize in the range [0.01, 0.1] by grid search; Optimize in the range [-1,1] by gradient descent; and Optimize in the range [0.001, 0.01] by random search; Gaussian white noise is used with a mean of 0 and a variance of 0.01.

[0045] 2. The smoothing equation is specifically expressed as follows:

[0046] ;

[0047] In the formula, is the smoothed feature map; is the diffusion function; is the fidelity coefficient; is the input feature map; is the cross derivative coefficient; is the error term.

[0048] The parameter acquisition method is:

[0049] Defined as ,in is the contrast parameter, which is optimized in the range of [1,10] by bisection method; Optimize in the range [0.1,1] using the Adam optimizer; Optimization in the range [0.001, 0.01] by simulated annealing algorithm; Laplace noise is used with a location parameter of 0 and a scale parameter of 0.005.

[0050] 3. The edge enhancement equation is specifically expressed as follows:

[0051] ;

[0052] In the formula, It is the feature map after edge enhancement; is the diffusion coefficient; is the edge stop function coefficient; is the edge threshold; is the coefficient of the reaction term; is the threshold of the response item; is the coefficient of the third-order derivative term; is the disturbance term.

[0053] The parameter acquisition method is:

[0054] Optimize in the range [0.1,1] by particle swarm optimization; Optimization in the range [1,10] by genetic algorithm; Adaptive calculation through Otsu algorithm; Optimize in the range [0.1,1] by cross validation; Set to 0.5; Optimize in the range [0.001, 0.01] by Bayesian optimization; A uniformly distributed noise is used with a range of [-0.01, 0.01].

[0055] 4. The feature fusion equation is specifically expressed as follows:

[0056] ;

[0057] In the formula, is the fused feature map; is the fusion function; They are image features, infrared spectrum features and reflection features respectively; is the diffusion coefficient; is the nonlinear term coefficient; is the bistable term coefficient; is a random disturbance term.

[0058] The parameter acquisition method is:

[0059] Defined as ,in is the weight coefficient, which is optimized in the range of [0,1] by the alternating direction multiplier method and satisfies ; Optimize by gradient descent in the range [0.01, 0.1]; Optimize in the range [0.001, 0.01] by grid search; Optimize in the range [0.1,1] by random search; Poisson noise is used with an intensity parameter of 0.01.

[0060] The second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are run in a computer, they are used to execute the above-mentioned method for rapid identification of solid waste in port cargo.

[0061] The third aspect of the present invention provides a rapid identification system for solid waste in port cargo, which includes the above-mentioned computer-readable storage medium, as well as a multi-spectral camera, a hyperspectral imager, red, green and blue laser transmitters and a laser radar.

[0062] Compared with the prior art, the method, medium and system for rapid identification of solid waste in port cargo provided by the present invention have the following beneficial effects:

[0063] First, the present invention adopts a multimodal data fusion method to effectively improve the accuracy and robustness of solid waste identification. By simultaneously collecting image data, infrared spectrum data, and reflection data of lasers of different colors, this method can capture the characteristic information of solid waste from multiple dimensions. This combination of multimodal data can not only make up for the limitations of a single data source, but also provide a more comprehensive and rich feature description. For example, image data can provide information on the shape and color of the waste, infrared spectrum data can reflect the chemical composition of the waste, and laser reflection data can capture surface texture and material characteristics. This multi-dimensional feature fusion significantly improves the recognition model's ability to distinguish different types of solid waste, especially for those wastes that are similar in appearance but different in composition.

[0064] Secondly, the present invention realizes the dynamic adjustment of the importance of different modal data by adopting a feature fusion method with adjustable weights. This flexible fusion strategy enables the recognition system to adaptively adjust the contribution weights of various features according to the actual application scenario and the characteristics of the waste type. For example, in some cases, infrared spectral features may be more recognizable than image features, and the accuracy of recognition can be improved by increasing their weights. This dynamic adjustment mechanism greatly enhances the adaptability and generalization ability of the model, enabling it to maintain stable recognition performance in different port environments and waste types.

[0065] Again, the present invention innovatively introduces a resolution-based knowledge distillation training strategy, which effectively solves the problem of high-resolution data acquisition and processing. Through gradual distillation from a high-resolution model to a low-resolution model, this method successfully transfers the recognition capability of the high-resolution model to the low-resolution model. This strategy not only reduces the requirements for hardware equipment, but also significantly improves the efficiency of recognition. In particular, in practical applications, a model with appropriate resolution can be selected according to the specific situation, while ensuring recognition accuracy and maximizing computational efficiency.

[0066] In addition, the distillation training strategy of the present invention also has the advantage of reducing the need for labeled data. By using the high-resolution model as a teacher, additional supervision information can be provided to the low-resolution model, thereby improving the learning effect of the model under the condition of limited labeled data. This not only reduces the cost of data collection and annotation, but also improves the scalability and adaptability of the model.

[0067] Another significant advantage of the method of the present invention is its adaptability to data of different resolutions. In practical applications, due to the influence of factors such as equipment performance and environmental conditions, the resolution of the collected data may vary. This method solves the technical problem that the existing technology can only perform recognition based on high-resolution images, consumes a lot of computing resources and has low recognition efficiency by training a series of recognition models with different resolutions and selecting the closest resolution model in the recognition stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flow chart of the method provided by the present invention;

[0069] Figure 2 It is a schematic diagram of the optimization process of three feature weights in the multimodal feature fusion process;

[0070] Figure 3 It is a schematic diagram of the distribution of solid waste samples in the three-dimensional feature space;

[0071] Figure 4 This is a schematic diagram of the solid waste identification accuracy under different resolution combinations;

[0072] Figure 5 Schematic diagram of the performance comparison of knowledge distillation models between different resolution combinations;

[0073] Figure 6 Figure 2 shows the computing resource consumption of different models. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0075] like Figure 1 As shown, it is a flow chart of a method for rapid identification of solid waste in port cargo provided by the present invention, and the method comprises the following steps:

[0076] S10, collecting multiple groups of multimodal data of different resolution groups of solid waste in a large area and corresponding manual annotation of the solid waste, the multimodal data including image data, infrared spectrum data and reflection data of lasers of different colors;

[0077] S20, preprocessing and feature extraction are performed on the image data, infrared spectrum data and reflection data of different color lasers to obtain image features, infrared spectrum features and reflection features;

[0078] S30, fusing the image features, infrared spectrum features, and reflection features corresponding to the multimodal data of each resolution group in a fusion method with adjustable weights to obtain fused features;

[0079] S40, using the fusion features corresponding to the multimodal data with the highest resolution group and the corresponding manual annotations of solid waste as a training set, training a teacher model, recorded as the highest resolution solid waste recognition model;

[0080] S50, in descending order of resolution groups, starting from the multimodal data of the second highest resolution group and ending with the multimodal data of the lowest resolution group, distillation training is performed in sequence, specifically, using the solid waste recognition model corresponding to the previous resolution group as the teacher model, using the fusion features corresponding to the current resolution group and the corresponding solid waste manual annotations as the training set, training a student model, and finally obtaining a solid waste recognition model for each resolution;

[0081] S60, for the port cargo to be identified, collect its multimodal data, and preprocess and extract features of the collected multimodal data to obtain image features, infrared spectrum features, and reflection features;

[0082] S70. According to the multimodal data resolution group of the port cargo to be identified, select a solid waste identification model with the closest resolution group, input the extracted features into the selected solid waste identification model, and obtain the identification result of the solid waste.

[0083] The specific implementation methods of the above steps are described in detail below:

[0084] Step S10, collects multiple groups of multimodal data of different resolution groups of solid waste in a large area and the corresponding manual labeling of solid waste. First, the target area is determined to be a large area of ​​more than 1 hectare. Then, a variety of sensing devices are used to obtain image data, infrared spectrum data and reflection data of solid waste in the area. Among them, the resolution range of image data is 10 meters to 0.1 meters; the number of bands of infrared spectrum data ranges from 100 bands to 1000 bands; the spectral resolution range of reflection data is 5 nanometers to 0.5 nanometers. Each data acquisition uses a different resolution combination, with a total of 10 sequence resolution combinations: [0.1 m, 1000 band, 0.5 nm], [0.2 m, 900 band, 1 nm], [0.5 m, 800 band, 1.5 nm], [1 m, 700 band, 2 nm], [2 m, 600 band, 2.5 nm], [3 m, 500 band, 3 nm], [4 m, 400 band, 3.5 nm], [5 m, 300 band, 4 nm], [7 m, 200 band, 4.5 nm], [10 m, 100 band, 5 nm]. At the same time, when collecting data for each resolution group, professionals are required to manually annotate the collected solid waste samples. Through this step, multimodal data of different resolution groups and their corresponding manual annotations are obtained.

[0085] Step S20, preprocessing and feature extraction are performed on the image data, infrared spectrum data and reflection data of different color lasers. Specifically, for image data, color correction, denoising, histogram equalization and other preprocessing operations are first performed, and then color features, texture features and shape features are extracted. For infrared spectrum data, band correction, baseline correction, spectrum normalization and other preprocessing are first performed, and then features such as spectral reflectivity, absorption peak position and intensity are extracted. For reflection data, noise filtering, signal correction and other preprocessing are first performed, and then the reflection intensity and reflection time features of red, green and blue lasers are extracted. Through this step, rich feature information is extracted from the original multimodal data.

[0086] Step S30, for each resolution group, the image features, infrared spectrum features, and reflection features corresponding to the multimodal data are fused using a fusion method with adjustable weights to obtain fused features. Specifically, a feature fusion function is defined: ,in They are image features, infrared spectrum features and reflection features, is an adjustable weight coefficient that satisfies By optimizing these weight coefficients by the alternating direction multiplier method, the fusion features of each resolution group can be obtained. This step aims to make full use of the complementarity of different modal features and improve the feature expression ability.

[0087] Step S40, using the fusion features corresponding to the multimodal data with the highest resolution group and the corresponding manual annotations of solid waste as the training set, a teacher model is trained, which is recorded as the highest resolution solid waste recognition model. Specifically, a deep convolutional neural network is used as the teacher model, which includes a multi-layer convolution layer, a pooling layer, a residual connection feature extraction module, a spatial and channel attention mechanism module, and a fully connected classification module. The model is trained with the highest resolution data so that it can efficiently extract multimodal features and accurately classify solid waste.

[0088] Step S50, distillation training is performed in sequence according to the order of resolution groups from high to low, starting from the multimodal data with the second highest resolution group and ending with the multimodal data with the lowest resolution group. Specifically, for each resolution group, the solid waste recognition model corresponding to the previous resolution group is used as the teacher model, and the fused features and manual annotations of the current resolution group are used as the training set to train a student model. The student model includes an embedded equations module, a lightweight convolutional feature extraction module, a self-attention mechanism module, and a fully connected classification module. Knowledge distillation is achieved by minimizing the cross entropy loss output by the teacher model and the student model and the L2 distance of the feature representation. In this way, solid waste recognition models for different resolutions are obtained.

[0089] Step S60, for the port cargo to be identified, collect its multimodal data, and preprocess and extract features of the collected multimodal data to obtain image features, infrared spectrum features and reflection features. The specific operation is similar to step S20, and the feature information of the solid waste to be identified is obtained by preprocessing and extracting features of the image, spectrum and reflection data of the sample to be identified.

[0090] Step S70, according to the multimodal data resolution group of the port cargo to be identified, select the solid waste identification model with the closest resolution group, input the extracted features into the selected solid waste identification model, and obtain the identification result of the solid waste. First, determine the resolution group to which the sample to be identified belongs based on the image resolution, number of spectral bands, laser spectral resolution and other information. Then, from the solid waste identification models of each resolution group trained in the aforementioned step S50, select the model that is closest to the resolution group of the sample to be identified. Finally, input the features extracted in step S60 into the model for inference to obtain the classification result of the solid waste to be identified.

[0091] In order to further improve the generalization ability of the model, the following extension operations were performed:

[0092] 1. Cross-validate solid waste identification models with different resolutions. Specifically, use data from one resolution group to test models trained with other resolution groups to evaluate the generalization ability of the model on data with different resolutions.

[0093] 2. Combine data of different modalities with different resolutions to form fusion features of more resolution groups. For example, combine the image resolution in [0.1m, 1000 band, 0.5nm] with the infrared resolution in [0.2m, 900 band, 1nm] and the laser resolution in [0.5m, 800 band, 1.5nm] to form a new resolution group [0.1m, 900 band, 1.5nm]. Through this cross-combination method, 90 additional resolution combinations were generated.

[0094] Specifically, the technical principle of the present invention is based on multimodal data fusion, knowledge distillation and hierarchical recognition model. Specifically:

[0095] First, multimodal data fusion is one of the core technologies of this method. By simultaneously collecting image data, infrared spectrum data, and reflection data of lasers of different colors, this method can capture the characteristics of solid waste from multiple dimensions. The principle of this multimodal data acquisition is based on the different sensitivities of different types of sensors to material properties. For example, visible light cameras mainly capture the appearance characteristics of objects, infrared spectrometers can detect the molecular vibration information of substances, and laser reflection can reflect the surface structure and material properties. By combining these different types of data, a more comprehensive and robust feature representation can be constructed. This multidimensional feature representation helps to improve the model's ability to distinguish different types of solid waste, especially when dealing with wastes with similar appearance but different compositions.

[0096] Secondly, this method adopts a feature fusion strategy with adjustable weights. This design is based on the understanding that the contribution of different features in different scenarios may vary. By introducing an adjustable weight mechanism, the system can dynamically adjust the importance of various features according to the actual application scenario and the characteristics of the waste type. The rationality of this adaptive fusion strategy lies in its ability to maximize the advantages of various features while minimizing the impact of their limitations. For example, under certain lighting conditions, image features may not be as reliable as infrared spectral features. In this case, increasing the weight of infrared features can improve recognition accuracy. This flexible fusion mechanism not only improves the adaptability of the model, but also enhances its robustness in complex and changing environments.

[0097] Another key technical principle of the present invention is the resolution-based knowledge distillation training strategy. Knowledge distillation is a model compression and knowledge transfer technology, and its core idea is to use a complex "teacher" model to guide the learning of a simple "student" model. In this method, the high-resolution model serves as the teacher and the low-resolution model serves as the student. The rationality of this design lies in that high-resolution data usually contains richer detail information, so models trained based on high-resolution data often have stronger recognition capabilities. However, the acquisition and processing of high-resolution data often requires more resources and time. Through knowledge distillation, the "knowledge" of the high-resolution model can be effectively transferred to the low-resolution model, thereby significantly improving the operating efficiency of the model while maintaining a high recognition accuracy.

[0098] In addition, this method adopts a hierarchical approach to knowledge distillation, that is, starting from the highest resolution model, gradually transferring knowledge to the low-resolution model. This progressive distillation strategy helps to maintain the continuity and consistency of knowledge and reduce the loss of information during the transfer process. At the same time, this strategy can also adapt to data of different resolutions and enhance the generalization ability of the model. The rationality of this design lies in that it fully considers the problem of data quality differences that may be faced in practical applications and provides a flexible and effective solution.

[0099] In the process of model training, this method also cleverly utilizes manually annotated data. By combining manual annotation with model prediction results, additional supervision information can be introduced in the knowledge distillation process, thereby improving the learning effect of the student model. This design can not only make full use of limited annotated data, but also correct possible errors in the teacher model to a certain extent, improving the reliability of the entire recognition system.

[0100] Another important principle of this method is to select the most suitable recognition model for the sample to be recognized. By comparing the resolution of the data to be recognized with the trained resolution models, the model with the closest resolution is selected for recognition. The rationality of this design lies in that it can maximize the matching of the feature distribution of the input data and the model, thereby obtaining the best recognition effect. At the same time, this strategy also improves the adaptability of the method, enabling it to handle input data of different quality and resolution.

[0101] A specific embodiment 1 of the method provided by the first aspect of the present invention is provided below. The specific implementation of each step in this embodiment 1 is described in detail as follows: For step S10, the target area is first determined to be a large area with an area of ​​more than 1 hectare. Then, multiple sensing devices are used to obtain multimodal data of solid waste in the area, including image data, infrared spectrum data, and reflection data. Specifically, the resolution range of the image data is meters, the number of bands of infrared spectral data ranges from Band, the spectral resolution range of reflectance data is Nanometers. A total of 10 sequence resolution combinations are set, namely:

[0102]

[0103] .

[0104] When collecting data for each resolution group, professionals are required to manually label the collected solid waste samples. Through this step, multimodal data of different resolution groups and their corresponding manual labels are obtained.

[0105] For step S20, firstly, the image data, infrared spectrum data and reflection data are preprocessed and feature extracted respectively. For the image data, the preprocessing includes color correction, denoising, histogram equalization and other operations, and the feature extraction includes color feature extraction. , Texture features and shape features For infrared spectral data, preprocessing includes band correction, baseline correction, spectral normalization and other operations, and feature extraction includes spectral reflectance , absorption peak position and the absorption peak intensity For reflection data, preprocessing includes noise filtering, signal correction and other operations, and feature extraction includes the reflection intensity of red, green and blue lasers. and reflection time .

[0106] For step S30, define a feature fusion function ,in is an adjustable weight coefficient that satisfies The alternating direction multiplier method is used to optimize these weight coefficients so that the fusion features It can maximize the representation of the discriminative information of the sample.

[0107] For step S40, a deep convolutional neural network is used as the teacher model. The model includes multiple convolutional layers , Pooling layer , residual connection feature extraction module , spatial attention mechanism module , channel attention mechanism module And the fully connected classification module The fusion features corresponding to the multimodal data with the highest resolution group And the corresponding solid waste manual labeling As a training set, train this teacher model.

[0108] For step S50, distillation training is performed in order from high to low resolution groups, starting from the multimodal data with the second highest resolution group. , the previous resolution group The corresponding solid waste recognition model is used as the teacher model, and the current resolution group Fusion features and manual annotation As a training set, train a student model. The student model includes the embedded equations module , lightweight convolutional feature extraction module , Self-attention mechanism module and a fully connected classification module By minimizing the cross entropy loss between the outputs of the teacher model and the student model And the L2 distance of feature representation To achieve knowledge distillation.

[0109] In step S60, firstly, image data of the port cargo to be identified is collected. , infrared spectrum data and reflection data Then, the multimodal data are subjected to the same preprocessing and feature extraction operations as step S20 to obtain image features , infrared spectral characteristics and reflection features .

[0110] In step S70, first, according to the image resolution of the sample to be identified, , number of spectral bands , laser spectral resolution and other information to determine the resolution group to which it belongs Then, from the solid waste recognition models of each resolution group trained in the above step S50, select the solid waste recognition model with the resolution group of the sample to be identified. The closest model. Finally, the features extracted in step S60 are , and Input the selected solid waste identification model to obtain the classification results of the solid waste to be identified .

[0111] In order to better understand and implement the present invention, the following provides Example 2 of a specific application scenario of the present invention; a port management department urgently needs a fast and accurate solid waste identification method to improve the efficiency of cargo clearance and the level of environmental management. The department decided to use the multimodal fusion and knowledge distillation technology proposed in the present invention to automatically identify solid waste entering and leaving the port.

[0112] First, the department determined that the target identification area was a large freight terminal with an area of ​​more than 1 hectare. Subsequently, a variety of sensor devices were used to collect data on solid waste in the area. Specifically, a multispectral camera was used to obtain image data, and the measurement range covered the visible light band and the near-infrared band; a hyperspectral imager was used to scan and obtain spectral data of the short-wave infrared, medium-wave infrared and long-wave infrared bands; a laser radar system was used to emit red, green and blue lasers and receive reflected signals, and the reflection data of lasers of different wavelengths was obtained.

[0113] For the above three modal data, the department designed 10 different resolution combinations, as shown in Table 1. Among them, the resolution range of image data is 10 meters to 0.1 meters, the number of bands of infrared spectrum data ranges from 100 bands to 1000 bands, and the spectral resolution range of laser reflection data is 5 nanometers to 0.5 nanometers. At the same time, for the data collection of each resolution group, professionals manually annotated the solid waste samples. Through this series of work, the department finally obtained a rich multimodal solid waste data set and its corresponding annotation information.

[0114] Table 1. 10 solid waste data resolution combinations

[0115] Serial number Image resolution (meters) Number of spectral bands Laser resolution (nanometers) 1 0.1 1000 0.5 2 0.2 900 1 3 0.5 800 1.5 4 1 700 2 5 2 600 2.5 6 3 500 3 7 4 400 3.5 8 5 300 4 9 7 200 4.5 10 10 100 5

[0116] Next, the department preprocessed and extracted features from the multimodal data. For image data, color correction, denoising, and histogram equalization were first preprocessed, and then color features (such as RGB histogram), texture features (such as Gabor filter response), and shape features (such as Hu moment) were extracted. For infrared spectral data, band correction, baseline correction, and spectral normalization were preprocessed, and features such as spectral reflectance, absorption peak position, and intensity were extracted. For laser reflection data, noise filtering and signal correction were performed to obtain the reflection intensity and reflection time characteristics of lasers of different wavelengths. Through the above steps, the department extracted rich feature information from the original multimodal data.

[0117] Based on feature extraction, the department designed a feature fusion method with adjustable weights. Specifically, the following feature fusion function is defined:

[0118] ;

[0119] in, is the image feature, is the infrared spectrum feature, is the laser reflection characteristic, and is an adjustable weight coefficient, satisfying By optimizing these weight coefficients, the department is able to maximize the fusion feature The ability to distinguish samples.

[0120] Figure 2 The optimization process of the three feature weights in the multimodal feature fusion process is described. The red line in the figure represents the image feature weight, the green line represents the infrared spectrum feature weight, and the blue line represents the laser reflection feature weight. As the number of iterations increases, the weights of the three features gradually converge to the optimal value. The final image feature weight is about 0.4, the infrared spectrum feature weight is about 0.35, and the laser reflection feature weight is about 0.25. Figure 3 It is a three-dimensional scatter plot that shows the distribution of solid waste samples in the three-dimensional feature space. Points of different colors represent different types of solid waste. It can be seen from the figure that the samples form relatively scattered clusters in the feature space, indicating that the extracted features have good discriminability.

[0121] With the fusion features, the department began to train the solid waste identification model. First, the fusion features and manual annotations corresponding to the data with the highest resolution group [0.1 meters, 1000 bands, 0.5 nanometers] were used as the training set, and a deep convolutional neural network was trained as the teacher model. The teacher model includes multiple convolutional layers, pooling layers, residual connection feature extraction modules, attention mechanism modules, and fully connected classification modules, which can efficiently extract discriminant features from multimodal data and accurately classify solid waste.

[0122] Next, the department used knowledge distillation to train student models suitable for different resolution scenarios. Specifically, in descending order of resolution, the teacher model of the previous resolution group was used as a reference, and the fusion features and manual annotations of the current resolution group were used to train the student model. Taking the [0.2 m, 900 band, 1 nm] resolution group as an example, the output and fusion features of the previous teacher model were used to train the student model. As a training basis, a new student model is trained. The student model includes an embedded equations module, a lightweight convolutional feature extraction module, a self-attention mechanism module, and a fully connected classification module. It effectively learns knowledge from the teacher by minimizing the cross entropy loss of the teacher-student model output and the L2 distance of the feature representation.

[0123] Through the above distillation training process, the department finally obtained a series of solid waste identification models suitable for scenarios with different resolutions. Figure 4The solid waste recognition accuracy under different resolution combinations is shown. As can be seen from the figure, as the resolution decreases (from left to right), the recognition accuracy of the model shows a downward trend. The highest resolution combination (0.1 meter / 1000 bands) achieved an accuracy of 95%, while the lowest resolution combination (10 meters / 100 bands) dropped to 72%. This shows that the resolution of the data has a significant impact on the recognition performance. Figure 5 This is a heat map showing the performance comparison of the knowledge distillation model between different resolution combinations. The horizontal axis represents the resolution combination of the test data, and the vertical axis represents the resolution combination of the training data. The darker the color, the better the performance, and the value represents the recognition accuracy. From the heat map, it can be seen that the knowledge transfer effect between similar resolution combinations is better. Figure 6 The bar chart compares the computing resource consumption of different models. The left column (blue) represents the computing time (seconds), and the right column (orange) represents the memory usage (GB). From the teacher model to each student model, the computing resource consumption gradually decreases, indicating that knowledge distillation has indeed achieved the goal of lightweighting the model.

[0124] The following is an example of a specific case:

[0125] One day, the port received a batch of goods to be inspected. The port staff first used a variety of sensor equipment to collect data on the batch of goods and obtained image data, infrared spectrum data, and laser reflection data. By preprocessing and extracting features from these data, the image features were obtained. , infrared spectral characteristics and laser reflection characteristics .

[0126] Next, the port staff determined that the cargo belonged to the [2m, 600 band, 2.5nm] resolution group based on the multimodal data resolution of the cargo. From the recognition models of each resolution group obtained through the above training, the student model corresponding to the resolution group was selected. The model includes the following modules:

[0127] 1. Embedded equation system module: used for preprocessing operations such as adjusting feature scale, noise reduction, and edge enhancement.

[0128] 2. Lightweight convolutional feature extraction module: uses efficient convolutional network structure to extract deep features.

[0129] 3. Self-attention mechanism module: Use the self-attention mechanism to focus on important features.

[0130] 4. Fully connected classification module: performs final solid waste classification.

[0131] Port staff will , and The student model was input in sequence, and after a series of feature enhancement and classification calculations, the identification result of the solid waste in the batch of goods was finally obtained. The entire identification process took only 2 minutes, greatly improving work efficiency.

[0132] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for rapid identification of solid waste in port cargo, characterized in that: The following steps are involved: S10, collecting multiple groups of multimodal data of different resolution groups of solid waste in a large area and corresponding manual annotations of the solid waste, wherein the multimodal data includes image data, infrared spectrum data, and reflection data of lasers of different colors; S20, preprocessing and feature extraction are performed on the image data, infrared spectrum data and reflection data of different color lasers to obtain image features, infrared spectrum features and reflection features; S30, fusing the image features, infrared spectrum features, and reflection features corresponding to the multimodal data of each resolution group in a fusion method with adjustable weights to obtain fused features; S40, using the fusion features corresponding to the multimodal data with the highest resolution group and the corresponding manual annotations of solid waste as a training set, training a teacher model, recorded as the highest resolution solid waste recognition model; S50, in descending order of resolution groups, starting from the multimodal data of the second highest resolution group and ending with the multimodal data of the lowest resolution group, distillation training is performed in sequence, specifically, using the solid waste recognition model corresponding to the previous resolution group as the teacher model, using the fusion features corresponding to the current resolution group and the corresponding solid waste manual annotations as the training set, training a student model, and finally obtaining a solid waste recognition model for each resolution; S60, for the port cargo to be identified, collect its multimodal data, and preprocess and extract features of the collected multimodal data to obtain image features, infrared spectrum features, and reflection features; S70. According to the multimodal data resolution group of the port cargo to be identified, select a solid waste identification model with a resolution group closest to the solid waste identification model, input the extracted features into the selected solid waste identification model, and obtain the identification result of the solid waste.

2. A method for rapid identification of solid waste from port cargo according to claim 1, characterized in that: The different resolution groups, for different modalities, specifically have image data resolutions of 10 meters to 0.1 meters, infrared spectrum data resolutions of 100 bands to 1000 bands, and reflection data resolutions of different color lasers of 5 nanometers to 0.5 nanometers. Each resolution group is represented by a vector, specifically [image resolution, infrared resolution, laser resolution].

3. A method for rapid identification of solid waste from port cargo according to claim 2, characterized in that: The different resolution groups include at least 10 sequence resolution groups, specifically [0.1 meter, 1000 band, 0.5 nanometer], [0.2 meter, 900 band, 1 nanometer], [0.5 meter, 800 band, 1.5 nanometer], [1 meter, 700 band, 2 nanometers], [2 meters, 600 band, 2.5 nanometers], [3 meters, 500 band, 3 nanometers], [4 meters, 400 band, 3.5 nanometers], [5 meters, 300 band, 4 nanometers], [7 meters, 200 band, 4.5 nanometers], and [10 meters, 100 band, 5 nanometers].

4. A method for rapid identification of solid waste from port cargo according to claim 3, characterized in that: It also includes the step of cross-validating solid waste identification models of different resolutions. Specifically, the data of one resolution group is used to test the models trained by other resolution groups to evaluate the generalization ability of the model on data of different resolutions.

5. A method for rapid identification of solid waste from port cargo according to claim 4, characterized in that: The image data specifically refers to a multispectral image including visible light bands and near-infrared bands, and is obtained by shooting with a multispectral camera; the infrared spectral data specifically refers to spectral data including short-wave infrared, medium-wave infrared and long-wave infrared bands, and is obtained by scanning with a hyperspectral imager; the reflection data of lasers of different colors specifically refers to reflection intensity data after irradiation with red, green and blue lasers, and is obtained by emitting lasers through a lidar system and receiving reflected signals.

6. A method for rapid identification of solid waste from port cargo according to claim 5, characterized in that: The image features include color features, texture features and shape features; the infrared spectrum features include spectral reflectivity, absorption peak position and intensity; the reflection features include reflection intensity and reflection time of lasers of different colors.

7. A method for rapid identification of solid waste from port cargo according to claim 6, characterized in that: The teacher model adopts a deep convolutional neural network structure, including multiple convolutional layers, pooling layers, feature extraction modules, attention mechanism modules and classification modules; the feature extraction module uses residual connections to improve feature extraction capabilities; the attention mechanism module uses a combination of spatial and channel attention; the classification module uses a fully connected layer to achieve final classification; The student model specifically includes an embedded equation group module, a feature extraction module, an attention mechanism module, and a classification module. The embedded equation group module includes a scale equation, a smoothing equation, an edge enhancement equation, and a feature fusion equation. The feature extraction module adopts a lightweight convolutional neural network structure. The attention mechanism module uses a self-attention mechanism. The classification module uses a fully connected layer with dropout. The feature extraction module is used to extract deep features, the input is the fused features, and the output is the deep features; The attention mechanism module is used to focus on important features, the input is deep features, and the output is weighted features; The classification module is used to classify solid waste, the input is the weighted features, and the output is the classification result.

8. A method for rapid identification of solid waste from port cargo according to claim 7, characterized in that: The scaling equation is used to adjust the scale of the feature, the input is the original feature, and the output is the feature after the scale is adjusted; The smoothing equation is used to reduce the influence of noise, the input is the scaled feature, and the output is the smoothed feature; The edge enhancement equation is used to highlight the edge of the target, the input is the smoothed feature, and the output is the edge enhanced feature; The feature fusion equation is used to fuse multimodal features, the input is the edge-enhanced feature, and the output is the fused feature.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed in a computer, they are used to execute the method for quickly identifying solid waste in port cargo as described in any one of claims 1-8.

10. A rapid identification system for solid waste in port cargo, characterized in that: It comprises the computer-readable storage medium as claimed in claim 9, as well as a multispectral camera, a hyperspectral imager, red, green and blue laser transmitters and a laser radar.

Images