Intelligent sorting method and system based on multi-source data

Abstract

The present invention provides an intelligent sorting method and system based on multi-source data, which relates to the technical field of waste sorting, and includes: collecting waste photographed images, waste text descriptions and classification results in the historical waste sorting process; obtaining image features through waste photographed images; obtaining text sequence features through waste text descriptions; inputting image features and text sequence features into a generative network to obtain first text information and second text information; calculating losses between image features and text sequence features, between first text information and second text information, and between second text information and waste text descriptions, and training a model; constructing a decision tree; and obtaining sorting results by matching image-generated text information with a decision tree in a real-time sorting process; the present invention utilizes image recognition and natural language processing technologies to achieve rapid and accurate sorting of waste, thereby greatly improving processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of article sorting, and also to the fields of natural language processing, image recognition and data processing, and in particular to an intelligent sorting method and system based on multi-source data. Background Art

[0002] In traditional waste sorting systems, manual selection or simple mechanical classification methods are usually used. These methods have some disadvantages, such as low efficiency, low accuracy, and reliance on manual experience. With the development of artificial intelligence technology, intelligent sorting systems based on image recognition have become a new solution. Traditional image recognition systems mainly rely on manually designed feature extractors, but these methods often require a lot of domain expert knowledge and manually labeled data. In contrast, image feature extraction methods based on convolutional neural networks (CNNs) can automatically learn image features, reduce dependence on artificial features, and improve recognition accuracy and robustness. However, image information alone may not be able to fully describe the characteristics of items. Therefore, it has become a necessary step to incorporate the text description corresponding to the waste into the sorting system. Using natural language processing technology to process the text description of waste can better understand the characteristics of the waste, and combine it with image information, thereby improving the sorting system's ability to understand and accuracy of the waste. In addition, the introduction of generative networks further enriches the expression of waste information. By generating the first text information and the second text information, the characteristics of the waste can be expressed in a richer way, providing more reference for subsequent classification and selection. Finally, in the real-time process, the loss function is used for model training, and combined with database query technology, the intelligent sorting system can quickly and accurately classify and select waste products according to the text description of the waste, improving the sorting efficiency and accuracy and reducing labor costs and error rates. Summary of the invention

[0003] The present invention provides an intelligent sorting method based on multi-source data, aiming to solve the problems of low efficiency and low accuracy in traditional waste sorting systems.

[0004] The present invention is achieved through the following technical solutions:

[0005] An intelligent sorting method based on multi-source data comprises the following steps:

[0006] Build and train the sorting model in advance. This step includes the following steps:

[0007] S1. Collect the waste images, waste text descriptions and classification results during the historical waste sorting process;

[0008] S2. The feature extraction network extracts features from the waste images to obtain image features;

[0009] S3. Performing natural language processing on the waste text description through a natural language processing model to obtain text sequence features;

[0010] S4. The image features are input into the generative network to obtain the first text information; the text sequence features are input into the generative network to obtain the second text information;

[0011] S5. Perform a first loss calculation on the image features and the text sequence features to constrain the image features generated by the feature extraction network; perform a second loss calculation between the first text information and the second text information, and perform a third loss calculation between the second text information and the waste text description to constrain the consistency of the text description between the first text information and the second text information;

[0012] S6. Use the constraints of the first loss, the second loss, and the third loss to train the entire sorting model, that is, train the feature extraction network, the generative network, and the natural language processing model; construct a selection criterion decision tree based on the waste text description and the corresponding classification results;

[0013] During the real-time sorting process, images of waste products are collected, and then a generated text representation is obtained through the trained feature extraction network and generative network. The generated text representation here is the first text information. The generated text representation is matched with the selection criterion decision tree to obtain a real-time classification result, and sorting is performed based on this real-time classification result.

[0014] Preferably, the photographed image of waste is a color image, and the text description of the waste includes size information, property information, material information, and color information.

[0015] Specifically, the feature extraction network extracts features from the waste image to obtain image features, including:

[0016] Taking images of waste Perform standardization to obtain a standardized image ; Standardize the image Input feature extraction network to get image features .

[0017] Preferably, the feature extraction network includes N convolutional layers, one pooling layer and one Sigmoid layer, from the standardized image Get image features The process is as follows:

[0018] S21. Standardized image Perform N layers of convolution operations to obtain convolution image features ;

[0019] S22. Convolutional image features Perform pooling layer operation to obtain pooling features ;

[0020] S23. Pooling features Perform Sigmoid operation to obtain nonlinear features , which is expressed as follows,

[0021] ;

[0022] S24. For nonlinear features To compress the feature dimension, first convert the nonlinear features The process dimension is expanded, the features are concatenated according to the first dimension, converted into a one-dimensional vector, and then the data representation dimension is further compressed through full connection to realize the vector representation of image features and obtain image features. , expressed as:

[0023] ;

[0024] in Indicates that the multi-dimensional vector is vectorized and expanded. represents the weight matrix and b represents the bias.

[0025] Furthermore, the text features obtained by performing natural language processing on the waste text description in S3 include:

[0026] S31. Perform text preprocessing on the waste text description and define a text Tokenizer;

[0027] S32. Use the vocabulary provided by the Bert pre-trained model to convert the text Tokenizer into a Token sequence to obtain a text sequence ;

[0028] S33. Sequence the text Through the fully connected neural network, feature extraction is performed to convert the text sequence Convert to image features The dimension size of the text sequence feature is obtained .

[0029] Furthermore, the step of inputting the image features into the generative network to obtain the first text information in S4 includes:

[0030] S41. Image features Input the generative network, and use the generative network to transform the image features Convert to text sequence The scale size of the first text information is obtained ;

[0031] The text sequence feature is input into the generative network in S4 to obtain the second text information, which includes:

[0032] S42. Text sequence features Transformed into text sequence through generative network The scale size of the second text information is obtained .

[0033] Furthermore, the generative network is a UNet network or a VNet network.

[0034] Furthermore, in S5, a first loss calculation is performed on the image features and the text sequence features, and the loss function used is MSE loss, specifically:

[0035] ;

[0036] Where N is the number of samples, Represents image features, Represents text sequence features.

[0037] Furthermore, the first text information generated in S5 , Second text message The loss function used is KL divergence distribution, including the first text information With the second text message The second loss L2 between the second text information and the text sequence The third loss L3 between , the corresponding specific loss is:

[0038] ;

[0039] ;

[0040] in, is the activation function, and N is the number of samples.

[0041] At the same time, the present invention also discloses an intelligent sorting system based on multi-source data, the system is used to sort waste products, and the sorting method adopted in the system is the above-mentioned intelligent sorting method based on multi-source data.

[0042] By adopting the above technical solution, the present invention has the following advantages:

[0043] First, the use of image recognition and natural language processing technology can achieve fast and accurate sorting of waste, greatly improving processing efficiency.

[0044] Secondly, the image features are extracted through convolutional neural networks and combined with natural language processing to generate text features, which effectively improves the accuracy of waste recognition.

[0045] In addition, the system can automatically process waste images and text descriptions without manual intervention, saving human resource costs.

[0046] Finally, the image features and text features are comprehensively considered, so that the sorting system can understand the properties and characteristics of the waste more comprehensively. These advantages make the scheme have high practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention or the prior art description. It should be understood that the following drawings are only some embodiments of the present invention and should not be regarded as limiting the scope. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a main process flow chart of the present invention;

[0049] Figure 2 It is the overall principle block diagram of the present invention;

[0050] Figure 3 Construct a diagram for a decision tree. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] like Figure 1 As shown, this embodiment provides an intelligent sorting method based on multi-source data, comprising the following steps:

[0053] An intelligent sorting method based on multi-source data includes a model pre-construction and training process, and a real-time sorting process. The principle block diagram of the present invention is as follows: Figure 2 The advance construction and training process includes the following steps:

[0054] S1. Collect the waste images, waste text descriptions and classification results in the historical waste sorting process; the images in the historical data are color images and multi-angle images; the waste text description includes size information, property information, material information, etc. More specifically, for example, material information can be divided into metal and non-metal, and non-metal can be divided into plastic, wood, paper, glass, cloth, etc.; property information can be divided into harmful and harmless, and harmful can be divided into some specific ones, such as batteries, fluorescent tubes, etc.; size information can be divided into large, medium, and small pieces, or more specifically, size division, less than 10cm, greater than or equal to 10cm and less than or equal to 50cm, greater than or equal to 50cm centimeters or 100cm, greater than 100cm. The waste text description can also include color information and shape information, and the size information can also be replaced by volume information. Of course, the division here is just an example. In the actual sorting process, the manufacturer can set it in advance according to the specific sorting waste. In the process of data set preparation, these waste text descriptions and corresponding classification results need to be manually labeled. The specific classification result can be the final waste category or a number. Different numbers correspond to different sorting ports, which is determined according to the practical scenario of the actual user. Examples of text information and classification results are shown in Table 1. Table 1 shows a simple example;

[0055] Table 1 Example table of text information and classification results:

[0056] .

[0057] S2. The feature extraction network extracts features from the waste images to obtain image features; including:

[0058] Taking images of waste Perform standardization to obtain a standardized image ; Specifically, calculate the mean value of image pixels and standard deviation , the image pixel information is scaled according to the mean and standard deviation to obtain the standardized waste image ; Standardize the image Input feature extraction network to get image features The feature extraction network includes N convolutional layers, a pooling layer and a Sigmoid layer. Get image features The process is as follows:

[0059] S21. Standardized image Perform N layers of convolution operations to capture the high-dimensional features of the waste image and obtain the convolution image features. , for the standardized image The data is convolved to obtain the convolution image features. , which is expressed as follows:

[0060] ;

[0061] in is the value of the convolution kernel, l is the number of network layers for the convolution operation, w is the convolution step size on the X axis, and h is the convolution step size on the Y axis. and Represents the length and width of the image respectively.

[0062] S23. Convolution image Perform pooling layer operation, which can reduce the amount of calculation and enhance the role of effective information. After the pooling operation, the pooling features are obtained ;

[0063] S23. Pooling features Perform Sigmoid operation to obtain nonlinear features It is expressed as follows:

[0064] ;

[0065] S24. For nonlinear features To compress the feature dimension, first convert the nonlinear features The process dimension is expanded, the features are concatenated according to the first dimension, converted into a one-dimensional vector, and then the data representation dimension is further compressed through full connection to realize the vector representation of image features and obtain image features. , expressed as:

[0066] ;

[0067] in Indicates that the multi-dimensional vector is vectorized and expanded. represents the weight matrix and b represents the bias.

[0068] S3. Perform natural language processing on the waste text description through a natural language processing model to obtain text sequence features. Specifically,

[0069] S31. Perform text preprocessing on the waste text description and define a text Tokenizer; specifically, perform character-level extraction on the Chinese text using the BertTokenizer method in the Transformers package, and separate each word and punctuation mark. The Transformers package is available.

[0070] S32. Use the vocabulary provided by the Bert pre-training model to convert the text Tokenizer into a Token sequence of a certain length of text, and use 0 values ​​to fill the length of the Token sequence that is less than the length to obtain the text sequence ;

[0071] S33. Sequence the text Through the fully connected neural network, feature extraction is performed to convert the text sequence Convert to image features The dimension size of the text sequence feature is obtained .

[0072] S4. Inputting image features into a generative network to obtain first text information; inputting text sequence features into a generative network to obtain second text information; comprising:

[0073] S41. Image features Input the generative network, and use the generative network to transform the image features Convert to text sequence The scale size of the first text information is obtained ;

[0074] S42. Text sequence features Transformed into text sequence through generative network The scale size of the second text information is obtained The order of S41 and S42 can be swapped.

[0075] The generative network is a UNet network or a VNet network. Specifically, the generative network here can be other variant structures of the Unet network, such as UNet+, UNet++, TranUNet. Since the original UNet itself processes two-dimensional data, in the specific implementation of the present invention, it is necessary to adjust the input dimension and output dimension of the generative network. After adjusting from one dimension to two dimensions, it is adjusted to input into the generative network again. The output of the generative network is two-dimensional data. Finally, the two-dimensional data is adjusted to one dimension, and the first text information is obtained. Similarly, the image features After adjusting from one dimension to two dimensions, it is adjusted to input into the generative network again. The output of the generative network is two-dimensional data. Finally, the two-dimensional data is adjusted to one dimension, and the second text information is obtained. In another specific implementation, the generative network may also use multiple fully connected layers (such as 5 layers). If the fully connected layers are used as the generative network, there is no need to adjust the dimensions of the input and output of the generative network.

[0076] S5. Perform a first loss calculation on the image features and the text sequence features to constrain the image features generated by the feature extraction network; perform a second loss calculation on the first text information and the second text information, perform a loss constraint on the second text information and the waste text description, and constrain the consistency of the text description between the first text information and the second text information. Specifically, the loss function used in the first loss calculation is the MSE loss, which is:

[0077] ;

[0078] Where N is the number of samples.

[0079] The second loss is as follows:

[0080] ;

[0081] The third loss is specifically:

[0082] ;

[0083] in, is the activation function, and N is the number of samples.

[0084] S6. The entire sorting model is trained using the constraints of the first loss, the second loss, and the third loss, that is, the feature extraction network, the generative network, and the natural language processing model are trained, and the waste text description and the corresponding classification results are selected to construct a criterion database. A decision tree is a tree structure. The first node in the tree contains the full set of samples. Each node in the tree structure represents a test on an attribute, each branch represents a test output, and each leaf node represents a category. The construction of a decision tree includes data preprocessing, trial modeling limits, parameter tuning, post-pruning, and model extraction. The construction of a decision tree can use an existing machine learning library to construct a decision tree, such as Python's scikit-learn library. Therefore, the specific decision tree construction process is not elaborated in detail in this invention, which is a prior art. The present invention establishes a decision tree algorithm based on the size information, property information, material information, color information, etc. described above for information comparison. In actual operation, the hierarchy and classification results of the specific decision tree are adjusted to the relevant information according to actual needs. For example, the waste can be divided into solid and non-solid according to the material of the waste, and the subsequent layer feature division is performed based on the feature selection of the first layer. Finally, the division standards of different layers are used as the basis for intelligent selection to confirm the selection basis, and the selection criterion database is constructed according to the corresponding relationships under different descriptions. Figure 3 A simple decision tree of an embodiment of the present invention is shown. The decision tree here is just a simple example, which is intended to provide a better understanding of the present invention. The decision tree constructed in a specific scenario may be more complex than this decision tree.

[0085] During the real-time sorting process, images of waste products are collected, and then the generated text representation is obtained through the trained feature extraction network and generative network. The generated text representation is matched with the selection criteria database through a decision tree to obtain real-time classification results, and sorting is performed based on this real-time classification result.

[0086] Through the above steps, the intelligent sorting system of the present invention can make full use of image recognition and natural language processing technology to achieve a comprehensive understanding and accurate classification of waste, greatly improve sorting efficiency and accuracy, and reduce labor costs and error rates.

[0087] It should be understood that the premise of the present invention for waste sorting is that most of the waste to be sorted is recyclable waste and there is no liquid or fluid waste, which greatly reduces the categories of classification. The key point of sorting is whether the waste can be accurately classified. The existing method is through manual screening, which is inefficient and prone to errors. In order to reduce costs and improve efficiency, the present invention provides an intelligent sorting method and system based on multi-source data. The present invention classifies texts by means of decision trees. Decision tree classification is suitable for such classifications with relatively simple classifications or fewer categories. The key point of the final classification accuracy lies in how to improve the accuracy of the obtained text. In the real-time sorting process, it is fast to obtain images of sorted waste, and text can be quickly obtained through images, thereby improving the efficiency of sorting. Moreover, the present invention trains the overall model through the first loss, the second loss and the third loss, and associates the image and the text to obtain more accurate text information, thereby improving the accuracy of sorting.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent sorting method based on multi-source data, characterized in that: Build and train the sorting model in advance, including the following steps: S1. Collect the waste images, waste text descriptions and classification results during the historical waste sorting process; S2. The feature extraction network extracts features from the waste images to obtain image features; S3. Performing natural language processing on the waste text description through a natural language processing model to obtain text sequence features; S4. The image features are input into the generative network to obtain the first text information; the text sequence features are input into the generative network to obtain the second text information; S5. Perform a first loss calculation on the image features and the text sequence features to constrain the image features generated by the feature extraction network; perform a second loss calculation between the first text information and the second text information, and perform a third loss calculation between the second text information and the waste text description to constrain the consistency of the text description between the first text information and the second text information; S6. Use the constraints of the first loss, the second loss, and the third loss to train the entire sorting model; construct a selection criterion decision tree based on the waste text description and the corresponding classification results; During the real-time sorting process, images of waste products are collected, and then a generated text representation is obtained through the trained feature extraction network and generative network. The generated text representation here is the first text information. The generated text representation is matched with the selection criterion decision tree to obtain a real-time classification result, and sorting is performed based on this real-time classification result.

2. The intelligent sorting method based on multi-source data according to claim 1 is characterized in that: The photographed image of the waste is a color image, and the text description of the waste includes size information, property information, material information, and color information.

3. The intelligent sorting method based on multi-source data according to claim 1, characterized in that: The feature extraction network extracts features from the waste image to obtain image features, including: Taking images of waste Perform standardization to obtain a standardized image ; Standardize the image Input feature extraction network to get image features .

4. The intelligent sorting method based on multi-source data according to claim 3 is characterized in that: The feature extraction network includes N convolutional layers, one pooling layer and one Sigmoid layer. Get image features The process is as follows: S21. Standardized image Perform N layers of convolution operations to obtain convolution image features ; S22. Convolutional image features Perform pooling layer operation to obtain pooling features ; S23. Pooling features Perform Sigmoid operation to obtain nonlinear features , which is expressed as follows, ； S24. For nonlinear features To compress the feature dimension, first convert the nonlinear features The process dimension is expanded, the features are concatenated according to the first dimension, converted into a one-dimensional vector, and then the data representation dimension is further compressed through full connection to realize the vector representation of image features and obtain image features. , expressed as: ； in Indicates that the multi-dimensional vector is vectorized and expanded. represents the weight matrix and b represents the bias.

5. The intelligent sorting method based on multi-source data according to claim 1, characterized in that: In S3, the text sequence features obtained by performing natural language processing on the waste text description through the natural language processing model include: S31. Perform text preprocessing on the waste text description and define a text Tokenizer; S32. Use the vocabulary provided by the Bert pre-trained model to convert the text Tokenizer into a Token sequence to obtain a text sequence ; S33. Sequence the text Through the fully connected neural network, feature extraction is performed to convert the text sequence Convert to image features The dimension size of the text sequence feature is obtained .

6. The intelligent sorting method based on multi-source data according to claim 1, characterized in that: The image feature is input into the generative network in S4 to obtain the first text information, which includes: S41. Image features Input the generative network, and use the generative network to transform the image features Convert to text sequence The scale size of the first text information is obtained ; The text sequence feature is input into the generative network in S4 to obtain the second text information, which includes: S42. Text sequence features Transformed into text sequence through generative network The scale size of the second text information is obtained .

7. The intelligent sorting method based on multi-source data according to claim 6 is characterized in that: The generative network is a UNet network or a VNet network.

8. The intelligent sorting method based on multi-source data according to claim 5, characterized in that: In S5, the first loss calculation is performed on the image features and the text sequence features, and the loss function used is the MSE loss, specifically: ； Where N is the number of samples, Represents image features, Represents text sequence features.

9. The intelligent sorting method based on multi-source data according to claim 1, characterized in that: The first text information generated in S5 , Second text message The loss function used is KL divergence distribution, including the first text information With the second text message The second loss L2 between the second text information and the text sequence The third loss L3 between , the corresponding specific loss is: ； ； in, is the activation function and N is the number of samples.

10. An intelligent sorting system based on multi-source data, characterized in that: The system is used to sort waste products, and the sorting method adopted in the system is the intelligent sorting method based on multi-source data as described in any one of claims 1 to 9.

Images