A garbage classification method integrating knowledge graph and image recognition

Abstract

The present invention discloses a garbage classification method integrating knowledge graph and image recognition, including: 1. constructing a knowledge graph to capture the relationship between garbage entities and categories; 2. using an Ac automaton tree to extract the garbage entity name in the question; 3. using a TransR model to extract the text vector of the garbage entity; 4. setting a question word list, extracting the question word in the question, and classifying the question according to whether the question word is in the question word list and whether there is a garbage entity name in the question; 5. returning different answer templates for different types of questions; 6. constructing an image recognition model to obtain the vector of the garbage picture; 7. combining the text vector and the picture vector of the garbage entity with the attention mechanism to splice the corresponding vectors, and finally returning the category of the garbage after obtaining the fusion vector of the garbage entity. The present invention integrates knowledge graph and image recognition, and can realize garbage classification more stably, conveniently and efficiently.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and mainly relates to a garbage classification method integrating knowledge graph and image recognition. Background Art

[0002] With the growth of population and the improvement of living standards, the output of garbage has shown an explosive growth trend. A large amount of garbage has put tremendous pressure on the environment, not only causing pollution of air, water and soil, but also causing serious damage to the ecosystem. Garbage classification, as an important measure that can effectively reduce the amount of garbage, promote resource recycling and protect the environment, has gradually received widespread attention. The traditional manual sorting method of garbage sorting is mainly completed by manual sorting and manual operation. First, garbage collectors will collect various types of garbage from sources such as households and units according to the collection requirements of different types of garbage. The collected garbage will be sent to designated sorting places, such as garbage transfer stations or sorting centers. There, the staff will sort and screen the garbage. According to the type and nature of the garbage, the staff will use manual or mechanical means to sort the garbage into different garbage bins or processing equipment. This traditional manual sorting method requires a lot of manpower and time, and the sorting accuracy and efficiency are relatively low. In addition, since the sorting process mainly relies on manual judgment and operation, it is also prone to errors and safety hazards. With the development of science and technology and the promotion of application, modern intelligent garbage sorting systems have gradually replaced the traditional manual sorting method. However, most of these garbage classification studies are based on entities, that is, research is done on garbage bins, such as smart garbage bins, or classification systems are installed in garbage stations, such as garbage classification houses. Although these methods can improve classification efficiency and accuracy, there may be errors in identification and sorting when dealing with some special types of garbage, resulting in reduced classification accuracy, and they are still not flexible enough and are subject to regional restrictions. Summary of the invention

[0003] In view of the shortcomings of current garbage classification research methods, the present invention proposes a garbage classification method that integrates knowledge graph and image recognition, in order to realize the prediction of garbage categories through text question answering or image recognition, thereby improving the efficiency and accuracy of garbage classification.

[0004] The present invention adopts the following technical solution to solve the technical problem:

[0005] The garbage classification method integrating knowledge graph and image recognition of the present invention is characterized in that it is carried out according to the following steps:

[0006] Step 1: Build a knowledge graph question answering model:

[0007] Step 1.1, obtain the spam information set R= ,in, For the types of spam information, and includes spam entity names and spam attributes, where the spam attributes include: spam types and spam images; T is the number of spam information types; t∈[1,T] ;

[0008] Step 1.2: Collect spam information The association information between each garbage entity name and its garbage attributes, and between each garbage attribute, and the knowledge graph containing garbage entity nodes, garbage attribute nodes and association information is constructed by using prior knowledge. , and get the text data of the junk entity Text data of garbage type attributes ;

[0009] Step 1.3: Use the knowledge graph representation learning model TransR to extract the entity vector corresponding to each garbage entity , Indicates The entity vector corresponding to the garbage entity, t∈[1,T] ;

[0010] Step 1.4: and Merge into a text data set, and traverse the text data set to build the Ac automaton tree;

[0011] Step 1.5: Based on and Building a dictionary , the dictionary The key in any key-value pair in contains the name of a spam entity or the type of spam;

[0012] Step 1.6: Match the input question with the Ac automaton tree to obtain matching spam entity names / spam categories, based on words in the dictionary Find the corresponding Key-value pairs ,in, represents the key-value pair corresponding to the k-th matching word, and contains the kth matching word and its word type, k∈[1,M] ;

[0013] Step 1.7: Extract question words from the question to obtain the question words, and compare them with the set question word list. If the question words are in the question word list, and If the words matched in contain junk entity names, it means that the type of the input question is a valid question and and the type of question q form a dictionary of questions , otherwise, it means the type of the input question is an invalid question;

[0014] Step 1.8: Dictionary for the question Parse and extract After finding the junk entity name in the question, create a query statement cypher for the neo4j graph database containing the junk entity name and its type attribute by combining the question type q and the junk entity name contained in the question;

[0015] Step 1.9, execute the query statement cypher, so as to generate the answer template a corresponding to the garbage type according to the question type q;

[0016] Step 2: Construction and training of image recognition model:

[0017] Step 2.1, get the T class dimension The garbage image samples are normalized and preprocessed with data enhancement to obtain the preprocessed image sequence as training samples, denoted as ,in, represents the t-th type of preprocessed junk image samples, and , Represents the t-th type of preprocessed junk image sample The pictures, Indicates height, Indicates width, Indicates the number of channels; Represents the t-th type of preprocessed junk image sample Total number of;

[0018] Step 2.2: Build an image recognition network consisting of a backbone network and a classification network;

[0019] Step 2.2.1: The backbone network is based on the VGG16 network and contains five convolutional pooling blocks. The convolution kernel size in each convolutional pooling block is , the pooling kernel size is ;

[0020] Will Input into the backbone network for processing and output the tth class Pictures The corresponding eigenvector ;

[0021] Step 2.2.2, the classification network includes a flattening layer, two fully connected layers, a batch normalization layer, a dropout layer, and an attention mechanism layer;

[0022] Will The input is flattened into a one-dimensional vector in the flattening layer, and then input into the first fully connected layer with the activation function of relu, the batch normalization layer, and the dropout layer for processing to obtain the tth class Vector grunge pictures , thus obtaining the vector set of the t-th type of garbage pictures ;

[0023] Step 2.2.3, the attention mechanism layer with softmax activation function converts the entity vector and The text vector and the picture vector representing the same garbage are combined to obtain the fusion vector of the t-th type of garbage , Represents the t-th type of preprocessed junk image sample The vector of j-th garbage in ;

[0024] Step 2.2.4: Garbage vector Input to the second fully connected layer with Softmax activation function for processing, and obtain The predicted probability distribution of , and select the category with the highest probability as The prediction category of

[0025] Step 2.2.5: Based on The predicted category and the true category are used to construct a cross entropy loss function, and the image recognition network is optimized using the Adam optimization method, and the cross entropy loss function is calculated to update the parameters of the network until the cross entropy loss function converges, thereby obtaining an optimal garbage classification model;

[0026] Step 3, obtain the new question input by the user and process it according to steps 1.6 to 1.9 to obtain the garbage type answer of the new question;

[0027] Step 4: Input the garbage image taken by the user into the optimal garbage classification model for identification, and obtain the predicted type of the garbage image.

[0028] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the garbage classification method, and the processor is configured to execute the program stored in the memory.

[0029] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the garbage classification method when executed by a processor.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention utilizes the knowledge graph question-answering model to learn the relationship between garbage and categories, so as to more accurately infer which category the garbage asked by the user belongs to.

[0032] 2. The present invention uses the existing data set to construct an Ac automaton tree, which can accurately scan the problem and obtain the junk entities in the problem with higher accuracy.

[0033] 3. Compared with the original layer of VGG16, the present invention adds a batch normalization layer to the image recognition model. Batch normalization reduces the internal covariate shift by converting the mean of the image matrix to zero and the variance to one. This makes the learning objectives of each layer of the network more consistent, which is beneficial to the parameter optimization process, alleviates the problems of gradient vanishing and gradient exploding, and thus improves the accuracy of garbage classification.

[0034] 3. The present invention integrates the text information of garbage into the image recognition model and utilizes the multimodal information of garbage to more accurately identify garbage.

[0035] 4. The present invention can be used on computers and mobile phones, has a wide range of applications, and has practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0037] In this embodiment, a garbage classification method that integrates knowledge graph and image recognition is to construct a text question answering model by discovering the relationship between garbage and its type attributes through knowledge graph, and use the knowledge graph representation learning model transR to obtain the text vector of the garbage entity; at the same time, an image recognition model based on VGG16 is also constructed.

[0038] The input of the text question answering model consists of three parts: text data of junk entity names , text data of garbage category , and user input questions, using and Construct an Ac automaton tree and send the question into the Ac automaton tree for matching to extract the junk entity names in the question.

[0039] Create a list of question words, traverse the questions to determine whether they contain corresponding question words, classify the questions according to whether they contain corresponding junk entity names and question words, and create a query statement cypher for the neo4j graph database containing the junk entity name and its type attributes according to different question types, and execute the query statement cypher to generate an answer template corresponding to the junk type according to the question type.

[0040] After preprocessing the image such as data enhancement, the image is sent to the image recognition model to train the model and generate the vector of the garbage image. The text vector and image vector of the garbage obtained above are connected using the attention mechanism to represent the two vectors of the same garbage, and the multimodal information is used to achieve more accurate recognition of garbage. The Adam optimization function and the cross entropy loss function are selected to obtain the optimal model.

[0041] Specifically, if Figure 1 As shown, the method is performed in the following steps:

[0042] Step 1: Build a knowledge graph question answering model:

[0043] Step 1.1, obtain the spam information set R= ,in, For the types of spam information, and includes spam entity names and spam attributes, where spam attributes include: spam types and spam images; The form is T is the number of types of spam information, t ∈[1, T ] ;

[0044] Step 1.2: Collect spam information The association information between each garbage entity name and its garbage attributes, and between each garbage attribute, and the knowledge graph containing garbage entity nodes, garbage attribute nodes and association information is constructed by using prior knowledge. , and get the text data of the junk entity Text data of garbage type attributes ;

[0045] Step 1.3: Use the knowledge graph representation learning model TransR to extract the entity vector corresponding to each garbage entity , Indicates The entity vector corresponding to the garbage entity, t ∈[1, T ] ;

[0046] Step 1.4: and Merge into a text data set, and traverse the text data set to build the Ac automaton tree;

[0047] Step 1.5: Based on and Building a dictionary ,dictionary The key in any key-value pair contains the garbage entity name or garbage category; for example: {'plastic bottle': ['rubbish'],...,'recyclable garbage': [''category]}.

[0048] Step 1.6: Match the input question with the Ac automaton tree to obtain matching spam entity names / spam categories, based on words in the dictionary Find the corresponding Key-value pairs ,in, represents the key-value pair corresponding to the k-th matching word, and contains the kth matching word and its word type, k ∈[1, M ] ;

[0049] Step 1.7: Extract question words from the question, obtain the question words of the question, and compare them with the set question word list. If the question words of the question are in the question word list, and If the words matched in contain junk entity names, it means that the type of the input question is a valid question and and the type of question q form a dictionary of questions , otherwise, it means the type of the input question is an invalid question;

[0050] Step 1.8: Dictionary for the question Parse and extract After finding the junk entity name in the question, create a query statement cypher for the neo4j graph database containing the junk entity name and its type attribute by combining the question type q and the junk entity name contained in the question;

[0051] Step 1.9: Execute the query statement cypher to generate an answer template a corresponding to the type of garbage according to the question type q.

[0052] Step 2: Construction and training of image recognition model:

[0053] Step 2.1, get the T class dimension The garbage image samples are normalized and preprocessed with data enhancement to obtain the preprocessed image sequence as training samples, denoted as ,in, represents the t-th type of preprocessed junk image samples, and , Represents the t-th type of preprocessed junk image sample The pictures, Indicates height, Indicates width, Indicates the number of channels; Represents the t-th type of preprocessed junk image sample The total number of .

[0054] Step 2.2: Build an image recognition network consisting of a backbone network and a classification network;

[0055] Step 2.2.1: The backbone network is based on the VGG16 network and contains five convolutional pooling blocks. The convolution kernel size in each convolutional pooling block is , the pooling kernel size is ;

[0056] Will Input into the backbone network for processing and output the tth class Pictures The corresponding eigenvector ;

[0057] Step 2.2.2, the classification network includes a flattening layer, two fully connected layers, a batch normalization layer, a dropout layer, and an attention mechanism layer;

[0058] Will The input is flattened into a one-dimensional vector in the flattening layer, and then input into the first fully connected layer with the activation function of relu, the batch normalization layer, and the dropout layer for processing to obtain the tth class Vector grunge pictures , thus obtaining the vector set of the t-th type of garbage pictures .

[0059] Step 2.2.3, the attention mechanism layer with softmax activation function converts the entity vector and The text vector and picture vector representing the same garbage in are combined, so as to obtain the t-th type of preprocessed garbage picture sample using formula (1): The vector of j-th garbage in , thus obtaining the t-th type of garbage fusion vector :

[0060]

[0061] In formula (1), express , and have:

[0062]

[0063] In formula (1), express The transposed vector of .

[0064] Step 2.2.4: Garbage vector Input to the second fully connected layer with Softmax activation function for processing, and obtain The predicted probability distribution of , and select the category with the highest probability as The prediction category of

[0065] Step 2.2.5: Based on The predicted category and the true category are used to construct the cross entropy loss function L using formula (3), and the image recognition network is trained using the Adam optimization method. The cross entropy loss function L is calculated to update the network parameters until the cross entropy loss function converges, thereby obtaining the optimal garbage classification model.

[0066]

[0067] In formula (3), is the junk image sample after preprocessing of the tth category The true label value of the jth garbage in is the junk image sample after preprocessing of the tth category The predicted value of the j-th garbage in .

[0068] Step 3, obtain the new question input by the user and process it according to steps 1.6 to 1.9 to obtain the garbage type answer of the new question;

[0069] Step 4: Input the garbage image taken by the user into the optimal garbage classification model for identification, and obtain the predicted type of the garbage image.

[0070] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0071] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0072] In summary, the present invention uses the knowledge graph question-answering model and the Ac automaton tree to extract junk entities in the question, so that the question can be better parsed and the corresponding query statement can be established to generate the answer and return it to the user. In building the image recognition model, the multimodal information of the junk is used to connect the text vector and the image vector to enrich the junk information and improve the recognition accuracy. In addition, the cross entropy loss function is also considered to be established. When the prediction result is inconsistent with the label value, a larger error signal will be given, so that the model pays more attention to the misclassified samples. It helps to improve the accuracy of the model and train it using the Adam optimizer so that the model can achieve better classification results.

Claims

1. A garbage classification method integrating knowledge graph and image recognition, characterized in that: The steps are as follows: Step 1: Build a knowledge graph question answering model: Step 1.1, obtain the spam information set R= ,in, For the types of spam information, and includes spam entity names and spam attributes, where the spam attributes include: spam types and spam images; T is the number of spam information types; ; Step 1.2: Collect spam information The association information between each garbage entity name and its garbage attributes, and between each garbage attribute, and the knowledge graph containing garbage entity nodes, garbage attribute nodes and association information is constructed by using prior knowledge. , and get the text data of the junk entity Text data of garbage type attributes ; Step 1.3: Use the knowledge graph representation learning model TransR to extract the entity vector corresponding to each garbage entity , Indicates The entity vector corresponding to the garbage entity, ; Step 1.4: and Merge into a text data set, and traverse the text data set to build the Ac automaton tree; Step 1.5: Based on and Building a dictionary , the dictionary The key in any key-value pair in contains the name of a spam entity or the type of spam; Step 1.6: Match the input question with the Ac automaton tree to obtain matching spam entity names / spam categories, based on words in the dictionary Find the corresponding Key-value pairs ,in, represents the key-value pair corresponding to the k-th matching word, and contains the kth matching word and its word type, ; Step 1.7: Extract question words from the question to obtain the question words, and compare them with the set question word list. If the question words are in the question word list, and If the words matched in contain junk entity names, it means that the type of the input question is a valid question and and the type of question q form a dictionary of questions , otherwise, it means the type of the input question is an invalid question; Step 1.8: Dictionary for the question Parse and extract After finding the junk entity name in the question, create a query statement cypher for the neo4j graph database containing the junk entity name and its type attribute by combining the question type q and the junk entity name contained in the question; Step 1.9, execute the query statement cypher, so as to generate the answer template a corresponding to the garbage type according to the question type q; Step 2: Construction and training of image recognition model: Step 2.1, get the T class dimension The garbage image samples are normalized and preprocessed with data enhancement to obtain the preprocessed image sequence as training samples, denoted as ,in, represents the t-th type of preprocessed junk image samples, and , Represents the t-th type of preprocessed junk image sample The pictures, Indicates height, Indicates width, Indicates the number of channels; Represents the t-th type of preprocessed junk image sample Total number of; Step 2.2: Build an image recognition network consisting of a backbone network and a classification network; Step 2.2.1: The backbone network is based on the VGG16 network and contains five convolutional pooling blocks. The convolution kernel size in each convolutional pooling block is , the pooling kernel size is ; Will Input into the backbone network for processing and output the tth class Pictures The corresponding eigenvector ; Step 2.2.2, the classification network includes a flattening layer, two fully connected layers, a batch normalization layer, a dropout layer, and an attention mechanism layer; Will The input is flattened into a one-dimensional vector in the flattening layer, and then input into the first fully connected layer with the activation function of relu, the batch normalization layer, and the dropout layer for processing to obtain the tth class Vector grunge pictures , thus obtaining the vector set of the t-th type of garbage pictures ; Step 2.2.3, the attention mechanism layer with softmax activation function converts the entity vector and The text vector and the picture vector representing the same garbage are combined to obtain the fusion vector of the t-th type of garbage , Represents the t-th type of preprocessed junk image sample The vector of j-th garbage in ; Step 2.2.4: Garbage vector Input to the second fully connected layer with Softmax activation function for processing, and obtain The predicted probability distribution of , and select the category with the highest probability as The prediction category of Step 2.2.5: Based on The predicted category and the true category are used to construct a cross entropy loss function, and the image recognition network is optimized using the Adam optimization method, and the cross entropy loss function is calculated to update the parameters of the network until the cross entropy loss function converges, thereby obtaining an optimal garbage classification model; Step 3, obtain the new question input by the user and process it according to steps 1.6 to 1.9 to obtain the garbage type answer of the new question; Step 4: Input the garbage image taken by the user into the optimal garbage classification model for identification, and obtain the predicted type of the garbage image.

2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the garbage classification method according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the garbage sorting method according to claim 1 are executed.

Images