Part category identification method and system based on deep learning and knowledge graph

Abstract

The invention discloses a part category identification method and system based on deep learning and a knowledge graph, and the method comprises the steps: obtaining a part picture data set, and carrying out the preprocessing and data enhancement of the data set; a recognition model is constructed and trained, the trained model is used to perform feature extraction on the part pictures and store the features, and feature vectors contained in different part pictures are obtained; feature information is extracted from a part picture needing to be recognized through a model, and the feature information is compared with feature vectors of existing pictures to obtain a most similar result; and automatically searching corresponding information in the knowledge graph by using the query statement according to a comparison result, and feeding back the information. The detailed information is obtained from the knowledge graph by identifying the part picture, and the method can play a role in the aspects of equipment operation and maintenance and the like. According to the method, a large amount of required information is rapidly fed back according to a single picture, the information retrieval speed under the scenes of equipment maintenance, part replacement, automatic overhaul and the like is effectively increased, and the method has good reference significance for improving the part information query efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of equipment operation and maintenance, knowledge graphs and deep learning, and relates to a part category recognition method and system based on deep learning and knowledge graphs. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, higher requirements are placed on the identification and management of parts. Traditional methods of part identification often rely on manual inspection or simple image processing technology, which has limitations in efficiency and accuracy. In order to improve the efficiency and accuracy of part identification, deep learning technology has been introduced into the field of part identification, which can achieve fast and accurate part identification by automatically extracting image features and performing intelligent matching.

[0003] In the feature extraction process of part images, neural networks can identify information such as edges, textures, shapes, etc. in the image and convert them into high-dimensional feature vectors. These feature vectors can characterize the key attributes of the parts and provide a basis for subsequent similarity comparisons. Image feature matching is the process of comparing the feature vector of the input image with the feature vectors saved in the database. By calculating the distance or similarity between the two feature vectors, the part image that is most similar to the input image can be found.

[0004] Knowledge graph is a structured semantic knowledge base that stores the relationship between entities in the form of a graph. In the part recognition system, knowledge graph can be used to store relevant information about parts, such as the name, specification, and purpose of the parts. Through knowledge graph query, relevant information can be quickly retrieved based on the characteristics or attributes of the parts, including part introduction, possible problems and solutions, etc., to provide information feedback to users.

[0005] Existing part classification and part recognition technologies usually use the Softmax layer as the last layer of the neural network to obtain classification and recognition results. This method can achieve good results when there are fewer types of parts, but its accuracy is difficult to guarantee when there are more types of parts. At the same time, this method requires special model training based on the number of part types and part image data in the data set. When there is any change in the data set or the type of part to be recognized changes, the model needs to be retrained, and the scalability is poor. In addition, the data returned by this method only contains the recognition results, lacking more detailed and structured information feedback. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art and provide a method and system for part category recognition based on deep learning and knowledge graph.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] The present invention proposes a method for identifying part categories based on deep learning and knowledge graph, comprising:

[0009] Obtaining the image of the part to be queried and the preprocessed image of the part;

[0010] Based on the VGG11 model, a part image recognition model is established, and the preprocessed part image is input into the part image recognition model to obtain a trained image recognition model;

[0011] Use the trained image recognition model to extract features from the preprocessed part images to obtain feature vectors contained in different part images;

[0012] Input the image of the part to be queried into the trained image recognition model to obtain the feature vector;

[0013] The feature vectors and feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition.

[0014] Preferably, obtaining the preprocessed part image is specifically as follows:

[0015] The part image is screened, captured, scaled and expanded in sequence to obtain a pre-processed part image.

[0016] Preferably, the part image recognition model is established based on the VGG11 model, specifically:

[0017] The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

[0018] Preferably, the part image recognition model has 5 convolution blocks, a total of 8 convolution layers and three fully connected layers. The outputs of the first two fully connected layers are spliced ​​as the input of the third fully connected layer, and the output of the last fully connected layer is the classification result during model training; each convolution block is followed by a maximum pooling layer, each convolution layer is followed by a BN layer and a ReLU activation function, each fully connected layer is followed by a ReLU activation function and a Dropout layer, and the feature vector after splicing the first two fully connected layers is taken as the extracted image feature, and the recognition result is obtained by comparing the image features when used.

[0019] Preferably, the feature vectors and feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically:

[0020] The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

[0021] The present invention proposes a part category recognition system based on deep learning and knowledge graph, comprising:

[0022] An image acquisition module, wherein the image acquisition module is used to acquire the image of the part to be queried and the preprocessed part image;

[0023] A model building module, wherein the model building module is used to establish a part image recognition model based on the VGG11 model, input the preprocessed part image into the part image recognition model, and obtain a trained image recognition model;

[0024] A feature extraction module, which is used to extract features from pre-processed part images using a trained image recognition model to obtain feature vectors contained in different part images;

[0025] An image processing module, wherein the image processing module is used to input the image of the part to be queried into a trained image recognition model to obtain a feature vector;

[0026] The data processing module is used to process the feature vectors and the feature vectors contained in different part images to obtain detailed part information and realize part category recognition.

[0027] Preferably, the part image recognition model is established based on the VGG11 model, specifically:

[0028] The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

[0029] Preferably, the feature vectors and feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically:

[0030] The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

[0031] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a part category recognition method based on deep learning and knowledge graph are implemented.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a part category recognition method based on deep learning and knowledge graph.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention proposes a method for identifying part categories based on deep learning and knowledge graph. Compared with the use of image classification models for part identification, the present invention uses neural networks to extract image features and perform feature comparison. Compared with general image classification models, the number of parts that can be identified is greatly increased; and when new part types are added, there is no need to retrain the model, and the scalability is good; the feedback of the most similar pictures is added, which can effectively generate feedback to users and facilitate manual confirmation of identification results; the use of knowledge graphs as detailed information sources increases the amount of information in the feedback results and improves the structure of the feedback information. The large amount of information required for rapid feedback based on a single picture effectively speeds up the information retrieval speed in scenarios such as equipment maintenance, part replacement, and automatic maintenance, and has a good reference significance for enterprises to improve the efficiency of part information query in various scenarios. It does not rely on model classification results, and uses feature comparison to find the most similar results; when adding new part types, it can be directly added without retraining; it has more detailed knowledge feedback; it uses the comparison results to automatically obtain knowledge feedback, and there is no need to accurately predict the part name.

[0035] The present invention proposes a part category recognition system based on deep learning and knowledge graph, which obtains part category information and realizes part category recognition by dividing the system into image acquisition module, model construction module, feature extraction module, image processing module and data processing module. The modular concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 This is a flow chart of the method for part category recognition based on deep learning and knowledge graph of the present invention.

[0038] Figure 2 It is a detailed flow chart of the identification method of the present invention.

[0039] Figure 3 This is a structural diagram of the part image recognition model based on the VGG11 model of the present invention, wherein the maxpool layer is the end of each convolutional block.

[0040] Figure 4 This is an example of knowledge feedback obtained based on image recognition results of the present invention.

[0041] Figure 5 This is a data format diagram used in the knowledge graph of the present invention.

[0042] Figure 6 This is a diagram of the part category recognition system based on deep learning and knowledge graph of the present invention.

[0043] Figure 7 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0047] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0048] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0049] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0051] The present invention proposes a method for part category recognition based on deep learning and knowledge graph, such as Figure 1 As shown, including:

[0052] S1. Obtain the image of the part to be queried and the preprocessed part image;

[0053] Get the preprocessed part image, specifically:

[0054] The part image is screened, captured, scaled and expanded in sequence to obtain a pre-processed part image.

[0055] S2. Based on the VGG11 model, a part image recognition model is established, and the preprocessed part image is input into the part image recognition model to obtain a trained image recognition model;

[0056] The part image recognition model is established based on the VGG11 model, specifically:

[0057] The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

[0058] The part image recognition model has 5 convolution blocks, a total of 8 convolution layers and three fully connected layers. The outputs of the first two fully connected layers are spliced ​​as the input of the third fully connected layer, and the output of the last fully connected layer is the classification result during model training; each convolution block is followed by a maximum pooling layer, each convolution layer is followed by a BN layer and a ReLU activation function, each fully connected layer is followed by a ReLU activation function and a Dropout layer, and the feature vector after splicing the first two fully connected layers is taken as the extracted image feature. When used, the recognition result is obtained based on the image feature comparison.

[0059] S3, using the trained image recognition model to extract features from the preprocessed part images, and obtaining feature vectors contained in different part images;

[0060] S4, inputting the image of the part to be queried into the trained image recognition model to obtain a feature vector;

[0061] S5. Process the feature vectors and feature vectors contained in different part images to obtain detailed part information and realize part category recognition.

[0062] The feature vectors and the feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically:

[0063] The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

[0064] Combine the following Figure 2 The method is described in detail:

[0065] Step 1: Data construction: Collect various parts pictures from multiple angles and models to form a data set;

[0066] The dataset is a public dataset or a dataset constructed by collecting network pictures. It is advisable to use real pictures in actual application scenarios. When constructing a dataset by collecting from the Internet or taking pictures by yourself, it is necessary to name the file name of each picture in the form of "part name + number" according to the content of the picture, or use a table to save the picture and the "part name + number" information in a one-to-one correspondence.

[0067] The construction of the knowledge graph mainly includes the creation of part entities and the creation of relationships between entities. According to the recognition requirements and information feedback needs, the entity information created mainly includes numbers, names, various attributes that require feedback, etc., while the creation of relationships is determined based on the specific connection between parts.

[0068] Step 2: Data preprocessing: Filter the data obtained in step 1, remove images that obviously do not meet the requirements, crop all images according to a fixed aspect ratio, scale them to a uniform resolution that meets the requirements of the deep learning model for images; divide the data set into training sets and test sets according to a certain ratio.

[0069] Including screening, interception, scaling, and expansion of data set images. The specific process is as follows:

[0070] Step 201: Screening. According to the model input requirements, delete the pictures that are obviously not clear enough, have distorted brightness, or are not fully displayed by the camera. Especially when the self-training model is selected, such data is noise data and will affect the training of the deep learning model.

[0071] Step 202: interception: According to the model input requirements, a suitable aspect ratio is selected, and all pictures that do not meet the uniform aspect ratio are intercepted to extract the key parts for subsequent processing.

[0072] Step 203: Scaling: According to the input requirements of the deep learning model, all images in the data set are scaled to a uniform resolution size. The present invention uses a part image data set of 224*224 pixels.

[0073] Step 204: Expand. Expand the data set by flipping, rotating, adjusting the brightness, etc. The new data obtained by this series of methods is often of lower quality than the original image. This step can be omitted if the prepared data set has sufficient data.

[0074] Step 3: Deep learning model setting: Build an image recognition model based on the VGG11 model and train the image recognition model.

[0075] Step 301. The present invention chooses to build an improved image recognition model based on VGG11. Compared with the VGG16 model, the reason for choosing VGG11 as the basis for optimization is that the number of VGG11 model parameters is much smaller than the former, which is conducive to the miniaturization of the project; and the overfitting phenomenon of VGG11 is greatly reduced compared with the former. The improvement is that a BN layer (Batch Normalization) is added after each convolution layer of the VGG11 model to reduce the overfitting phenomenon during model training; the outputs of the first two (both 1024 dimensions) of the three fully connected layers in the model are spliced ​​into 2048 dimensions and used as the input of the third fully connected layer, so as to obtain more feature information, which is more conducive to subsequent feature extraction and image recognition. The detailed structure of the model is shown in the attached Figure 3 .

[0076] Specifically, the improved model based on VGG11 has 5 convolution blocks, a total of 8 convolution layers and three fully connected layers. The outputs of the first two fully connected layers are concatenated as the input of the third fully connected layer, and the output of the last fully connected layer is the classification result during model training. Among them, each convolution block is followed by a maximum pooling layer, each convolution layer is followed by a BN layer and a ReLU activation function, and each fully connected layer is followed by a ReLU activation function and a Dropout layer.

[0077] Among them, the convolution kernel size of all convolutional layers is 3*3, and the parameter padding = 1 (that is, the number of circles of 0 padded around the data, so that there is a certain space between the element content and the border, ensuring that the size of each channel data remains unchanged after convolution); the window size of all maximum pooling layers is 2*2, and the step size is 2 (that is, the size of each edge of each channel data is halved); Dropout is a regularization technique to prevent overfitting of deep neural networks, which increases the generalization ability of the model by randomly closing some neurons during training. The principle of the ReLU activation function is as follows:

[0078]

[0079] When the image data of size 224*224*3 (224*224 pixels, 3 channels, the same applies below) passes through the first convolution block, the feature map size is 112*112*64; after the second convolution block, the size is 56*56*128; after the third convolution block, the size is 28*28*256; after the fourth convolution block, the size is 14*14*512; after the fifth convolution block, the size is 7*7*512; after the first fully connected layer, the size is 1*1*1024; after the second fully connected layer, the size is 1*1*1024; after the concatenation operation, the size is 1*1*2048; and after the last fully connected layer, the size becomes 1*1*50, and the number of channels corresponds to the number of categories in the classification task.

[0080] Step 302: Obtain model parameters. Set the same hyperparameters on the same device to train and compare VGG11, VGG11 with BN layer, and the model built in S301. Then save the model parameters for feature extraction in steps 4 and 5:

[0081] (1) Dataset division. Use the device image dataset prepared in S1 to perform data preprocessing in S2, and then divide the dataset into a training set and a test set according to a certain ratio. The model is first trained on the training set, and then debugged and improved on the test set.

[0082] (2) Model hyperparameter setting. When starting to train the model, the main training hyperparameters to be set include: Epochs, batch size, image size, learning rate, Dropout value, and the training optimizer Adam and the loss function cross entropy loss function.

[0083] Table 1. Hyperparameter configuration

[0084] Epochs 150 Batch-size 32 Img-size 224*224*3 Learning_rate 1e-4 (reduced to 0.92 times every four rounds) Dropout 0.5 Optimizer Adam Loss Function Cross Entropy Loss Function

[0085] The cross entropy loss function is defined as follows:

[0086]

[0087] Among them, n is the total number of labels, P(X i ) is the probability of the i-th class in the true distribution, Q(X i ) is the probability of the i-th category in the prediction result.

[0088] The cross entropy loss function of the Pytorch library comes with a Softmax layer to convert the result into a probability form, which is defined as:

[0089]

[0090] where y i is the output value of the ith node, and m is the number of output nodes, that is, the number of categories. The Softmax function can be used to convert the output value of multiple categories into the probability value of each possible category, indicating the possibility of belonging to each category.

[0091] The Adam optimizer adjusts the learning rate of each parameter by calculating the first-order moment estimate and the second-order moment estimate of the gradient, thereby achieving more efficient network training.

[0092] (3) Model training. The model uses the Python programming language. After epochs of training, the model converges and the performance is significantly improved. The accuracy on the test set reaches the expected level.

[0093] (4) Model comparison. After multiple rounds of sufficient training, the best results of each model were compared. The VGG11 model achieved a maximum accuracy of 71.2% in the test set, while the training set accuracy was 99.9%, showing obvious overfitting; the VGG11 with the BN layer achieved a maximum accuracy of 84.0% in the test set, while the training set accuracy was 89.9%, showing better results than the VGG11 model without the BN layer; the optimized model achieved a maximum accuracy of 89.2% in the test set, which was better than the ordinary VGG11 model.

[0094] Finally, the model parameters are saved.

[0095] Step 4: Extract features from dataset images: Use the trained model and model parameters to extract features from all images in the dataset. The specific method is as follows:

[0096] Load the trained model and parameters, remove the last fully connected layer, input the images in the dataset into the model, retain the outputs of the first two fully connected layers and concatenate them into a 2048-dimensional feature vector. Get the feature array containing the feature vector and save it in a file in npz format in the form of key-value pairs, where the key name is the file name of each image, that is, the combination of the part name and number. This completes the preparation for image recognition.

[0097] Step 5, feature comparison: For the part images that need to be queried, the model selected in S3 and the trained parameters are also used to extract features to obtain feature vectors, and the feature vectors are compared one by one with the data saved in step 4 to select the most similar images.

[0098] Use the saved model and parameters to extract features from the part image to be queried to obtain a feature vector, compare the feature vector with all the data in the npz file in turn to obtain the cosine similarity of the two, take the image corresponding to the result with the highest similarity as the search result, and output the image corresponding to the search result for manual confirmation by the user to reduce the risk of error. The cosine similarity calculation formula is as follows:

[0099]

[0100] Step 6: Query the graph: Based on the comparison results obtained in step 5, obtain information feedback about the part from the knowledge graph to support part information retrieval needs in different scenarios, such as Figure 4 shown.

[0101] According to the image name corresponding to the search result, the part name and other information are obtained. The Cypher query language is used to query the detailed information of the part in the knowledge graph of the neo4j graph database and feedback to the user. Specifically: the part name can be obtained from the key name of the search result; the knowledge graph in neo4j needs to be built in advance to save the required part information and relationship information. The data format used to build the knowledge graph is as follows: Figure 5 shown.

[0102] The present invention proposes a method for identifying part categories based on deep learning and knowledge graphs, which solves the problems of long recognition time and low recognition accuracy in the related technologies in the field of part recognition and retrieval, and realizes the feedback of relevant detailed information in combination with knowledge graphs. Compared with using an image classification model for part recognition, the present invention uses a neural network to extract image features and perform feature comparison, which greatly increases the types of parts that can be recognized compared with general image classification models; and when the data set changes (such as adding a new type of part), there is no need to retrain the model, and the scalability is good; the feedback of the most similar pictures is added, which can effectively generate feedback to the user, which is convenient for manual confirmation of the recognition results; using the knowledge graph as a source of detailed information increases the amount of information in the feedback results and improves the structure of the feedback information. Compared with directly querying the knowledge graph to obtain information, the present invention adds an image detection link, which can query the information of parts that are not known in advance, and has richer usage scenarios. It can use two methods, public data sets and self-built data sets, which are easy to expand and transplant. By screening, intercepting, scaling, and expanding the pictures, the standard unification of the model input pictures is achieved, and the data volume of the model training data set is expanded. Combining with the knowledge graph can effectively improve the density and reliability of the feedback information.

[0103] Example 2

[0104] The present invention proposes a part category recognition system based on deep learning and knowledge graph, such as Figure 6 As shown, including:

[0105] An image acquisition module, wherein the image acquisition module is used to acquire the image of the part to be queried and the preprocessed part image;

[0106] Get the preprocessed part image, specifically:

[0107] The part image is screened, captured, scaled and expanded in sequence to obtain a pre-processed part image.

[0108] A model building module, wherein the model building module is used to establish a part image recognition model based on the VGG11 model, input the preprocessed part image into the part image recognition model, and obtain a trained image recognition model;

[0109] The part image recognition model is established based on the VGG11 model, specifically:

[0110] The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

[0111] The part image recognition model has 5 convolution blocks, a total of 8 convolution layers and three fully connected layers. The outputs of the first two fully connected layers are spliced ​​as the input of the third fully connected layer, and the output of the last fully connected layer is the classification result during model training; each convolution block is followed by a maximum pooling layer, each convolution layer is followed by a BN layer and a ReLU activation function, each fully connected layer is followed by a ReLU activation function and a Dropout layer, and the feature vector after splicing the first two fully connected layers is taken as the extracted image feature. When used, the recognition result is obtained based on the image feature comparison.

[0112] A feature extraction module, which is used to extract features from pre-processed part images using a trained image recognition model to obtain feature vectors contained in different part images;

[0113] An image processing module, wherein the image processing module is used to input the image of the part to be queried into a trained image recognition model to obtain a feature vector;

[0114] The data processing module is used to process the feature vectors and the feature vectors contained in different part images to obtain detailed part information and realize part category recognition.

[0115] The feature vectors and the feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically:

[0116] The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

[0117] Example 3

[0118] See also Figure 7 As shown, the present invention also provides an electronic device 100 for a part category recognition method based on deep learning and knowledge graph; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0119] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the method for identifying part categories based on deep learning and knowledge graphs described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0120] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0121] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for identifying part categories based on deep learning and knowledge graphs, and the processor 102 may execute the plurality of instructions to implement:

[0122] Obtaining the image of the part to be queried and the preprocessed image of the part;

[0123] Based on the VGG11 model, a part image recognition model is established, and the preprocessed part image is input into the part image recognition model to obtain a trained image recognition model;

[0124] Use the trained image recognition model to extract features from the preprocessed part images to obtain feature vectors contained in different part images;

[0125] Input the image of the part to be queried into the trained image recognition model to obtain the feature vector;

[0126] The feature vectors and feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition.

[0127] Example 4

[0128] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0129] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for part category recognition based on deep learning and knowledge graph, characterized in that: include: Obtaining the image of the part to be queried and the preprocessed image of the part; Based on the VGG11 model, a part image recognition model is established, and the preprocessed part image is input into the part image recognition model to obtain a trained image recognition model; Use the trained image recognition model to extract features from the preprocessed part images to obtain feature vectors contained in different part images; Input the image of the part to be queried into the trained image recognition model to obtain the feature vector; The feature vectors and feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition.

2. The method for part category recognition based on deep learning and knowledge graph according to claim 1, characterized in that: Get the preprocessed part image, specifically: The part image is screened, captured, scaled and expanded in sequence to obtain a pre-processed part image.

3. The method for part category recognition based on deep learning and knowledge graph according to claim 1, characterized in that: The part image recognition model is established based on the VGG11 model, specifically: The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

4. The method for part category recognition based on deep learning and knowledge graph according to claim 3 is characterized in that: The part image recognition model has 5 convolution blocks, a total of 8 convolution layers and three fully connected layers. The outputs of the first two fully connected layers are spliced ​​as the input of the third fully connected layer, and the output of the last fully connected layer is the classification result during model training; each convolution block is followed by a maximum pooling layer, each convolution layer is followed by a BN layer and a ReLU activation function, each fully connected layer is followed by a ReLU activation function and a Dropout layer, and the feature vector after splicing the first two fully connected layers is taken as the extracted image feature. When used, the recognition result is obtained based on the image feature comparison.

5. The method for part category recognition based on deep learning and knowledge graph according to claim 1 is characterized in that: The feature vectors and the feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically: The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

6. A part category recognition system based on deep learning and knowledge graph, characterized in that: include: An image acquisition module, wherein the image acquisition module is used to acquire the image of the part to be queried and the preprocessed part image; A model building module, wherein the model building module is used to establish a part image recognition model based on the VGG11 model, input the preprocessed part image into the part image recognition model, and obtain a trained image recognition model; A feature extraction module, which is used to extract features from pre-processed part images using a trained image recognition model to obtain feature vectors contained in different part images; An image processing module, wherein the image processing module is used to input the image of the part to be queried into a trained image recognition model to obtain a feature vector; The data processing module is used to process the feature vectors and the feature vectors contained in different part images to obtain detailed part information and realize part category recognition.

7. The part category recognition system based on deep learning and knowledge graph according to claim 6 is characterized in that: The part image recognition model is established based on the VGG11 model, specifically: The part image recognition model adds a BN layer after each convolutional layer of the VGG11 model, concatenates the outputs of the first two of the three fully connected layers in the VGG11 model into 2048 dimensions and uses them as the input of the third fully connected layer.

8. The part category recognition system based on deep learning and knowledge graph according to claim 6 is characterized in that: The feature vectors and the feature vectors contained in different part images are processed to obtain detailed part information and realize part category recognition, specifically: The feature vector is compared with the feature vectors contained in different part images in turn to obtain the cosine similarity between the two, and the image corresponding to the result with the highest similarity is taken as the search result. Based on the search result, detailed part information is obtained from the knowledge graph.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the part category recognition method based on deep learning and knowledge graph as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the part category recognition method based on deep learning and knowledge graph as described in any one of claims 1 to 5 are implemented.

Images