Patent image retrieval system, method and apparatus based on deep learning

Abstract

A patent image retrieval method, apparatus, device and system based on deep learning. The retrieval system comprises: a storage module (101), a processing module (102), a feature extraction module (103), and a calculation module (104). The storage module is used for storing a patent image dataset by means of a HDFS, and storing a patent text dataset by means of a MongoDB, wherein the patent image dataset comprises the drawings in a patent, and the patent text dataset comprises text data in the patent other than the drawings. The processing module is used for allocating the patent image dataset and the patent text dataset to the feature extraction module by means of a load balancing strategy. The feature extraction module is used for extracting a patent drawing feature vector of the patent image dataset and a patent text feature vector of the patent text dataset on the basis of an OCR algorithm, a ResNet model, and the NLP technology, and on the basis of the OCR algorithm, the ResNet model, and the NLP technology, extracting a text feature vector and an image feature vector of an image to be retrieved uploaded by a user. The calculation module is used for calculating the similarity between the image to be retrieved and the patent in the storage module on the basis of the patent drawing feature vector, the patent text feature vector, the text feature vector, and the image feature vector.

Description

Patent image retrieval system, method and device based on deep learning

[0001] This application claims priority to the Chinese patent application filed with the Patent Office of China on December 28, 2023, with application number 2023118429152 and invention name “Patent image retrieval system, method and device based on deep learning”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] This document relates to the field of image processing, and in particular to a patent image retrieval system, method and device based on deep learning. Background Art

[0003] With the rapid development of the information age, the number of patent applications is increasing day by day, and the government and enterprises are paying more and more attention to the patent application situation. In order to obtain patent rights, enterprises first need to determine whether their inventions are novel, and use the patent search system to find out whether there are similar patents, thereby ensuring the uniqueness of the patent.

[0004] Patent search systems usually match related patents in the form of keywords, and you need to have a very good understanding of patents to accurately find keywords.

[0005] However, when searching for similar patents, searchers often need to rely on keyword searches. Since most keywords contain multiple meanings, the search results are prone to generate a lot of noise, resulting in low patent search efficiency and inaccurate searches. Summary of the Invention

[0006] In view of the above solutions, this application aims to propose a patent image retrieval system, method and device based on deep learning to solve at least one of the above technical problems.

[0007] In a first aspect, one or more embodiments of this specification provide a patent image retrieval system based on deep learning, including: a storage module, a processing module, a feature extraction module, and a calculation module;

[0008] The storage module is used to store the patent image dataset through HDFS and the patent text dataset through MongoDB; the patent image dataset includes the drawings in the patent specification, and the patent text dataset includes the text data in the patent excluding the drawings in the patent specification;

[0009] The processing module is used to distribute the patent dataset and the image dataset to the feature extraction module through a load balancing strategy;

[0010] The feature extraction module is used to extract patent drawing feature vectors from the patent image dataset and patent text feature vectors from the patent text dataset based on the OCR algorithm, ResNet model, and NLP technology; and to extract text feature vectors and image feature vectors from images uploaded by users to be retrieved based on the OCR algorithm, ResNet model, and NLP technology;

[0011] The calculation module is used to calculate the similarity between the image to be retrieved and the patent in the storage module based on the patent drawing feature vector, the patent text feature vector, the text feature vector and the image feature vector.

[0012] Furthermore, it also includes a business module,

[0013] Used to receive the image to be retrieved and upload the image to be retrieved to the processing module; obtain corresponding patent image data and patent text data from HDFS and MongoDB databases based on the similarity obtained by the calculation module; combine the obtained patent image data and the obtained patent text data into a complete patent.

[0014] Furthermore, the business module is further configured to:

[0015] The acquired patents are sorted according to a preset similarity threshold.

[0016] Furthermore, the storage module is specifically used to:

[0017] Use the HDFS distributed file system to store patent image datasets and obtain the image storage path;

[0018] The storage path of the patent text dataset and the patent image dataset in HDFS is stored in the MongoDB distributed document storage database.

[0019] Furthermore, the processing module is specifically configured to:

[0020] Create a resilient distributed dataset (RDD);

[0021] Read the patent image data and the image storage path in HDFS to each partition of the RDD;

[0022] Obtaining patent text data corresponding to the patent image data according to the image storage path and the MongoDB database;

[0023] Obtain the patent text data into each partition of the RDD;

[0024] Through the load balancing strategy, each partition is assigned to the feature extraction module.

[0025] Furthermore, the feature extraction module includes a patent search unit;

[0026] The RDD partition includes a patent image dataset and a first patent text dataset;

[0027] The patent search unit is used to extract the second patent text dataset from the patent image dataset in each partition of the RDD through an OCR algorithm;

[0028] Based on the ResNet model, feature extraction is performed on the patent image dataset in each RDD partition to obtain the patent drawing feature vector;

[0029] Based on NLP technology, feature extraction is performed on the first patent text data set and the second patent text data set in each RDD partition to obtain a first patent text data feature vector and a second patent text data feature vector;

[0030] Based on the feature vector multiplication fusion formula, the first patent text feature vector and the second patent text feature vector are fused to obtain a patent text feature vector.

[0031] Furthermore, the feature extraction module also includes a user picture retrieval unit;

[0032] The user image retrieval unit is used to extract text data from the image to be retrieved through the OCR algorithm;

[0033] Based on NLP technology, feature extraction is performed on the text data in the image to be retrieved to obtain a text feature vector;

[0034] Based on the ResNet model, feature extraction is performed on the image to be retrieved to obtain an image feature vector.

[0035] Furthermore, the calculation module is pre-configured with a Faiss library, and the calculation module includes a calculation unit and a storage unit;

[0036] The storage unit is specifically used for:

[0037] Based on the Faiss library, the patent drawing feature vector and the patent text feature vector are stored;

[0038] The computing unit is specifically configured to:

[0039] Calculating the similarity between the patent drawing feature vector and the image feature vector;

[0040] Calculating the similarity between the patent text feature vector and the text feature vector;

[0041] The similarity is calculated using the following method:

[0042] Among them, A i Represents the patent drawing feature vector or patent text feature vector in the existing Faiss database; B i The image feature vector or text feature vector representing the image to be retrieved.

[0043] Secondly, the embodiments of the present application provide a patent image retrieval method based on deep learning, including:

[0044] The patent image dataset is stored in HDFS, and the patent text dataset is stored in MongoDB. The patent image dataset includes the drawings in the patent specification, and the patent text dataset includes the text data in the patent excluding the drawings in the patent specification.

[0045] Allocating the patent image dataset and the patent text dataset to a feature extraction module through a load balancing strategy;

[0046] Extracting patent image feature vectors from the patent image dataset and patent text feature vectors from the patent text dataset based on the OCR algorithm, ResNet model, and NLP technology; extracting text feature vectors and image feature vectors from user-uploaded images to be retrieved based on the OCR algorithm, ResNet model, and NLP technology;

[0047] Based on the patent drawing feature vector, the patent text feature vector, the text feature vector and the image feature vector, the similarity between the image to be retrieved and the patent in the storage module is calculated.

[0048] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the image retrieval system instructions described in any one of the first aspects.

[0049] Compared with the existing technology, this application can at least achieve the following technical effects:

[0050] This application uses OCR algorithms, ResNet models, and NLP technologies to extract features from text and drawings in patent data and images to be retrieved. It then calculates the similarity between the feature vectors of the images to be retrieved and the feature vectors of the patent dataset based on the Faiss library, enabling users to accurately retrieve patents similar to the images to be retrieved without relying on keywords, thereby improving user retrieval efficiency.

[0051] Through the deep learning system, the search images and patent data are searched from the two aspects of text features and image features respectively, which can locate the patents more accurately, make up for the search personnel's incomplete consideration of patents, and make the user's search more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] FIG1 is a schematic diagram of the device structure of a patent image retrieval system based on deep learning provided by one or more embodiments of this specification;

[0054] FIG2 is a flow chart of a patent image retrieval method based on deep learning provided in one or more embodiments of this specification;

[0055] FIG3 is a schematic diagram of a retrieval structure of a patent image retrieval system based on deep learning provided in one or more embodiments of this specification;

[0056] FIG4 is a schematic diagram of a retrieval scenario of a patent image retrieval method based on deep learning in one or more embodiments of this specification. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0058] Existing patents typically require keyword searches before filing to determine if the patent is prior art, thus avoiding duplicate applications and wasting manpower and resources. When searching patent documents using keywords, due to their high scalability, many keywords have the same or similar meanings, each corresponding to a different patent. Searching for similar patents among numerous patents can result in long search times, inaccuracies, and low efficiency.

[0059] In order to overcome the above technical problems, this application proposes a patent image retrieval system based on deep learning, including:

[0060] In the embodiment of the present application, the image retrieval system includes: a storage module 101, a processing module 102, a feature extraction module 103, and a calculation module 104, as shown in FIG1;

[0061] The storage module 101, the processing module 102, the feature extraction module 103 and the calculation module 104 are specifically:

[0062] The storage module corresponds to two server clusters: an HDFS cluster and a MongoDB cluster. The HDFS cluster stores the patent image dataset, and the MongoDB cluster stores the patent text dataset. The processing module and the feature extraction module each correspond to a Spark server cluster. The cluster server corresponding to the processing module partitions the dataset through a load balancing strategy, and the cluster corresponding to the feature extraction module is used to extract features from the data in each partition. The computing module corresponds to a server and is used to calculate the similarity between datasets.

[0063] Specifically, the storage module 101 is used to store the patent text dataset and the patent image dataset through HDFS and MongoDB; the patent image dataset includes the specification drawings in the patent, and the patent text dataset includes the text data in the patent other than the specification drawings; the processing module 102 is used to distribute the patent dataset and the image dataset to the feature extraction module through a load balancing strategy; the feature extraction module 103 is used to obtain the patent drawing feature vector and text feature vector and the text feature vector and image feature vector of the user-uploaded image to be retrieved based on the OCR algorithm, ResNet model and NLP technology; the calculation module 104 is used to store the patent drawing feature vector and text feature vector through the Faiss library, and calculate the cosine similarity based on the image to be retrieved.

[0064] Among them, the feature extraction module 103 includes multiple server nodes, and the processing module 102 distributes the patent image dataset and the patent text dataset to each server node through a load balancing strategy. Then, each server node processes the patent data and image data in parallel to extract their text feature vectors and image feature vectors.

[0065] In an embodiment of the present application, the business module 105 is used to receive the image to be retrieved and upload the image to be retrieved to the processing module; according to the similarity obtained by the calculation module, the corresponding patent image data and patent text data are obtained from the HDFS and MongoDB databases; and the obtained patent image data and the obtained patent text data are combined into a complete patent.

[0066] Specifically, the text and image are compared using a similarity threshold to determine the degree of similarity between the image to be retrieved and the patents in the patent feature database. For example, when identifying images, the similarity threshold is set to 0.8-0.9. When the similarity between two images exceeds this threshold, they are considered to have the same technical solutions.

[0067] The business module 105 is further specifically configured to sort the patent data according to similarity and display the acquired patent data based on a similarity threshold.

[0068] The storage module 101 is specifically used to store the patent image dataset and obtain the image storage path through the HDFS distributed file system; and to store the text data and the storage path of the patent image in HDFS through the MongoDB distributed document storage database.

[0069] The processing module 102 is specifically used to create a resilient distributed dataset (RDD); read the patent image data and storage path in HDFS to each partition of the RDD; obtain the patent text data corresponding to the patent image data based on the storage path and the MongoDB database; obtain the patent text data to each partition of the RDD; and match the data in the partition to the feature extraction module through a load balancing strategy.

[0070] The feature extraction module 103 includes a patent retrieval unit; the patent retrieval unit is used to extract the patent text data set from the patent image data set in each RDD partition through the OCR algorithm; obtain the patent drawing feature vector through the ResNet model and the patent image data set in each RDD partition; obtain the first patent text data feature vector and the second patent text data feature vector through the NLP technology and the patent text data set in each RDD partition; based on the feature vector multiplication and fusion formula, the first patent text feature vector and the second patent text feature vector are fused to obtain the patent text feature vector.

[0071] The feature extraction module 103 also includes a user image retrieval unit; the user image retrieval unit is used to extract the text data of the image to be retrieved through the OCR algorithm; obtain a text feature vector through NLP technology and the text data of the image to be retrieved; and obtain an image feature vector through the ResNet model of the image to be retrieved.

[0072] The calculation module 104 is provided with a Faiss library, including a calculation unit and a storage unit;

[0073] The storage unit is specifically used to: store the patent drawing feature vector and the patent text feature vector based on the Faiss library;

[0074] The calculation unit is specifically used to: calculate the similarity between the patent drawing feature vector and the image feature vector;

[0075] Calculating the similarity between the patent text feature vector and the text feature vector;

[0076] The similarity is calculated using the following method:

[0077] Among them, A i Represents the patent drawing feature vector or patent text feature vector in the existing Faiss database; B i The image feature vector or text feature vector representing the image to be retrieved.

[0078] This application proposes a patent image retrieval method based on deep learning, as shown in Figure 2, which includes the following steps:

[0079] Example 1

[0080] Step S1: Based on the storage module, the patent text dataset and the patent image dataset are stored.

[0081] Specifically, global patent data and specification drawing data sets are obtained and stored separately.

[0082] In the embodiments of the present application, due to the rapid development of the patent industry in recent years, the amount of patent data generated is huge. If a traditional relational database is used to store large-scale patent data, there are still many disadvantages. If a large amount of patent data is simply stored on one server, when the number of users searching for patents online at the same time increases sharply, the backend server will be overloaded or even crash. In addition, storing such a large amount of patent data on a network server makes it difficult to effectively guarantee the security and reliability of the patent data. Therefore, based on the analysis of the above problems and combined with the currently popular distributed storage technology for big data, this application adopts the HDFS+MongoDB distributed storage method that can be deployed on a Spark cluster to store these tens of thousands of patent data.

[0083] The storage structure in HDFS (Hadoop Distributed File System) is a master node (NameNode) that controls multiple slave nodes (DataNodes). Clients submit requests to the NameNode, which then assigns tasks to DataNodes to meet user needs. The NameNode refers to the master node in the HDFS system's master-slave architecture. There is only one NameNode in each HDFS system. This master node is primarily responsible for managing the system's file namespace, data block metadata, client access to files, and assigning specific tasks to DataNodes. The DataNode refers to the slave node in the HDFS system's master-slave architecture. Each HDFS system can contain multiple DataNodes, which primarily store data block information. Each DataNode contains multiple data blocks and replicas of those blocks from other DataNodes. Storing proprietary data solely in HDFS can easily lead to the NameNode as a single point of bottleneck in real-world applications. In HDFS, the NameNode serves as the communication bridge between clients and DataNodes. When a user sends an image search request to the NameNode, the NameNode finds the DateNode corresponding to similar patent information based on the user's request. Only then does the user interact with the DateNode for specific data information. Each image query involves task-specific interaction with the DateNode through the NameNode. If a large number of users submit requests to the cluster simultaneously from different PCs, the performance of a single NameNode can become a bottleneck for the entire cluster.

[0084] Based on the above-mentioned problems encountered in patent data storage, this application adopts the following approach to handle them.

[0085] The data storage method is:

[0086] A patent database contains data such as publication number, application number, claims, specification, and drawings. Due to the single-point bottleneck problem in HDFS, MongoDB (a distributed document storage database) is used to optimize the single-node bottleneck problem in the HDFS distributed storage solution. Therefore, the patent data is stored in MongoDB and HDFS respectively to improve the access efficiency of patent documents.

[0087] Specifically, the optimization solution is to optimize the original architecture of the cluster controlled by one master node into one controlled by multiple master nodes. Since the drawings in the patent specification occupy a large amount of space, the drawings are stored in HDFS. The file size in HDFS is usually in the GB to TB level. HDFS is designed to support the storage of large files, which increases the storage space while also improving access efficiency. Based on the above reasons, the design of this application stores the drawings in HDFS, so that when crawling patent data, the pictures in HDFS can be directly loaded, thereby making the calculation of the pictures faster and more efficient, and better adapting to the distributed computing framework.

[0088] In an embodiment of the present application, patent data other than the drawings in the specification are stored in MongoDB. MongoDB is a product between relational databases and non-relational databases, using efficient binary data storage and achieving efficient data storage. The present application stores the text data in the global patent data in MongoDB, and persists all data information such as the storage path of each image in HDFS in the MongoDB database, so that each master node in HDFS can establish communication with the MongoDB database. This storage method not only ensures the integrity of the data information of the entire cluster but also greatly reduces the burden on a single master node, thereby achieving efficient storage and query of patent data. Among them, the storage path of HDFS is stored as a document, and each document contains a unique document ID, for example: the publication number is used as the document ID of the patent.

[0089] Specifically, for each storage operation, the storage path of the specification drawings in HDFS is recorded and stored with the publication number as the folder name. For example, the HDFS storage path is:

[0090] hdfs: / / <hdfs-namenode> : <port>images / CN202340643A / Attachment 1.jpg.

[0091] After the image is stored, each image has an absolute path. The absolute path includes the current path + relative path. The current path of all images is consistent and is not stored, which can save storage space, such as: hdfs: / / <hdfs-namenode> : <port>images / ; relative paths are used to distinguish different images from different patents, such as: / CN202340643U / Attachment 1.jpg. The relative paths are then stored in MongoDB, and the publication number and the relative path of the image are associated, so that patents can be searched by publication number or path.

[0092] Step S2: Based on the processing module and the feature extraction module, the patent drawing feature vector and the patent text feature vector are extracted in parallel.

[0093] Specifically, due to the large amount of patent data, in order to meet the high-speed processing requirements of feature extraction and shorten the feature extraction processing time of images and text, this application uses the Spark computing engine to establish parallel processing based on distributed storage.

[0094] Patent data feature extraction is implemented based on Spark. As shown in Figure 3, an RDD is created. When multiple RDDs (Resilient Distributed Datasets) exist, they are AF, each containing multiple partitions. When data processing begins, Spark reads data from HDFS and MongoDB, creating two storage blocks: Block 1 and Block 2. The initial RDDs in these two blocks are A and C, respectively. RDD A is processed into RDD B using the flatMap operation, and RDD C is mapped into RDD D. RDD D is then combined into RDD E using the reduceByKey function. After processing Blocks 1 and 2, Block 3 shuffles and merges the results of RDD B and RDD E using the join operation to create RDD F. RDD F contains the processed results. Finally, the results of RDD F are stored in the corresponding distributed database using the saveAsSequenceFile method to implement the Action operation.

[0095] Specifically, a Spark function is used to read the drawings from HDFS into partitions of an RDD. Based on the InputSplit parameter, the path to each drawing in HDFS is mapped to each partition of the RDD. A block in HDFS is loaded as a partition. The data in the partitions is then matched to the server nodes of the corresponding cluster using a load balancing strategy for processing, fully leveraging the cluster's computing power. Each node uses the mapPartitions function to batch process the storage paths of the drawings in each partition. Based on the storage path of each drawing, the corresponding patent text data in MongoDB is searched. Based on the patent text data, the specific implementation method is located to obtain the text description related to the image. The patent data stored in the partition is obtained and calculations are performed on each partition of the RDD. Finally, based on each node in the Spark cluster, feature extraction is performed on the patent image data and the corresponding patent text data to obtain a patent drawing feature vector and a first patent text data feature vector. Feature extraction is then performed on the patent text data in the patent image data to obtain a second patent text data feature vector. The first and second patent text feature vectors are then fused using a feature vector multiplication fusion formula to obtain a patent text feature vector. At the same time, the patent publication number is used as the document ID to identify the patent. The extracted patent drawing feature vector and patent text feature vector are named according to the publication number. The extracted feature vectors are then stored in the Faiss library and MongoDB.

[0096] For example, a global patent dataset is obtained, with Hadoop HDFS storing the drawings of 1 million patents and MongoDB storing the text data of 1 million patents. A Spark cluster with 5 nodes is built. An RDD with 10 partitions is created, with each partition loaded with 128MB of data blocks and the storage path of the drawings in each partition. A load balancing strategy is then used to match server nodes to each partition. The server node uses the storage path of each drawing in the partition to search MongoDB for the text data of each drawing, obtain the text description of the drawing in the specific implementation, and then perform feature extraction on both the drawing and the text. The extracted features are named by the publication number when stored. The features extracted from HDFS and MongoDB are then merged based on the publication number. If both drawing features and text features are extracted from a patent drawing in HDFS, and a text feature is also extracted from the same patent in MongoDB, the text features in HDFS and MongoDB need to be merged so that the patent has only one text feature and one drawing feature. Both the text feature and the drawing feature of the patent are then stored in the patent feature database.

[0097] The image features and text features in the attached figure are extracted as follows:

[0098] Patents often describe their solutions using a combination of drawings and text. Therefore, the actual content of the drawings is divided into images and text descriptions. To improve the accuracy of image retrieval, it is necessary to perform feature extraction on both the images in the drawings and the corresponding text.

[0099] First, identify the drawings in the patent. Patents contain descriptions of the drawings and detailed implementations. The descriptions indicate the subject matter and icons of the drawings, while the detailed implementations describe the content of the drawings. Furthermore, the detailed implementations contain the phrase "as shown in Figure n." Based on the aforementioned patent characteristics, semantic analysis can be used to identify the text corresponding to each drawing in the patent document and each drawing. Feature extraction is then performed on each identified drawing and the text corresponding to each drawing to obtain the corresponding image features and text features.

[0100] For example, find the specific implementation method based on the picture name, then use NLP semantic analysis technology to extract text data 1 associated with the picture, use the OCR algorithm to extract the text in the illustrations of the specification to obtain illustration text data 2, and use the Word2Vec word vector model in the NLP semantic analysis technology to perform feature extraction on text data 1 and illustration text data 2 respectively to generate text feature vector 1 and illustration text feature vector 2.

[0101] It should be noted that there are also some texts in the drawings, which are important for understanding the drawings. When extracting features, these texts in the drawings should also be extracted to obtain corresponding text feature vectors.

[0102] According to the above method, the feature extraction process of the drawings and texts is as follows: when data processing starts, an RDD distributed data set is created in Spark, and text data and image data are read from the data storage space through RDD. The data storage space includes HDFS and MongoDB. The text data and image data are generated as tasks and pushed to each node. Each node scales, normalizes and crops the image to ensure the consistency of the image. Then, according to the storage path of the image, the text data associated with the image in the specific implementation method is located and semantic analysis is performed through NLP semantic analysis technology to obtain text data 1 related to the image; then, the text in the drawings of the specification is extracted through the OCR algorithm to obtain the drawing text data 2; the Word2Vec word vector model is used to extract features from the text data 1 and the drawing text data 2 respectively to generate text feature vector 1 and drawing text feature vector 2; then the deep learning framework PyTorch is used to load the pre-trained ResNet (Residual Network) model, the drawings of the specification are input into the ResNet model, and the image feature vectors of the required layers are extracted. The pre-trained model is usually trained on a large-scale image dataset and has a powerful image feature extraction capability.

[0103] Step S3: Based on the calculation module, calculate the similarity and build a patent feature database.

[0104] Specifically, a Faiss library is constructed as a patent feature database, and text feature vectors and image feature vectors are stored in the Faiss library and MongoDB. The Faiss library is used to perform similarity calculations. This application is set to adopt a cosine similarity algorithm, and similarity calculations are performed on feature vectors based on cosine similarity. The feature vectors are stored in MongoDB for backup to prevent data loss.

[0105] The Faiss library is primarily used for large-scale similarity search and dense vector analysis. It includes multiple similarity search methods and offers both CPU and GPU support. The Faiss library provides a range of algorithms and data structures. This application experimentally compares various similarity algorithms to identify a suitable similarity algorithm for this solution. This approach improves vector similarity retrieval speed and reduces memory usage with minimal loss of precision.

[0106] Step S4: Search patents based on business modules.

[0107] Specifically, users upload pictures to the server through the client. After receiving the pictures, the server uses Spark to extract features. In Spark, the pictures uploaded by the client are loaded based on RDD and the pictures are assigned to the server nodes. Then, a custom algorithm is called to extract the features of the pictures. When recognizing the customer's pictures, OCR (Optical Character Recognition) and ResNet are used to extract features of the pictures at the same time. The pictures are divided into the following three scenarios.

[0108] Only text: OCR can be used to recognize text in images, and the OCR results are used as the text information of the image. The Word2Vec word vector model is then used to extract features from the text information to obtain a text feature vector. However, the ResNet model cannot be used to extract feature vectors of images.

[0109] Image only: The OCR algorithm fails to recognize text in the image. The ResNet model then recognizes the image features and obtains the image feature vector.

[0110] There are both images and text: Use the OCR algorithm to recognize the text in the image and extract the image text information. Then use the Word2Vec word vector model to extract the features of the text information and obtain the text feature vector. Then use the ResNet model to recognize the image features and obtain the image feature vector.

[0111] The above-mentioned text feature vector and image feature vector are transferred into the patent feature database. The feature vectors in the patent feature database are compared with the text feature vectors and image feature vectors extracted after processing the images uploaded by the user, and the cosine similarity is calculated. The text feature vector in the patent feature database is compared with the text feature vector of the user, and the image feature vector in the patent feature database is compared with the image feature vector of the user. A cosine similarity threshold is set, and the similarity is sorted according to the threshold. The similarity is then returned to the client.

[0112] For example, if a customer uploads image A, as shown in Figure 4, the image retrieval process can be divided into the following scenarios:

[0113] 1. If there is only text in Figure A: extract the text feature vector and compare it with the text feature vectors of Patent 1, Patent 2, Patent 3 and other data in the patent feature database. If there are three drawings in Patent 1, namely Figure a, Figure b, and Figure c, the cosine similarity calculation between the text feature vector of Figure a and the text feature vector of Figure A is 60%. Then the cosine similarity calculation between the text feature vector of Figure a and the text feature vector of Figure A is 70%. The cosine similarity calculation between the text feature vector of Figure b and the text feature vector of Figure A is 70%. 75%, Figure b has an image feature vector but Figure A does not, so the similarity of the image feature vectors is not compared; the text feature vector of Figure c is compared with the text feature vector of Figure A, and the similarity is 75%, and the similarity of the text features of the attached drawings is 85%; there are multiple similarities in Patent 1, and the maximum percentage is selected as the similarity between Patent 1 and Figure A, so the similarity of Patent 1 is 85%; if the text similarity of Patent 2 is 90% and the text similarity of Patent 3 is 70%, and the patent similarities are sorted, the display order returned to the client is Patent 2, Patent 1, Patent 3, etc.

[0114] 2. If the recognized image A contains both text and images: extract the text feature vector and image feature vector, and calculate the similarity between the text feature vector and image feature vector in Figure A and the text feature vector and image feature vector of Patent 1, Patent 2, Patent 3 and other data in the patent feature library. The text feature similarity in Patent 1 is 90%, and the image feature similarity is 80%; the similarities of the text feature and image feature of Patent 2 are 80% and 90% respectively; the similarities of the text feature and image feature of Patent 3 are 80% and 80% respectively; the similarities of the text feature and image feature of Patent 4 are 50% and 70% respectively. First, sort from large to small according to the text feature similarity. If the text feature similarities are the same, sort the images by similarity, and then sort the similarities according to the similarity threshold. If the similarity threshold is 70%, the display order returned to the client is Patent 1, Patent 2, Patent 3, Patent 4, etc.

[0115] 3. If there is only one image in the identified image A: extract the image feature vector, calculate the similarity between the image feature vector and the image feature vectors of patent 1, patent 2, patent 3 and other data in the patent feature database, screen out the largest similarity among patent 1, patent 2, patent 3 and other data, and use it as the similarity between image A and patent 1, patent 2, patent 3 and other data. The similarity of patent 1 is 20%, the similarity of patent 2 is 45%, the similarity of patent 3 is 80%, etc. If the similarity threshold is 70%, then patents with a similarity greater than 70% are all ranked in front according to the percentage size, and the patent data is returned to the client in descending order, so the display order on the client is patent 3, patent 2, patent 1, etc.

[0116] Example 2

[0117] Step S1: The front-end client uploads the image to be retrieved.

[0118] Specifically, the patent search method is not limited to using a patent search client, WeChat applets, or a browser or client terminal device used to log in to the patent search website.

[0119] For example, terminal devices may include but are not limited to mobile phones, tablet computers, laptops, PCs, smart home appliances, smart wearable devices, etc.

[0120] Specifically, the client can upload one or more pictures, select the path and insert the picture in the input box, and click Upload.

[0121] Step S2: The backend server receives the image.

[0122] Specifically, the server obtains the images to be retrieved and classifies the images to be retrieved.

[0123] The text information of the image to be retrieved is extracted through the OCR algorithm, and then the image features of the image to be retrieved are identified using the ResNet model. Based on the above operations, the image to be retrieved is divided into the following three scenarios and processed accordingly.

[0124] The text information of the image to be retrieved can be extracted but the image features cannot be identified: text is extracted from the image to be retrieved based on the OCR algorithm, and then feature extraction is performed using the Word2Vec word vector model in the NLP semantic analysis technology to obtain a text feature vector.

[0125] The image to be retrieved cannot be used to extract text information, but image features can be identified: image recognition is performed on the image to be retrieved based on the ResNet model to obtain the image feature vector.

[0126] The image to be retrieved can both extract text information and recognize image features: first, text is extracted from the image to be retrieved based on the OCR algorithm, and then feature extraction is performed using the Word2Vec word vector model in NLP semantic analysis technology to obtain a text feature vector; then image recognition is performed on the image to be retrieved based on the ResNet model to obtain an image feature vector.

[0127] Step S3: Calculate similarity.

[0128] Specifically, similarity calculation is performed based on the text feature vectors and image feature vectors in the patent feature database and the text feature vectors and image feature vectors obtained from the image to be retrieved, and the patents are sorted by similarity, and the document ID that is most similar to the query vector is returned.

[0129] If the image is entirely text, the text feature vector of the image to be retrieved is compared with the text feature vector in the patent feature database, and the cosine similarity is calculated. If a patent has multiple drawings, each drawing will correspond to a text feature vector. The drawings with text feature vectors in a patent are compared with the text feature vectors in the image to be retrieved, and the cosine similarity is calculated. A patent will obtain multiple text feature similarity values. The similarities in the patent are then sorted, and the largest similarity is selected as the similarity between the patent and the image to be retrieved. In this application, the larger the similarity value, the closer the patent is to the image to be retrieved, and the higher it is presented in the front-end client.

[0130] For images, the image feature vectors of the image to be retrieved are compared with those in the patent feature database, and cosine similarity is calculated. If a patent has multiple drawings, each with an image feature vector, the drawings with image feature vectors in the patent are compared with the image feature vectors in the image to be retrieved, and cosine similarity is calculated. A patent will have multiple image feature similarity values. The similarities in the patent are then sorted, and the largest similarity is selected as the similarity between the patent and the image to be retrieved. In this application, the larger the similarity value, the closer the patent is to the image to be retrieved, and the higher it is presented in the front-end client.

[0131] If there are both text and pictures, the text feature vector and image feature vector of the picture to be retrieved are compared with the text feature vector and image feature vector in the patent feature database respectively, and the cosine similarity is calculated. If a patent has multiple drawings, each of which has a text feature vector or an image feature vector, the text feature vector of the image to be retrieved is first compared with the text feature vector in the patent database for similarity, and the cosine similarity is calculated. The patent will obtain multiple text feature similarity values, and then the similarities in the patent are sorted, and the largest similarity among the text features is selected; the image feature vector of the image to be retrieved is then compared with the image feature vector in the patent feature database for similarity, and the cosine similarity is calculated. The patent will obtain multiple image feature similarity values, and then the similarities in the patent are sorted, and the largest similarity among the image features is selected to obtain the text feature similarity value and image feature similarity value of the patent as the similarity between the patent and the image to be retrieved; the text feature similarity value and the image feature similarity value are then sorted, and a similarity threshold is set. If the cosine similarity of the text feature vector and the image feature vector are both greater than the threshold, the patent is ranked first when presented in the foreground. If one of the cosine similarity of the text feature vector and the image feature vector is greater than the threshold, the patent is ranked in the middle when presented in the foreground.

[0132] Based on the above method, this application compares the text features or image features represented by each figure of the patent and the text features represented by each figure in the specific implementation method with the text features or image features of the image to be retrieved, and calculates the cosine similarity. This method can accurately extract the meaning to be expressed by the image to be retrieved and the meaning to be expressed by the patent figure, and then compare the similarity between the two, and select the similarity value of the figure with the highest similarity to the image to be retrieved as the similarity between the patent and the image to be retrieved. This method can make patent retrieval more accurate and more efficient. Then, use the returned document ID to query the corresponding patent data from MongoDB (this can be achieved by mapping the document ID stored in the Faiss index and the document ID in MongoDB), and based on the storage path of the figure in MongoDB, obtain the figure information in HDFS, thereby obtaining the complete patent data. Finally, the basic information of the patent data is returned to the user together with the feature vector similarity.

[0133] For example: a customer uploads a patent flowchart to the front-end client, the back-end server receives the image, uses the OCR algorithm to extract the text in the image, and then uses the Word2Vec word vector model to extract features of the extracted text to generate a text feature vector; then based on the ResNet deep learning network model, the image feature vector of the patent image is extracted, and the cosine similarity of this text feature vector and image feature vector is compared with the vector in the patent feature database, and the similarity is sorted from large to small, and then similar patents are presented to the user in order.

[0134] The calculation formula of cosine similarity is as follows:

[0135] Among them, A i Represents the patent drawing feature vector or patent text feature vector in the existing Faiss database; B i The image feature vector or text feature vector representing the image to be retrieved.

[0136] If a customer uploads a picture, the OCR algorithm is used to extract the text in the image to be retrieved, and then the Word2Vec word vector model is used to extract features of the extracted text to obtain a text feature vector. Then, based on the ResNet deep learning network model, the image feature vector in the image to be retrieved is extracted. If the cosine similarity between the text feature vector and the image feature vector of Patent 1 in the feature library is 90% and 80% respectively, if the cosine similarity between the text feature vector and the image feature vector of Patent 2 in the feature library is 80% and 40% respectively, and the similarity threshold is 80%, then the similarity ranking is Patent 1, Patent 2, etc.

[0137] An embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the image retrieval system instructions described in any of the above embodiments.

[0138] It should be noted that the embodiment of the storage medium in this specification and the embodiment of the blockchain-based service provision method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding blockchain-based service provision method mentioned above, and the repeated parts will not be repeated.

[0139] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] In the 1930s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using "logic compiler" software. This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a hardware description language (HDL). There are many different HDLs, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also appreciate that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0141] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0142] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0143] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0144] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0146] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0148] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0149] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0150] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0152] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0153] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0154] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.< / port> < / hdfs-namenode> < / port> < / hdfs-namenode>

Claims

1. A patent image retrieval system based on deep learning, characterized in that, Including: A storage module, a processing module, a feature extraction module, and a calculation module; The storage module is used to store the patent image dataset through HDFS and store the patent text dataset through MongoDB. The patent image dataset includes the specification drawings in the patent, and the patent text dataset includes the text data in the patent except for the specification drawings; The processing module is used to allocate the patent image dataset and the patent text dataset to the feature extraction module through a load balancing strategy; The feature extraction module is used to extract the patent drawing feature vectors of the patent image dataset and the patent text feature vectors of the patent text dataset according to the OCR algorithm, the ResNet model, and NLP technology; extract the text feature vectors and image feature vectors of the to-be-retrieved image uploaded by the user according to the OCR algorithm, the ResNet model, and NLP technology; The calculation module is used to calculate the similarity between the to-be-retrieved image and the patents in the storage module according to the patent drawing feature vectors, the patent text feature vectors, the text feature vectors, and the image feature vectors.

2. The patent image retrieval system based on deep learning according to claim 1, wherein It further includes a business module, which is used to receive the to-be-retrieved image and upload the to-be-retrieved image to the processing module; obtain the corresponding patent image data and patent text data from the HDFS and MongoDB databases according to the similarity obtained by the calculation module; combine the obtained patent image data and the obtained patent text data into a complete patent.

3. The deep learning-based patent image retrieval system according to claim 2, wherein The business module is specifically further used for: Sort the obtained patents according to a preset similarity threshold.

4. The deep learning-based patent image retrieval system according to claim 1, wherein The storage module is specifically used for: Store the patent image dataset through the HDFS distributed file system and obtain the image storage path; Store the patent text dataset and the storage path of the patent image dataset in HDFS through the MongoDB distributed document storage database.

5. The deep learning-based patent image retrieval system according to claim 4, wherein The processing module is specifically used for: Create a resilient distributed dataset (RDD); Read the patent image data and the image storage path in HDFS into each partition of the RDD; Obtain the patent text data corresponding to the patent image data according to the image storage path and the MongoDB database; Obtain the patent text data into each partition of the RDD; Allocate each partition to the feature extraction module through a load balancing strategy.

6. The patent image retrieval system based on deep learning according to claim 5, characterized in that, The feature extraction module includes a patent retrieval unit; The RDD partition includes a patent image dataset and a first patent text dataset; The patent retrieval unit is used for: Extract the second patent text dataset in the patent image dataset in each partition of the RDD through the OCR algorithm. Based on the ResNet model, feature extraction is performed on the patent image dataset in each partition of the RDD to obtain patent drawing feature vectors; Based on NLP technology, feature extraction is performed on the first patent text dataset and the second patent text dataset in each partition of the RDD to obtain the first patent text data feature vector and the second patent text data feature vector; Based on the feature vector multiplication fusion formula, the first patent text feature vector and the second patent text feature vector are fused to obtain a patent text feature vector.

7. The deep learning-based patent image retrieval system according to claim 1, wherein The feature extraction module further includes a user image retrieval unit; The user image retrieval unit is used to extract the text data in the to-be-retrieved image through the OCR algorithm; Based on NLP technology, feature extraction is performed on the text data in the to-be-retrieved image to obtain a text feature vector; Based on the ResNet model, feature extraction is performed on the to-be-retrieved image to obtain an image feature vector.

8. The deep learning-based patent image retrieval system according to claim 1, wherein The calculation module is pre-set with a Faiss library, and the calculation module includes a calculation unit and a storage unit; The storage unit is specifically used for: Based on the Faiss library, storing the patent drawing feature vectors and the patent text feature vectors; The calculation unit is specifically used for: Calculating the similarity between the patent drawing feature vector and the image feature vector; Calculating the similarity between the patent text feature vector and the text feature vector; The similarity is calculated by the following method: Among them, A i represents the patent drawing feature vector or patent text feature vector in the existing Faiss library; B i represents the image feature vector or text feature vector of the picture to be retrieved.

9. A deep learning-based patent image retrieval method for the system according to any one of claims 1-8, characterized in that, Including: Storing the patent image dataset through HDFS; storing the patent text dataset through MongoDB; the patent image dataset includes the specification drawings in the patent, and the patent text dataset includes the text data in the patent except the specification drawings; Through the load balancing strategy, the patent image dataset and the patent text dataset are allocated to the feature extraction module; According to the OCR algorithm, the ResNet model and NLP technology, extracting the patent drawing feature vectors of the patent image dataset and the patent text feature vectors of the patent text dataset; according to the OCR algorithm, the ResNet model and NLP technology, extracting the text feature vectors and image feature vectors of the to-be-retrieved image uploaded by the user; According to the patent drawing feature vector, the patent text feature vector, the text feature vector and the image feature vector, calculating the similarity between the to-be-retrieved image and the patents in the storage module.

10. A computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the instructions of the image retrieval system according to any one of claims 1-8 are implemented.

Images