Patent image retrieval system, method and device based on deep learning

Abstract

The embodiment of the invention provides a patent image retrieval method, device, equipment and system based on deep learning, and the retrieval system comprises a storage module, a processing module, a feature extraction module and a calculation module. The storage module is used for storing a patent picture data set through an HDFS; a patent text data set is stored through a MongoDB; the patent picture data set comprises specification drawings in patents, and the patent text data set comprises text data except the specification drawings in the patents; the processing module is used for distributing the patent data set and the picture data set to the feature extraction module through a load balancing strategy; the feature extraction module is used for extracting a patent drawing feature vector of the patent picture data set and a patent text feature vector of the patent text data set according to an OCR algorithm, a ResNet model and an NLP technology; according to an OCR algorithm, a ResNet model and an NLP technology, extracting a text feature vector and an image feature vector of a to-be-retrieved picture uploaded by a user; and the calculation module is used for calculating the similarity between the to-be-retrieved picture and the patent in the storage module according to the patent drawing feature vector, the patent text feature vector, the text feature vector and the image feature vector. The problems of low patent retrieval efficiency and inaccurate retrieval are solved.

Description

Technical Field

[0001] This document relates to the field of image processing, and in particular to a patent image retrieval system, method and apparatus based on deep learning. Background Technology

[0002] With the rapid development of the information age, the number of patent applications is increasing day by day. Governments and enterprises are paying more and more attention to the application status of patents. In order to obtain patent rights, enterprises first need to determine whether their inventions are novel and search for similar patents through the patent search system to ensure the uniqueness of the patent.

[0003] Patent search systems typically match related patents using keywords, requiring a deep understanding of patents to accurately find the keywords.

[0004] However, when searching for similar patents, searchers often need to rely on keyword searches. Since most keywords have multiple meanings, the search results are prone to generating a lot of noise, resulting in low patent search efficiency and inaccurate results. Summary of the Invention

[0005] In view of the above solutions, this application aims to propose a patent image retrieval system, method and apparatus based on deep learning to solve at least one of the above technical problems.

[0006] In one aspect, one or more embodiments of this specification provide a patent image retrieval system based on deep learning, including: a storage module, a processing module, a feature extraction module, and a calculation module;

[0007] The storage module is used to store patent image datasets via HDFS and patent text datasets via MongoDB; the patent image datasets include the drawings in the patent specification, and the patent text datasets include the text data in the patent other than the drawings in the specification.

[0008] The processing module is used to distribute the patent dataset and the image dataset to the feature extraction module through a load balancing strategy;

[0009] The feature extraction module is used to extract the patent figure feature vector of the patent image dataset and the patent text feature vector of the patent text dataset according to the OCR algorithm, ResNet model and NLP technology; and to extract the text feature vector and image feature vector of the user-uploaded image to be retrieved according to the OCR algorithm, ResNet model and NLP technology.

[0010] The calculation module is used to calculate the similarity between the image to be retrieved and the patent in the storage module based on the feature vector of the patent figure drawing, the feature vector of the patent text, the text feature vector, and the image feature vector.

[0011] Furthermore, it also includes business modules.

[0012] The system is used to receive the image to be searched and upload it to the processing module; based on the similarity obtained by the calculation module, it retrieves the corresponding patent image data and patent text data from the HDFS and MongoDB databases; and combines the retrieved patent image data and the retrieved patent text data into a complete patent.

[0013] Furthermore, the aforementioned business module is specifically used for:

[0014] The obtained patents are sorted according to a preset similarity threshold.

[0015] Furthermore, the storage module is specifically used for:

[0016] The patent image dataset is stored using the HDFS distributed file system, and the image storage path is obtained.

[0017] The storage paths for patent text datasets and patent image datasets in HDFS are stored using the MongoDB distributed document storage database.

[0018] Furthermore, the processing module is specifically used for:

[0019] Create a resilient distributed dataset (RDD);

[0020] Read the patent image data and the image storage path from HDFS into each partition of the RDD;

[0021] Based on the image storage path and the MongoDB database, the patent text data corresponding to the patent image data is obtained;

[0022] Obtain the patent text data into each partition of the RDD;

[0023] The load balancing strategy distributes each partition to the feature extraction module.

[0024] Furthermore, the feature extraction module includes a patent retrieval unit;

[0025] The RDD partition includes a patent image dataset and a first patent text dataset;

[0026] The patent retrieval unit is used to extract the second patent text dataset from the patent image dataset in each partition of the RDD using an OCR algorithm;

[0027] Based on the ResNet model, feature extraction is performed on the patent image dataset in each partition of the RDD to obtain the feature vector of the patent figure.

[0028] Based on NLP technology, feature extraction is performed on the first patent text dataset and the second patent text dataset in each partition of the RDD to obtain the feature vector of the first patent text data and the feature vector of the second patent text data.

[0029] Based on the feature vector multiplication and fusion formula, the first patent text feature vector and the second patent text feature vector are fused to obtain the patent text feature vector.

[0030] Furthermore, the feature extraction module also includes a user image retrieval unit;

[0031] The user image retrieval unit is used to extract text data from the image to be retrieved using the OCR algorithm;

[0032] Based on NLP technology, feature extraction is performed on the text data in the image to be retrieved to obtain a text feature vector.

[0033] Based on the ResNet model, feature extraction is performed on the image to be retrieved to obtain an image feature vector.

[0034] Furthermore, the computing module is pre-configured with the Faiss library, and the computing module includes a computing unit and a storage unit;

[0035] The storage unit is specifically used for:

[0036] Based on the Faiss library, the feature vectors of the patent drawings and the feature vectors of the patent text are stored.

[0037] The computing unit is specifically used for:

[0038] Calculate the similarity between the feature vector of the patent figure drawing and the feature vector of the image;

[0039] Calculate the similarity between the patent text feature vector and the text feature vector;

[0040] The similarity is calculated using the following method:

[0041]

[0042] Among them, A i B represents the feature vector of patent figures or the feature vector of patent text in the existing Faiss library;i This represents the image feature vector or text feature vector of the image to be retrieved.

[0043] Secondly, embodiments of this application provide a patent image retrieval method based on deep learning, including:

[0044] The patent image dataset is stored using HDFS; the patent text dataset is stored using MongoDB; the patent image dataset includes the drawings in the patent specification, and the patent text dataset includes the text data in the patent other than the drawings in the specification.

[0045] The patent image dataset and the patent text dataset are allocated to the feature extraction module using a load balancing strategy.

[0046] Based on OCR algorithms, ResNet models, and NLP techniques, feature vectors of patent figures in the patent image dataset and feature vectors of patent text in the patent text dataset are extracted; text feature vectors and image feature vectors of user-uploaded images to be retrieved are extracted based on OCR algorithms, ResNet models, and NLP techniques.

[0047] Based on the feature vectors of the patent drawings, the patent text, the text feature vectors, and the image feature vectors, the similarity between the image to be retrieved and the patents in the storage module is calculated.

[0048] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the image retrieval system instructions described in any one of the first aspects.

[0049] Compared with the prior art, this application can achieve at least the following technical effects:

[0050] This application uses OCR algorithms, ResNet models, and NLP technology to extract features from text, illustrations, and images to be retrieved in patent data. Then, based on the Faiss library, it calculates the similarity between the feature vector of the image to be retrieved and the feature vector of the patent dataset. This allows users to accurately retrieve patents similar to the image to be retrieved without relying on keywords, thereby improving user retrieval efficiency.

[0051] By using a deep learning system to search for images and patent data from both textual and image features, the system can more accurately locate patents, compensating for the lack of comprehensive consideration of patents by searchers, and thus making the user's search more accurate. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram of the device structure of a patent image retrieval system based on deep learning, provided for one or more embodiments of this specification;

[0054] Figure 2 A flowchart illustrating a patent image retrieval method based on deep learning, provided for one or more embodiments of this specification;

[0055] Figure 3 A schematic diagram of the retrieval structure of a patent image retrieval system based on deep learning, provided for one or more embodiments of this specification;

[0056] Figure 4 This is a schematic diagram illustrating a retrieval scenario for a patent image retrieval method based on deep learning, which is one or more embodiments of this specification. Detailed Implementation

[0057] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0058] Existing patents typically undergo keyword searches before application to determine if the patent is prior art, thus avoiding duplicate applications and wasting human and material resources. However, when using keywords to search patent documents, the highly scalable nature of keywords—many keywords having the same or similar meanings, each corresponding to different patents—leads to problems such as long search times, inaccurate results, and low efficiency when searching for similar patents among numerous options.

[0059] To overcome the aforementioned technical problems, this application proposes a patent image retrieval system based on deep learning, comprising:

[0060] In this embodiment of the application, the image retrieval system includes: a storage module 101, a processing module 102, a feature extraction module 103, and a calculation module 104, such as... Figure 1 As shown;

[0061] Specifically, the storage module 101, processing module 102, feature extraction module 103, and calculation module 104 are as follows:

[0062] The storage module corresponds to two server clusters: an HDFS cluster and a MongoDB cluster. The HDFS cluster stores the patent image dataset, and the MongoDB cluster stores the patent text dataset. The processing module and the feature extraction module each correspond to a Spark server cluster. The cluster servers corresponding to the processing module partition the dataset using a load balancing strategy, and the cluster corresponding to the feature extraction module is used to extract features from the data in each partition. The computation module corresponds to a server used to calculate the similarity between datasets.

[0063] Specifically, storage module 101 stores patent text datasets and patent image datasets via HDFS and MongoDB; the patent image dataset contains the illustrations in the patent specification, and the patent text dataset contains the text data in the patent excluding the illustrations; processing module 102 distributes the patent dataset and image dataset to the feature extraction module using a load balancing strategy; feature extraction module 103 obtains the feature vectors and text feature vectors of the patent illustrations and the text feature vectors and image feature vectors of the user-uploaded images to be retrieved based on OCR algorithms, ResNet models, and NLP techniques; calculation module 104 stores the feature vectors and text feature vectors of the patent illustrations using the Faiss library and calculates the cosine similarity based on the images to be retrieved.

[0064] The feature extraction module 103 includes multiple server nodes. The processing module 102 distributes the patent image dataset and the patent text dataset to each server node through a load balancing strategy. Then, each server node processes the patent data and image data in parallel and extracts their text feature vectors and image feature vectors.

[0065] In this embodiment of the application, the business module 105 is used to receive the image to be searched and upload the image to be searched to the processing module; according to the similarity obtained by the calculation module, the corresponding patent image data and patent text data are obtained from the HDFS and MongoDB databases; and the obtained patent image data and obtained patent text data are combined into a complete patent.

[0066] Specifically, a similarity threshold is used to compare the similarity of text and images, thereby determining the degree of similarity between the image to be retrieved and the patents in the patent feature database. For example, when identifying images, the similarity threshold is set to 0.8-0.9. When the similarity between two images exceeds this threshold, their technical solutions are considered to be the same.

[0067] Specifically, business module 105 is also used to sort patent data according to similarity and to display the acquired patent data based on a similarity threshold.

[0068] The storage module 101 is specifically used to store the patent image dataset through the HDFS distributed file system and obtain the image storage path; and to store text data and the storage path of the patent images in HDFS through the MongoDB distributed document storage database.

[0069] The processing module 102 is specifically used to create an elastic distributed dataset RDD; read patent image data and storage paths from HDFS into various partitions of the RDD; obtain the patent text data corresponding to the patent image data based on the storage paths and the MongoDB database; obtain the patent text data into various partitions of the RDD; and match the data in the partitions to the feature extraction module through a load balancing strategy.

[0070] The feature extraction module 103 includes a patent retrieval unit. The patent retrieval unit is used to extract the patent text dataset from the patent image dataset in each partition of the RDD using an OCR algorithm; to obtain the patent figure feature vector using a ResNet model and the patent image dataset in each partition of the RDD; to obtain the first patent text data feature vector and the second patent text data feature vector using NLP technology and the patent text dataset in each partition of the RDD; and to fuse the first patent text feature vector and the second patent text feature vector based on the feature vector multiplication and fusion formula to obtain the patent text feature vector.

[0071] The feature extraction module 103 also includes a user image retrieval unit; the user image retrieval unit is used to extract text data of the image to be retrieved through an OCR algorithm; obtain a text feature vector through NLP technology and the text data of the image to be retrieved; and obtain an image feature vector through the image to be retrieved using a ResNet model.

[0072] The computing module 104 is equipped with a Faiss library, which includes computing units and storage units;

[0073] The storage unit is specifically used to: store the feature vectors of the patent drawings and the feature vectors of the patent text based on the Faiss library;

[0074] The calculation unit is specifically used to: calculate the similarity between the feature vector of the patent drawing and the feature vector of the image;

[0075] Calculate the similarity between the patent text feature vector and the text feature vector;

[0076] The similarity is calculated using the following method:

[0077]

[0078] Among them, A i B represents the feature vector of patent figures or the feature vector of patent text in the existing Faiss library; i This represents the image feature vector or text feature vector of the image to be retrieved.

[0079] This application proposes a patent image retrieval method based on deep learning, such as... Figure 2 As shown, it includes the following steps:

[0080] Example 1

[0081] Step S1: Based on the storage module, store the patent text dataset and the patent image dataset.

[0082] Specifically, global patent data and specification drawing datasets are acquired and stored separately.

[0083] In this embodiment, due to the rapid development of the patent industry in recent years, the amount of patent data generated is enormous. Using traditional relational databases to store such large-scale patent data has many drawbacks. Simply storing a massive amount of patent data on a single server can cause excessive load on the backend server, potentially leading to its crash, when the number of users simultaneously searching for patents increases dramatically. Furthermore, storing such a large amount of patent data on a single web server makes it difficult to effectively guarantee the security and reliability of the patent data. Therefore, based on the analysis of the above problems and combined with currently popular distributed storage technologies for big data, this application adopts an HDFS+MongoDB distributed storage method that can be deployed on a Spark cluster to store these tens of thousands of patent data entries.

[0084] In HDFS (Hadoop Distributed File System), storage is controlled by a master node (NameNode) and multiple slave nodes (DataNodes). Clients submit requests to the NameNode, which then assigns tasks to the DataNodes to fulfill user needs. The NameNode is the Master node in the HDFS master-slave architecture; there is only one NameNode in each HDFS system. This master node is primarily responsible for managing the namespace of files, metadata information of data blocks, client access to files, and the specific tasks assigned to data nodes. DataNodes are the slave nodes in the HDFS master-slave architecture. Each HDFS system can contain multiple DataNodes, whose main function is to store data block information. Each DataNode contains multiple data blocks and replicas of data blocks from other data nodes. If HDFS is used solely for storing proprietary data, a single point of failure (SPO) bottleneck can easily occur in practical applications. In HDFS, the NameNode acts as a communication bridge between clients and DataNodes. When a user sends an image search request to the NameNode, the NameNode locates the corresponding DateNode for similar patent information based on the user's request. Only then does the user interact with the DateNode to exchange specific data. Each image search operation involves interaction between the NameNode and DateNode. If a large number of users simultaneously submit requests to the cluster from different PCs, the performance of a single NameNode can become a bottleneck affecting the entire cluster.

[0085] Based on the problems encountered in the storage of the aforementioned patent data, this application adopts the following approach to address them.

[0086] Data storage method is as follows:

[0087] A patent database contains data such as publication number, application number, claims, specification, and drawings. Due to the single-point bottleneck problem in HDFS, MongoDB (a distributed document storage database) is used to optimize the single-node bottleneck problem in the HDFS distributed storage scheme. Therefore, the patent data is stored in MongoDB and HDFS respectively to improve the access efficiency of patent documents.

[0088] Specifically, this optimization scheme mainly involves transforming the original single-master-node architecture in the cluster into a multi-master-node architecture. Since the accompanying drawings in the patent specification occupy a significant amount of space, these drawings are stored in HDFS. File sizes in HDFS are typically in the GB to TB range. HDFS is designed to support large file storage, increasing storage space while also improving access efficiency. Based on these reasons, this application designs the storage of the patent specification drawings in HDFS. This allows for direct loading of images from HDFS when retrieving patent data, resulting in faster and more efficient image processing and better adaptation to distributed computing frameworks.

[0089] In this embodiment, patent data, excluding the accompanying drawings, is stored in MongoDB. MongoDB is a product between relational and non-relational databases, using efficient binary data storage to achieve efficient data storage. This application stores textual data from global patent data in MongoDB and persists all data information, such as the storage path of each image in HDFS, to the MongoDB database, enabling each master node in HDFS to establish communication with the MongoDB database. This storage method ensures the integrity of the entire cluster's data information while significantly reducing the burden on individual master nodes, thereby achieving efficient storage and retrieval of patent data. Specifically, the HDFS storage path is stored as a document, and each document contains a unique document ID, for example, using the publication number as the patent's document ID.

[0090] Specifically, for each storage operation, the storage path shown in the diagram in the HDFS manual is recorded, and the folder name is used as the publication number. For example, the HDFS storage path is:

[0091] hdfs: / / <hdfs-namenode> : <port>images / CN202340643A / attached Figure 1 .jpg.

[0092] Once the images are stored, each image has an absolute path, which includes the current path plus a relative path. The current path is the same for all images and is not stored, saving storage space. For example: hdfs: / / <hdfs-namenode> : <port>images / ; relative paths are used to distinguish different images from different patents, such as: / CN202340643U / attached Figure 1 The image is saved as a .jpg file, and the relative path is stored in MongoDB. The publication number and the relative path of the image are then linked, allowing the patent to be searched by publication number or path.

[0093] Step S2: Based on the processing module and the feature extraction module, extract the feature vectors of the patent figure and the patent text in parallel.

[0094] Specifically, due to the large amount of patent data, in order to meet the high-speed processing requirements of feature extraction and shorten the feature extraction processing time of images and text, this application adopts the Spark computing engine to establish parallel processing based on distributed storage.

[0095] Feature extraction of patent data based on Spark, such as Figure 3 As shown, when creating multiple RDDs (Resilient Distributed Datasets), each RDD contains multiple partitions. When data processing begins, Spark reads data from HDFS and MongoDB, forming two storage blocks: Block 1 and Block 2. The initial RDDs in these two blocks are A and C, respectively. RDD A is processed into RDD B using a flatMap operation, and RDD C is processed into RDD D using a map operation. Then, the reduceByKey function merges the RDDs into RDD E. After Block 1 and Block 2 are processed, Block 3 performs a join operation on the results of RDD B and RDD E, shuffling them to form RDD F. RDD F contains the processed results, and finally, the results in RDD F are stored in the corresponding distributed database using the saveAsSequenceFile method.

[0096] Specifically, Spark functions are used to read the attached images from HDFS into various partitions of the RDD. The path of each attached image in HDFS to its corresponding partition in the RDD is obtained based on the `InputSplit` parameter. A block in HDFS is loaded as a partition. The data in each partition is then distributed to the appropriate server nodes in the cluster for processing using a load balancing strategy, fully utilizing the cluster's computing power. Each node uses the `mapPartitions` function to batch process the storage path of the attached images in each partition. Based on the storage path of each attached image, the corresponding patent text data in MongoDB is searched, and the specific implementation method is located based on the patent text data to obtain the text description related to the image. The patent data stored in the partition is obtained and calculated using the partitions in the RDD. Finally, based on the nodes of the Spark cluster, features are extracted from the patent image data and the corresponding patent text data to obtain the patent attached image feature vector and the first patent text data feature vector. Then, features are extracted from the patent text data in the patent image data to obtain the second patent text data feature vector. Finally, based on the feature vector multiplication and fusion formula, the first and second patent text feature vectors are fused to obtain the patent text feature vector. Meanwhile, the patent publication number is used as the document ID to identify the patent. The extracted feature vectors of the patent figures and the patent text are named with the publication number. The extracted feature vectors are then stored in the Faiss database and MongoDB.

[0097] For example, given a global patent dataset, Hadoop-HDFS stores the illustrations of 1 million patents, and MongoDB stores the text data of 1 million patents, a Spark cluster with 5 nodes is built. An RDD with 10 partitions is created, loading 128MB blocks of data into each partition at a time and obtaining the storage path of the illustrations in each partition. A load balancing strategy is then used to match server nodes to each partition. The server nodes use the storage path of each illustration in the partition to find the text data of each illustration in MongoDB, obtaining the text description of the illustration in the specific implementation. Feature extraction is then performed on both the illustration and the text separately. The extracted features are named with the publication number during storage. The features extracted from HDFS and MongoDB are then merged based on the publication number. If a patent illustration in HDFS can yield both illustration and text features, and the same patent also has only one text feature extracted from MongoDB, then the text features from HDFS and MongoDB need to be merged so that the patent has only one text feature and one illustration feature. These two features are then stored in the patent feature database.

[0098] The methods for extracting image features and text features in the attached figure are as follows:

[0099] Patents typically describe solutions using a combination of illustrations and text; therefore, the illustrations contain both images and textual descriptions. To improve the accuracy of image retrieval, feature extraction needs to be performed on both the images and the corresponding text within the illustrations.

[0100] First, the accompanying drawings in the patent are identified. The patent contains descriptions of the drawings and detailed embodiments. The descriptions indicate the subject and icons of the images, while the detailed embodiments describe the content of the images. Additionally, the phrase "as shown in Figure n" appears in the detailed embodiments. Based on these patent characteristics, semantic analysis can be used to determine the text portions corresponding to each drawing in the patent document, as well as the individual drawings themselves. Then, feature extraction is performed on each identified drawing and its corresponding text portion to obtain the corresponding image and text features.

[0101] For example, the specific implementation method is found based on the image name, and then NLP semantic analysis technology is used to extract text data 1 associated with the image. The text in the accompanying drawings of the instruction manual is extracted using the OCR algorithm to obtain the accompanying drawing text data 2. The Word2Vec word vector model in NLP semantic analysis technology is used to extract features from text data 1 and accompanying drawing text data 2 respectively, generating text feature vector 1 and accompanying drawing text feature vector 2.

[0102] It should be noted that the attached images also contain text, which is crucial for understanding the images. During feature extraction, these text elements in the attached images also need to be extracted to obtain corresponding text feature vectors.

[0103] The feature extraction process for the attached figures and text according to the above method is as follows: When data processing begins, an RDD distributed dataset is created in Spark. Text and image data are read from the data storage space, including HDFS and MongoDB, through the RDD. The text and image data generation tasks are pushed to each node. Each node scales, normalizes, and crops the images to ensure image consistency. Then, based on the image storage path, the text data associated with the image in the specific implementation is located and semantic analysis is performed using NLP semantic analysis technology to obtain image-related text data 1. Next, the text in the attached figures is extracted using an OCR algorithm to obtain attached figure text data 2. The Word2Vec word vector model is used to extract features from text data 1 and attached figure text data 2, generating text feature vector 1 and attached figure text feature vector 2, respectively. Finally, a pre-trained ResNet (Residual Network) model is loaded using the deep learning framework PyTorch. The attached figures are input into the ResNet model, and the image feature vectors of the required layers are extracted. Pre-trained models are typically trained on large-scale image datasets and have powerful image feature extraction capabilities.

[0104] Step S3: Based on the calculation module, calculate the similarity and construct a patent feature database.

[0105] Specifically, the Faiss library is constructed as a patent feature database. Text feature vectors and image feature vectors are stored in the Faiss library and MongoDB. The Faiss library is used for similarity calculation. This application adopts the cosine similarity algorithm to calculate the similarity of feature vectors based on cosine similarity. The feature vectors are stored in MongoDB for backup to prevent data loss.

[0106] The Faiss library is primarily used for large-scale similarity search and dense vector analysis. It includes various similarity search methods and provides CPU and GPU versions. The Faiss library offers a series of algorithms and data structures. This application compares various similarity algorithms through experiments to obtain a similarity algorithm suitable for this scheme. This algorithm improves the retrieval speed of vector similarity and reduces memory usage with minimal loss of precision.

[0107] Step S4: Search for patents based on business modules.

[0108] Specifically, users upload images to the server via the client. After receiving the images, the server uses Spark to extract features. In Spark, the images uploaded by the client are loaded based on RDDs and distributed to server nodes. Then, a custom algorithm is called to implement image feature extraction. When recognizing the client's images, both OCR (Optical Character Recognition) and ResNet are used to extract features from the images simultaneously. The images are divided into the following three scenarios.

[0109] Only text: It can use OCR to recognize the text in the image, use the result of OCR as the text information of the image, and then use the Word2Vec word vector model to extract the features of the text information to obtain the text feature vector; but it cannot use the ResNet model to extract the feature vector of the image.

[0110] Image only: The OCR algorithm failed to identify text in the image; then the ResNet model was used to identify image features and obtain the image's feature vector.

[0111] The image contains both text and images: The OCR algorithm is used to recognize the text in the image and extract the text information. Then, the Word2Vec word vector model is used to extract the features of the text information to obtain the text feature vector. Finally, the ResNet model is used to recognize the image features and obtain the image feature vector.

[0112] The aforementioned text feature vectors and image feature vectors are input into the patent feature database. Based on the feature vectors in the patent feature database, the text feature vectors and image feature vectors extracted from the processed images uploaded by the user are compared to calculate the cosine similarity. The text feature vectors in the patent feature database are compared with the user's text feature vectors, and the image feature vectors in the patent feature database are compared with the user's image feature vectors. A cosine similarity threshold is set, and the similarity is sorted according to the threshold. The similar patents are then returned to the client.

[0113] For example: If the customer uploads image A, such as Figure 4 As shown, the image retrieval process can be divided into the following scenarios:

[0114] 1. If only text is identified in Figure A: Extract the text feature vector and compare it with the text feature vectors of Patent 1, Patent 2, Patent 3, etc. in the patent feature database. If Patent 1 contains three attached figures (Figure a, Figure b, and Figure c), calculate the cosine similarity between the text feature vector of Figure a and the text feature vector of Figure A. The similarity is 60%. Then, calculate the cosine similarity between the text feature vector of Figure a and the text feature vector of Figure A. The similarity is 70%. Calculate the cosine similarity between the text feature vector of Figure b and the text feature vector of Figure A. The similarity is... 75%. Figure b has image feature vectors, but Figure A does not, so the similarity of the image feature vectors is not compared. The text feature vector of Figure c is compared with the text feature vector of Figure A, and the similarity is 75%. The similarity of the text features in the attached figure is 85%. There are multiple similarities in Patent 1, and the largest percentage is selected as the similarity between Patent 1 and Figure A. Therefore, the similarity of Patent 1 is 85%. If the text similarity of Patent 2 is 90% and the text similarity of Patent 3 is 70%, then the patent similarities are sorted. Therefore, the display order returned to the client is Patent 2, Patent 1, Patent 3, etc.

[0115] 2. If Figure A contains both text and images: Extract text feature vectors and image feature vectors. Calculate the similarity between the text feature vectors and image feature vectors in Figure A and the text feature vectors and image feature vectors in the patent feature library, such as Patent 1, Patent 2, and Patent 3. In Patent 1, the text feature similarity is 90%, and the image feature similarity is 80%. In Patent 2, the text feature and image feature similarity are 80% and 90%, respectively. In Patent 3, the text feature and image feature similarity are 80% and 80%, respectively. In Patent 4, the text feature and image feature similarity are 50% and 70%, respectively. First, sort the similarities from largest to smallest based on the text feature similarity. If the text feature similarities are the same, sort by image similarity. Then, sort the similarities based on a similarity threshold. If the similarity threshold is 70%, the display order returned to the client will be Patent 1, Patent 2, Patent 3, Patent 4, etc.

[0116] 3. If only image A is identified: Extract the image feature vector. Calculate the similarity between the image feature vector and the image feature vectors of patents 1, 2, 3, etc. in the patent feature database. Select the highest similarity among the patents 1, 2, 3, etc., and use it as the similarity between image A and the patents 1, 2, 3, etc. The similarity of patent 1 is 20%, patent 2 is 45%, patent 3 is 80%, etc. If the similarity threshold is 70%, then patents with a similarity greater than 70% are ranked first by percentage, and the patent data is returned to the client in descending order. Therefore, the display order on the client is patent 3, patent 2, patent 1, etc.

[0117] Example 2

[0118] Step S1: The client uploads the image to be searched.

[0119] Specifically, patent search methods are not limited to using patent search clients, WeChat mini-programs, or browsers or client terminals used to log in to patent search websites.

[0120] For example, terminal devices may include, but are not limited to: mobile phones, tablets, laptops, PCs, smart home appliances, smart wearable devices, etc.

[0121] Specifically, the client can upload one or more images by selecting the path and inserting the images in the input box, and then clicking upload.

[0122] Step S2: The backend server receives the image.

[0123] Specifically, the server retrieves the images to be searched and categorizes them.

[0124] The text information of the image to be retrieved is extracted using the OCR algorithm, and then the image features of the image to be retrieved are identified using the ResNet model. Based on the above operations, the image to be retrieved is divided into the following three scenarios and processed accordingly.

[0125] The text information of the image to be retrieved can be extracted but the image features cannot be identified: The text of the image to be retrieved is extracted based on the OCR algorithm, and then the Word2Vec word vector model in NLP semantic analysis technology is used to extract features to obtain the text feature vector.

[0126] The image to be retrieved cannot be used to extract text information but can identify image features: Based on the ResNet model, image recognition is performed on the image to be retrieved to obtain image feature vectors.

[0127] The image to be retrieved can extract both text information and image features: First, text is extracted from the image to be retrieved based on the OCR algorithm, then features are extracted using the Word2Vec word vector model in NLP semantic analysis technology to obtain text feature vectors; then, image recognition is performed on the image to be retrieved based on the ResNet model to obtain image feature vectors.

[0128] Step S3: Calculate similarity.

[0129] Specifically, the similarity between the text feature vectors and image feature vectors in the patent feature database and the text feature vectors and image feature vectors obtained from the image to be retrieved is calculated, and the patents are sorted by similarity to return the document ID that is most similar to the query vector.

[0130] The process involves comparing the text feature vector of the image to be retrieved with the text feature vectors in the patent feature database and calculating the cosine similarity. If a patent contains multiple figures, each figure corresponds to a text feature vector. The figures with text feature vectors in a patent are compared with the text feature vectors in the image to be retrieved, and the cosine similarity is calculated. A patent will generate multiple text feature similarity values. These similarities are then ranked, and the highest similarity is selected as the similarity between the patent and the image to be retrieved. In this application, a higher similarity value indicates a closer resemblance between the patent and the image to be retrieved, and thus, the higher the similarity value will be displayed on the client-side.

[0131] The database consists entirely of images. The image feature vectors of the image to be searched are compared with image feature vectors in the patent feature database, and a cosine similarity is calculated. If a patent has multiple attached images, each with an image feature vector, the attached images containing the image feature vectors of the patent are compared with the image feature vectors of the image to be searched, and a cosine similarity is calculated. A patent will generate multiple image feature similarity values. These similarities are then ranked, and the highest similarity is selected as the similarity between the patent and the image to be searched. In this application, a higher similarity value indicates a closer resemblance between the patent and the image to be searched, and thus, the higher the similarity value is displayed on the client-side.

[0132] The system contains both text and images. The text feature vector and image feature vector of the image to be retrieved are compared with the text feature vector and image feature vector in the patent feature database, and the cosine similarity is calculated. If a patent has multiple illustrations in its specification, each illustration containing either text or image feature vectors, the process is as follows: First, the text feature vector of the image to be retrieved is compared with the text feature vectors in the patent database, and a cosine similarity is calculated. This yields multiple text feature similarity values ​​for the patent. These similarities are then sorted, and the highest similarity among the text features is selected. Next, the image feature vector of the image to be retrieved is compared with the image feature vectors in the patent feature database, and a cosine similarity is calculated. This yields multiple image feature similarity values ​​for the patent. These similarities are then sorted, and the highest similarity among the image features is selected. The combined text and image feature similarity values ​​of the patent are used as the similarity between the patent and the image to be retrieved. Finally, the text and image feature similarity values ​​are sorted, and a similarity threshold is set. If the cosine similarity of both the text and image feature vectors is greater than the threshold, the patent is displayed first. If either the text or image feature vector has a cosine similarity greater than the threshold, the patent is displayed in the middle.

[0133] Based on the above method, this application compares the text or image features represented by each figure in the patent, as well as the text features represented by each figure in a specific embodiment, with the text or image features of the image to be retrieved, and calculates the cosine similarity. This method can accurately extract the meaning expressed by the image to be retrieved and the meaning expressed by the patent figures, and then compare their similarity. The similarity value of the figure with the highest similarity to the image to be retrieved is selected as the similarity between the patent and the image to be retrieved. This method makes patent retrieval more accurate and efficient. Then, using the returned document ID, the corresponding patent data is queried from MongoDB (this can be achieved through the mapping between document IDs stored in the Faiss index and document IDs in MongoDB), and based on the storage path of the figures in MongoDB, the figure information in HDFS is obtained, thereby obtaining complete patent data. Finally, the basic information of the patent data and the feature vector similarity are returned and presented to the user.

[0134] For example, a customer uploads a patent flowchart to the front-end client. The back-end server receives the image, uses an OCR algorithm to extract the text from the image, and then uses the Word2Vec word vector model to extract features from the extracted text, generating text feature vectors. Next, it uses a ResNet deep learning network model to extract image feature vectors from the patent image. This text feature vector and image feature vector are then compared with vectors in the patent feature database using cosine similarity. The similarity is sorted from largest to smallest, and then similar patents are presented to the user in order.

[0135] The formula for calculating cosine similarity is as follows:

[0136]

[0137] Among them, A i B represents the feature vector of patent figures or the feature vector of patent text in the existing Faiss library; i This represents the image feature vector or text feature vector of the image to be retrieved.

[0138] If a customer uploads an image, the OCR algorithm is used to extract the text from the image to be retrieved. Then, the Word2Vec word vector model is used to extract features from the extracted text to obtain text feature vectors. Then, the ResNet deep learning network model is used to extract image feature vectors from the image to be retrieved. If the cosine similarity between the text feature vector and the image feature vector of Patent 1 in the feature library is 90% and 80% respectively, and the cosine similarity between the text feature vector and the image feature vector of Patent 2 in the feature library is 80% and 40% respectively, and the similarity threshold is 80%, then the similarity ranking is Patent 1, Patent 2, etc.

[0139] This application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the image retrieval system instructions described in any of the above embodiments.

[0140] It should be noted that the embodiments concerning storage media in this specification and the embodiments concerning blockchain-based service provision methods in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding blockchain-based service provision method described above, and the repeated parts will not be described again.

[0141] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0142] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0143] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0144] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0145] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0146] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0155] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0156] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.< / port> < / hdfs-namenode> < / port> < / hdfs-namenode>

Claims

1. A deep learning-based patent image retrieval system, characterized by, include: Storage module, processing module, feature extraction module, and computing module; The storage module is used to store patent image datasets via HDFS and patent text datasets via MongoDB; the patent image datasets include the drawings in the patent specification, and the patent text datasets include the text data in the patent other than the drawings in the specification. The processing module is used to distribute the patent image dataset and the patent text dataset to the feature extraction module through a load balancing strategy; The feature extraction module is used to extract the patent figure feature vector of the patent image dataset and the patent text feature vector of the patent text dataset according to the OCR algorithm, ResNet model and NLP technology; and to extract the text feature vector and image feature vector of the user-uploaded image to be retrieved according to the OCR algorithm, ResNet model and NLP technology. The calculation module is used to calculate the similarity between the image to be retrieved and the patent in the storage module based on the feature vector of the patent figure drawing, the feature vector of the patent text, the text feature vector, and the image feature vector. 2.The deep learning-based patent image retrieval system of claim 1, wherein, It also includes a business module, which is used to receive the image to be searched and upload the image to be searched to the processing module; obtain the corresponding patent image data and patent text data from HDFS and MongoDB databases based on the similarity obtained by the calculation module; and combine the obtained patent image data and the obtained patent text data into a complete patent.

3. The patent image retrieval system based on deep learning according to claim 2, characterized in that, The aforementioned business module is further used for: The obtained patents are sorted according to a preset similarity threshold.

4. The patent image retrieval system based on deep learning according to claim 1, characterized in that, The storage module is specifically used for: The patent image dataset is stored using the HDFS distributed file system, and the image storage path is obtained. The storage paths for patent text datasets and patent image datasets in HDFS are stored using the MongoDB distributed document storage database.

5. The patent image retrieval system based on deep learning according to claim 4, characterized in that, The processing module is specifically used for: Create a resilient distributed dataset (RDD); Read the patent image data and the image storage path from HDFS into each partition of the RDD; Based on the image storage path and the MongoDB database, the patent text data corresponding to the patent image data is obtained; Obtain the patent text data into each partition of the RDD; The load balancing strategy distributes each partition to the feature extraction module. 6.The deep learning-based patent image retrieval system according to claim 5, wherein, The feature extraction module includes a patent retrieval unit; The RDD partition includes a patent image dataset and a first patent text dataset; The patent search unit is used for: The second patent text dataset is extracted from the patent image dataset in each partition of the RDD using the OCR algorithm. Based on the ResNet model, feature extraction is performed on the patent image dataset in each partition of the RDD to obtain the feature vector of the patent figure. Based on NLP technology, feature extraction is performed on the first patent text dataset and the second patent text dataset in each partition of the RDD to obtain the feature vector of the first patent text data and the feature vector of the second patent text data. Based on the feature vector multiplication and fusion formula, the first patent text feature vector and the second patent text feature vector are fused to obtain the patent text feature vector.

7. The patent image retrieval system based on deep learning according to claim 1, characterized in that, The feature extraction module also includes a user image retrieval unit; The user image retrieval unit is used to extract text data from the image to be retrieved using an OCR algorithm; Based on NLP technology, feature extraction is performed on the text data in the image to be retrieved to obtain a text feature vector. Based on the ResNet model, feature extraction is performed on the image to be retrieved to obtain an image feature vector.

8. The patent image retrieval system based on deep learning according to claim 1, characterized in that, The computing module is pre-configured with the Faiss library, and the computing module includes a computing unit and a storage unit. The storage unit is specifically used for: Based on the Faiss library, the feature vectors of the patent drawings and the feature vectors of the patent text are stored. The computing unit is specifically used for: Calculate the similarity between the feature vector of the patent figure drawing and the feature vector of the image; Calculate the similarity between the patent text feature vector and the text feature vector; The similarity is calculated using the following method: Wherein, A i represents a patent drawing feature vector or a patent text feature vector in the existing Faiss library; B i represents the image feature vector or the text feature vector of the picture to be retrieved.

9. A deep learning based patent image retrieval method based on the system of any one of claims 1-8, characterized in that, include: The patent image dataset is stored using HDFS; the patent text dataset is stored using MongoDB; the patent image dataset includes the drawings in the patent specification, and the patent text dataset includes the text data in the patent other than the drawings in the specification. The patent image dataset and the patent text dataset are allocated to the feature extraction module using a load balancing strategy. Based on OCR algorithms, ResNet models, and NLP techniques, feature vectors of patent figures in the patent image dataset and feature vectors of patent text in the patent text dataset are extracted; text feature vectors and image feature vectors of user-uploaded images to be retrieved are extracted based on OCR algorithms, ResNet models, and NLP techniques. Based on the feature vectors of the patent drawings, the patent text, the text feature vectors, and the image feature vectors, the similarity between the image to be retrieved and the patents in the storage module is calculated.

10. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the image retrieval system instructions according to any one of claims 1-8.

Images