Image Search System

Abstract

To provide an image retrieval system with high retrieval accuracy.SOLUTION: An image retrieval system comprises a database and a processing unit. The database has a function of storing a plurality of pieces of database image data, and a database tag is associated with each piece of the database image data. The processing unit has a function of acquiring database image feature amount data representing a feature amount of the database image data for each piece of database image data. Further, the processing unit has a function of acquiring query image feature amount data representing a feature amount of query image data. Furthermore, the processing unit has a function of calculating, for each piece of database image data, a first similarity that is a similarity of the database image data to the query image data. The processing unit has a function of acquiring a query tag that is a tag associated with the query image data by using a part of the database tag.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] One aspect of the present invention relates to an image search system and an image search method.

[0002] One embodiment of the present invention is not limited to the above technical field, but examples of the technical field of one embodiment of the present invention include semiconductor devices, display devices, light-emitting devices, power storage devices, memory devices, electronic devices, lighting devices, driving methods thereof, and manufacturing methods thereof. [Background technology]

[0003] By conducting a prior art search for an invention before filing, you can investigate whether any related intellectual property rights exist. The prior art documents, such as domestic and international patent documents and papers, obtained through a prior art search can be used to confirm the novelty and inventive step of the invention and to decide whether to apply for a patent. In addition, by conducting an invalidation document search of prior art documents, you can investigate whether your own patent rights are at risk of being invalidated, or whether you can invalidate patent rights owned by others.

[0004] For example, the prior art search can be conducted by searching for prior art documents that contain drawings similar to the drawings embodying the pre-filing technology. Specifically, for example, by inputting drawings into an image search system, a user can search for prior art documents that contain drawings similar to the input drawings.

[0005] A search for images similar to an input image can be performed using, for example, a neural network. For example, Patent Document 1 discloses a method for determining the similarity between images using a neural network. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-207947 Summary of the Invention [Problem to be solved by the invention]

[0007] If the similarity between an input image and a search target image is calculated using only image data, for example, an image with a different concept from the input image may be retrieved. This may result in noise images being mixed into the search results, and the desired image may not be output. This may result in a decrease in the accuracy of the search for similar images.

[0008] In view of the above, an object of one embodiment of the present invention is to provide an image retrieval system with high search accuracy.Another object of one embodiment of the present invention is to provide an image retrieval system that can perform a search in a short time.Another object of one embodiment of the present invention is to provide an image retrieval system that can easily perform a search.Another object of one embodiment of the present invention is to provide a novel image retrieval system.

[0009] Another object of one embodiment of the present invention is to provide an image search method with high search accuracy.Another object of one embodiment of the present invention is to provide an image search method that allows searches to be performed in a short time.Another object of one embodiment of the present invention is to provide an image search method that allows searches to be performed easily.Another object of one embodiment of the present invention is to provide a novel image search method.

[0010] Note that the description of these problems does not preclude the existence of other problems. One embodiment of the present invention does not necessarily have to solve all of these problems. Problems other than these can be extracted from the description in the specification, drawings, and claims. [Means for solving the problem]

[0011] One aspect of the present invention includes a database, a processing unit, and an input unit, wherein the database has a function of storing document data and a plurality of database image data, the processing unit has a function of acquiring database image feature data representing features of the database image data for each of the plurality of database image data, the processing unit has a function of generating a plurality of database tags using the document data and linking the database tags to the database image data, the processing unit has a function of acquiring a database tag vector representing the database tag for each of the plurality of database tags, and the processing unit has a function of acquiring query image feature data representing features of the query image data when query image data is input to the input unit, is an image search system having a function of calculating a first similarity, which is the similarity of database image data to query image data, for each of multiple database image data; a processing unit having a function of acquiring a query tag linked to the query image data using a part of the database tag based on the first similarity; a processing unit having a function of acquiring a query tag vector representing the query tag; a processing unit having a function of acquiring first data including database image feature data and the database tag vector; a processing unit having a function of acquiring second data including the query image feature data and the query tag vector; and a processing unit having a function of calculating a second similarity, which is the similarity of the first data to the second data.

[0012] Alternatively, in the above aspect, the database tag may include a word.

[0013] Alternatively, in the above aspect, the processing unit may have a function of generating database tags by performing morphological analysis on the document data.

[0014] Alternatively, in the above aspect, the processing unit may have a first neural network and a second neural network, and the database image feature data and the query image feature data may be obtained using the first neural network, and the database tag vector and the query tag vector may be obtained using the second neural network.

[0015] Alternatively, in the above aspect, the first neural network may have a convolutional layer and a pooling layer, and the database image feature data and the query image feature data may be output from the pooling layer.

[0016] Alternatively, in the above aspect, the database tag vector and the query tag vector may be distributed representation vectors.

[0017] Alternatively, in the above aspect, the first similarity and the second similarity may be cosine similarity.

[0018] Alternatively, one aspect of the present invention is an image retrieval method using an image retrieval system having a database in which document data and a plurality of database images are stored, and an input unit, the image retrieval method comprising the steps of: acquiring, for each of the plurality of database image data, database image feature data representing features of the database image data; generating a plurality of database tags using the document data; linking the database tags to the database image data; acquiring, for each of the plurality of database tags, database tag vectors representing the database tags; inputting query image data to the input unit; acquiring query image feature data representing the features of the query image data; calculating, for each of the plurality of database image data, a first similarity which is the similarity of the database image data to the query image data; acquiring, based on the first similarity, a query tag linked to the query image data using a portion of the database tag; acquiring a query tag vector representing the query tag; acquiring first data including the database image feature data and the database tag vector, and second data including the query image feature data and the query tag vector; and calculating a second similarity which is the similarity of the first data to the second data.

[0019] Alternatively, in the above aspect, the database tag may include a word.

[0020] Alternatively, in the above aspect, database tags may be generated by performing morphological analysis on document data.

[0021] Alternatively, in the above aspect, the database image feature amount data and the query image feature amount data may be obtained using a first neural network, and the database tag vector and the query tag vector may be obtained using a second neural network.

[0022] Alternatively, in the above aspect, the first neural network may have a convolutional layer and a pooling layer, and the database image feature data and the query image feature data may be output from the pooling layer.

[0023] Alternatively, in the above aspect, the database tag vector and the query tag vector may be distributed representation vectors.

[0024] Alternatively, in the above aspect, the first similarity and the second similarity may be cosine similarity. [Effects of the Invention]

[0025] According to one aspect of the present invention, an image retrieval system with high search accuracy can be provided. Alternatively, according to one aspect of the present invention, an image retrieval system capable of performing a search in a short time can be provided. Alternatively, according to one aspect of the present invention, an image retrieval system capable of performing a search simply can be provided. Alternatively, according to one aspect of the present invention, a novel image retrieval system can be provided.

[0026] Alternatively, one aspect of the present invention can provide an image retrieval method with high retrieval accuracy. Alternatively, one aspect of the present invention can provide an image retrieval method that allows retrieval to be performed in a short time. Alternatively, one aspect of the present invention can provide an image retrieval method that allows retrieval to be performed simply. Alternatively, one aspect of the present invention can provide a novel image retrieval method.

[0027] Note that the description of these effects does not preclude the existence of other effects. One embodiment of the present invention does not necessarily have all of these effects. Effects other than these can be extracted from the description in the specification, drawings, and claims. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image search system. [Figure 2] FIG. 2 is a flowchart showing an example of a search data generation method. [Figure 3] 3A and 3B are diagrams showing examples of the configuration of a neural network. [Figure 4] FIG. 4 is a diagram illustrating an example of the convolution process and the pooling process. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of a neural network. [Figure 6] 6A and 6B are diagrams showing an example of a method for generating search data. [Figure 7] Fig. 7A is a diagram showing an example of a method for generating search data, and Fig. 7B is a diagram showing an example of the configuration of a neural network. [Figure 8] 8A and 8B are diagrams showing an example of a method for generating search data. [Figure 9] FIG. 9 is a flowchart showing an example of an image search method. [Figure 10] FIG. 10 is a diagram showing an example of an image search method. [Figure 11] 11A and 11B are diagrams showing an example of an image search method. [Figure 12] 12A and 12B are diagrams showing an example of an image search method. [Figure 13] FIG. 13 is a diagram showing an example of an image search method. [Figure 14] FIG. 14 is a flowchart showing an example of an image search method. [Figure 15] FIG. 15 is a diagram showing an example of an image search method. [Figure 16] 16A and 16B are diagrams showing an example of an image search method. [Figure 17] FIG. 17 is a flowchart showing an example of an image search method. [Figure 18] 18A and 18B are diagrams showing an example of an image search method. [Figure 19] FIG. 19 is a diagram showing an example of an image search method. [Figure 20] 20A, 20B1, and 20B2 are diagrams showing an example of an image search method. [Figure 21] 21A and 21B are diagrams showing an example of an image search method. [Figure 22] 22A and 22B are diagrams showing an example of an image search method. [Figure 23] FIG. 23 is a flowchart showing an example of an image search method. [Figure 24] 24A and 24B are diagrams showing an example of an image search method. [Figure 25] FIG. 25 is a diagram showing an example of an image search method. [Figure 26] FIG. 26 is a diagram showing an example of an image search method. DETAILED DESCRIPTION OF THE INVENTION

[0029] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be readily understood by those skilled in the art that various changes can be made in the form and details without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0030] (Embodiment 1) In this embodiment, an image search system and an image search method according to one embodiment of the present invention will be described with reference to drawings.

[0031] An image search system according to one embodiment of the present invention includes an input unit, a database, and a processing unit. The processing unit includes a first neural network and a second neural network. The first and second neural networks each include a layer having neurons.

[0032] In this specification, the term "neural network" refers to a general model that imitates the neural circuit network of a living organism, determines the connection strength between neurons through learning, and has problem-solving capabilities.

[0033] In this specification and the like, when discussing neural networks, the process of determining the connection strengths (also called weighting coefficients) between neurons from existing information is referred to as "learning."

[0034] In this specification and the like, the term "inference" refers to constructing a neural network using connection strengths obtained by learning and deriving a new conclusion from the network.

[0035] Image data is stored in the database. When a user of the image retrieval system according to one aspect of the present invention inputs image data into an input unit, the image retrieval system according to one aspect of the present invention searches the database for image data similar to the input image data and outputs the image data.

[0036] In this specification, image data stored in a database is referred to as database image data. Image data input to an input unit is referred to as query image data. Furthermore, database image data and query image data may be collectively referred to simply as image data.

[0037] An image search method using an image search system according to one embodiment of the present invention will be described below.

[0038] Image feature amount data can be acquired by inputting image data into a first neural network included in the processing unit.

[0039] In this specification, data representing the features of image data is referred to as image feature data. For example, data representing the features of database image data is referred to as database image feature data, and data representing the features of query image data is referred to as query image feature data.

[0040] The first neural network may be, for example, a convolutional neural network having a convolutional layer and a pooling layer. When the first neural network is a convolutional neural network, image data may be input to the first neural network, and data output from the pooling layer may be used as image feature data.

[0041] Furthermore, tags are linked to database image data. For example, document data linked to database image data is stored in a database, and tags can be linked by performing morphological analysis on the document data. The tags can be keywords that represent the concept, technical content, points of interest, etc. of the image corresponding to the database image data. For example, one tag can represent one word. Multiple tags can be linked to database image data.

[0042] In this specification, a tag linked to database image data is called a database tag, and a tag linked to query image data is called a query tag.

[0043] By inputting the tag into a second neural network included in the processing unit, the tag can be represented by a vector. For example, the tag can be represented by a 300-dimensional distributed representation vector.

[0044] In this specification, a vector representing a tag is called a tag vector. For example, a vector representing a database tag is called a database tag vector, and a vector representing a query tag is called a query tag vector. Furthermore, one tag vector refers to a tag vector corresponding to one tag.

[0045] In this specification, the term "vector" refers to a set of multiple values. The number of values ​​that make up a vector is called the number of dimensions. For example, a vector expressed as (5,1,4,3,2) can be said to be a five-dimensional vector. The values ​​that make up a vector are sometimes called components.

[0046] In an image retrieval method using an image retrieval system according to one aspect of the present invention, in addition to database image data, database image feature data representing the feature quantities of the database images are stored in advance in a database. Database tags associated with the database image data and database tag vectors representing the database tags are also stored in advance in the database. Note that the database tags themselves do not necessarily have to be stored in the database.

[0047] In an image retrieval method using an image retrieval system according to one embodiment of the present invention, when a user of the image retrieval system inputs query image data into an input unit, the query image data is input into a first neural network, and query image feature data is generated. Next, the database image feature data and the query image feature data are used to calculate the similarity of the database image data to the query image data. For example, cosine similarity is calculated. The calculation of the similarity to the query image data can be performed for each of, for example, all of the database image data.

[0048] Then, based on the calculation result of the similarity, a query tag is obtained using the database tag. For example, a database tag that appears frequently among database tags linked to database image data with high similarity can be used as the query tag. Note that the number of query tags can be the same as the number of database tags linked to one database image data, for example.

[0049] In this specification and the like, one piece of image data refers to image data representing one image that is displayed in one frame period, for example.

[0050] Next, first data including database image feature data and a database tag vector is acquired. Second data including query image feature data and a query tag vector is acquired. Thereafter, the similarity between the first data and the second data is calculated to correct the similarity of the database image data to the query image data. For example, the correction is performed by calculating the cosine similarity between the first data and the second data.

[0051] Here, one piece of first data may include, for example, one piece of database image feature data and a database tag vector corresponding to a database tag linked to the database image data corresponding to the database image feature data. That is, the number of first data may be the same as the number of database image feature data. Furthermore, one piece of second data may include query image feature data and the same number of query tag vectors as the database tag vectors included in one piece of first data.

[0052] Next, ranking data including information about the corrected similarity ranking is generated and output as search results to an external device of the image search system according to one embodiment of the present invention. By correcting the similarity of the database image data to the query image data in this manner, it is possible to prevent, for example, database images that have similar features to the query image but are conceptually different from the query image from being searched for. This prevents images that become noise from being mixed into the search results, preventing the desired image from being output. Therefore, the image search system according to one embodiment of the present invention can perform searches with high accuracy.

[0053] Furthermore, in an image search method using an image search system according to an embodiment of the present invention, query tags are acquired using database tags. This acquisition method is simpler than, for example, a method of acquiring query tags based on query image feature data. Therefore, the image search system according to an embodiment of the present invention can perform searches in a short time. Furthermore, the method of acquiring query tags using database tags can comprehensively acquire tags that represent the concept, technical content, points of interest, etc. of an image corresponding to query image data, compared to, for example, a method in which a user of the image search system according to an embodiment of the present invention specifies all query tags. Therefore, the image search system according to an embodiment of the present invention can perform searches simply and with high accuracy.

[0054] <1-1. Image search system> 1 is a block diagram showing an example of the configuration of an image search system 10. In the drawings attached to this specification, components are classified by function and shown as independent blocks in the block diagram, but in reality, it is difficult to completely separate components by function, and one component may be involved in multiple functions. Also, one function may be involved in multiple components; for example, multiple processes performed by the processing unit 13 may be executed by different servers.

[0055] The image retrieval system 10 has at least a processing unit 13. The image retrieval system 10 shown in FIG.

[0056] [Input section 11] Image data and the like are supplied to the input unit 11 from outside the image search system 10. The image data and the like supplied to the input unit 11 are supplied to the processing unit 13, the storage unit 15, or the database 17 via the transmission path 12. As described above, the image data input to the input unit 11 is called query image data.

[0057] [Transmission path 12] The transmission path 12 has a function of transmitting image data, etc. Information can be transmitted and received between the input unit 11, the processing unit 13, the storage unit 15, the database 17, and the output unit 19 via the transmission path 12.

[0058] [Processing section 13] The processing unit 13 has the function of performing calculations, inferences, etc. using image data, etc. supplied from the input unit 11, the storage unit 15, the database 17, etc. The processing unit 13 has a neural network and can perform calculations, inferences, etc. using the neural network. The processing unit 13 can also perform calculations, etc. without using a neural network. The processing unit 13 can supply calculation results, inference results, etc. to the storage unit 15, the database 17, the output unit 19, etc.

[0059] The processing unit 13 preferably uses a transistor having a metal oxide in its channel formation region. Because the off-state current of the transistor is extremely low, using the transistor as a switch for retaining charge (data) flowing into a capacitor element functioning as a memory element can ensure a long data retention period. By using the transistor in at least one of the register and cache memory of the processing unit 13, the processing unit 13 can be operated only when necessary, and can be turned off at other times by saving information from the previous processing in the memory element. In other words, normally-off computing is possible, enabling low power consumption in the image search system.

[0060] In this specification and the like, the term "metal oxide" refers to an oxide of a metal in a broad sense. Metal oxides are classified into oxide insulators, oxide conductors (including transparent oxide conductors), oxide semiconductors (also referred to as "oxide semiconductors" or simply as "OSs"), and the like. For example, when a metal oxide is used in a semiconductor layer of a transistor, the metal oxide may be referred to as an oxide semiconductor. In other words, when a metal oxide has at least one of an amplifying function, a rectifying function, and a switching function, the metal oxide can be referred to as a metal oxide semiconductor, or an OS for short.

[0061] In this specification and the like, a transistor whose channel formation region is formed using an oxide semiconductor or a metal oxide is referred to as an oxide semiconductor transistor or an OS transistor.

[0062] The metal oxide contained in the channel formation region preferably contains indium (In). When the metal oxide contained in the channel formation region contains indium, the carrier mobility (electron mobility) of the OS transistor is increased. Furthermore, the metal oxide contained in the channel formation region is preferably an oxide semiconductor containing element M. The element M is preferably aluminum (Al), gallium (Ga), tin (Sn), or the like. Other elements that can be used as element M include boron (B), silicon (Si), titanium (Ti), iron (Fe), nickel (Ni), germanium (Ge), yttrium (Y), zirconium (Zr), molybdenum (Mo), lanthanum (La), cerium (Ce), neodymium (Nd), hafnium (Hf), tantalum (Ta), and tungsten (W). However, a combination of the above elements may be used as element M. The element M is, for example, an element having a high binding energy with oxygen. For example, it is an element whose bond energy with oxygen is higher than that of indium. Furthermore, the metal oxide contained in the channel formation region is preferably a metal oxide containing zinc (Zn). Metal oxides containing zinc may be more likely to crystallize.

[0063] The metal oxide contained in the channel formation region is not limited to a metal oxide containing indium. The semiconductor layer may be a metal oxide containing zinc but not indium, such as zinc tin oxide or gallium tin oxide, a metal oxide containing gallium, or a metal oxide containing tin.

[0064] The processing unit 13 includes, for example, an arithmetic circuit or a central processing unit (CPU).

[0065] The processing unit 13 may have a microprocessor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The microprocessor may be implemented by a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array). The processing unit 13 can perform various data processing and program control by interpreting and executing instructions from various programs using the processor. Programs that can be executed by the processor are stored in at least one of a memory area of ​​the processor and the storage unit 15.

[0066] The processing unit 13 may have a main memory, which includes at least one of a volatile memory such as a random access memory (RAM) and a non-volatile memory such as a read only memory (ROM).

[0067] The RAM may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM), and is used as a virtual memory space allocated to the processing unit 13 as a working space. The operating system, application programs, program modules, program data, lookup tables, and the like stored in the storage unit 15 are loaded into the RAM for execution. The data, programs, and program modules loaded into the RAM are each directly accessed and operated by the processing unit 13.

[0068] ROM can store BIOS (Basic Input / Output System) and firmware, which do not require rewriting. Examples of ROM include mask ROM, OTPROM (One-Time Programmable Read Only Memory), and EPROM (Erasable Programmable Read Only Memory). Examples of EPROM include UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory), which allows stored data to be erased by exposure to ultraviolet light, EEPROM (Electrically Erasable Programmable Read Only Memory), and flash memory.

[0069] [Storage section 15] The storage unit 15 has a function of storing a program executed by the processing unit 13. The storage unit 15 may also have a function of storing the calculation results and inference results generated by the processing unit 13, as well as image data input to the input unit 11.

[0070] The storage unit 15 includes at least one of a volatile memory and a nonvolatile memory. The storage unit 15 may include a volatile memory such as a DRAM or an SRAM. The storage unit 15 may include a nonvolatile memory such as a ReRAM (Resistive Random Access Memory, also called a resistance change memory), a PRAM (Phase change Random Access Memory), an FeRAM (Ferroelectric Random Access Memory), an MRAM (Magnetoresistive Random Access Memory, also called a magnetoresistive memory), or a flash memory. The storage unit 15 may also include a recording media drive such as a hard disk drive (HDD) or a solid state drive (SSD).

[0071] [Database 17] The database 17 has a function of storing image data to be searched. As mentioned above, image data stored in the database is called database image data. The database 17 also has a function of storing calculation results and inference results generated by the processing unit 13. It may also have a function of storing image data, etc. input to the input unit 11. The storage unit 15 and the database 17 do not need to be separated from each other. For example, the image search system 10 may have a storage unit that has the functions of both the storage unit 15 and the database 17.

[0072] [Output section 19] The output unit 19 has a function of supplying information to the outside of the image search system 10. For example, the output unit 19 can supply the calculation results or inference results of the processing unit 13 to the outside.

[0073] <1-2. Image search method-1> First, we will explain the process that is performed in advance to perform a search using the image search system 10. Figure 2 is a flowchart showing an example of the method of this process.

[0074] [Step S01] First, the database image data GD DB is input to the processing unit 13. DB can be data representing drawings contained in intellectual property information. Here, intellectual property information includes, for example, patent documents (unexamined patent gazettes, patent gazettes, etc.), utility model gazettes, design gazettes, and publications such as papers. Publications published in countries around the world, not just those published in Japan, can be used as intellectual property information.

[0075] Intellectual property information is not limited to publications. For example, various files such as image files owned independently by users or user organizations of image search systems can also be included in the database image data GD. DB Furthermore, intellectual property information may include drawings that explain inventions, devices, or designs.

[0076] Also, the database image data GD DB may, for example, contain data representing drawings in patent documents of a particular applicant, or in patent documents in a particular technical field.

[0077] The image retrieval system 10 retrieves database image data GD similar to the query image data. DB Therefore, by using the image search system 10, it is possible to search for patent documents, papers, or industrial products similar to an invention before application. This allows for prior art research related to an invention before application. By understanding and reviewing related prior art, an invention can be strengthened and made into a strong patent that is difficult for other companies to circumvent.

[0078] Furthermore, by using the image search system 10, it is possible to search for patent documents, papers, or industrial products similar to an industrial product that has not yet been released. DBIf the company has data corresponding to the images described in its patent documents, it can confirm whether the technology related to the industrial product before its release has been sufficiently patented within the company. DB If a company has data corresponding to images in patent documents of other companies, it can check whether an industrial product before its release infringes on the intellectual property rights of other companies. By identifying and reviewing related prior art, it is possible to discover new inventions that can be used as strong patents to contribute to the company's business. Note that searches can be conducted not only for industrial products before their release, but also for industrial products after their release.

[0079] Furthermore, for example, it is possible to search for patent documents, papers, or industrial products similar to a specific patent using the image search system 10. In particular, by searching based on the filing date of the patent, it is possible to easily and accurately check whether the patent contains grounds for invalidation.

[0080] [Step S02] Next, the database image data GD DB is input to the neural network of the processing unit 13.

[0081] 3A is a diagram showing an example of the configuration of neural network 30, which is a neural network included in processing unit 13. Neural network 30 has layers 31[1] to 31[m] (m is an integer of 1 or more).

[0082] Layers 31[1] to 31[m] have neurons, and the neurons in each layer are connected to each other. For example, the neurons in layer 31[1] are connected to the neurons in layer 31[2]. The neurons in layer 31[2] are connected to the neurons in layer 31[1] and layer 31[3]. In other words, layers 31[1] to 31[m] form a hierarchical neural network.

[0083] Database image data GDDB is input to layer 31[1], which outputs data corresponding to the input image data. This data is input to layer 31[2], which outputs data corresponding to the input data. Data output from layer 31[m-1] is input to layer 31[m], which outputs data corresponding to the input data. From the above, layer 31[1] can be considered the input layer, layers 31[2] to 31[m-1] can be considered intermediate layers, and layer 31[m] can be considered the output layer.

[0084] The neural network 30 is trained in advance so that data output from, for example, layers 31[1] to 31[m] represent the feature quantities of image data input to the neural network 30. The training can be performed by unsupervised learning, supervised learning, or the like. Unsupervised learning is particularly preferable because it does not require teacher data (also called ground truth labels). Furthermore, whether the training is performed by unsupervised learning or supervised learning, a learning algorithm such as backpropagation can be used.

[0085] Here, the database image data GD DB It is preferable to use the above. This allows the data output from, for example, layers 31[1] to 31[m] to accurately represent the feature quantities of the image data input to the neural network 30. For example, the database image data GD stored in the database 17 can be DB The neural network 30 can perform training using all of the above as training data. For example, the database image data GD DB For example, the neural network 30 can perform training using a part of the database image data GD DB In addition to the above, the neural network 30 can perform learning using image data stored in the memory unit 15 and image data input to the processing unit 13 from outside the image retrieval system 10 via the input unit 11 as learning data.

[0086] As training data, the database image data GD DB For example, the neural network 30 can perform learning using only image data input to the processing unit 13 via the input unit 11 from outside the image search system 10 as learning data.

[0087] The neural network 30 can be a convolutional neural network (CNN). Fig. 3B is a diagram showing an example of the configuration of the neural network 30 when a CNN is applied as the neural network 30. Here, the neural network 30 to which a CNN is applied is referred to as neural network 30a.

[0088] The neural network 30a has a convolutional layer CL, a pooling layer PL, and a fully connected layer FCL. Fig. 3B shows an example in which the neural network 30a has m convolutional layers CL and m pooling layers PL (m is an integer equal to or greater than 1), and one fully connected layer FCL. The neural network 30a may have two or more fully connected layers FCL.

[0089] The convolutional layer CL has a function of performing convolution on data input to the convolutional layer CL. For example, the convolutional layer CL[1] has a function of performing convolution on image data input to the processing unit 13. The convolutional layer CL[2] has a function of performing convolution on data output from the pooling layer PL[1]. The convolutional layer CL[m] has a function of performing convolution on data output from the pooling layer PL[m-1].

[0090] Convolution is performed by repeatedly performing product-sum operations on the data input to the convolution layer CL and the weight filter. Through the convolution in the convolution layer CL, image features corresponding to the image data input to the neural network 30a are extracted.

[0091] The convolved data is transformed by an activation function and then output to the pooling layer PL. As the activation function, ReLU (Rectified Linear Units) or the like can be used. ReLU is a function that outputs "0" when the input value is negative and outputs the input value as is when the input value is "0" or greater. In addition, as the activation function, a sigmoid function, a tanh function, or the like can also be used.

[0092] The pooling layer PL has the function of pooling the data input from the convolutional layer CL. Pooling is a process of dividing the data into multiple regions, extracting predetermined data for each region, and arranging it in a matrix. Pooling can reduce the amount of data while retaining the features extracted by the convolutional layer CL. It can also increase robustness against small deviations in the input data. Note that maximum pooling, average pooling, Lp pooling, etc. can be used as pooling.

[0093] The fully connected layer FCL has the function of judging an image using the data output from the pooling layer PL[m]. The fully connected layer FCL has a configuration in which all nodes in a layer are connected to all nodes in the next layer. The data output from the convolution layer CL or the pooling layer PL is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. Then, the vector obtained by inference by the fully connected layer FCL is output from the fully connected layer FCL.

[0094] The configuration of the neural network 30a is not limited to that shown in FIG. 3B. For example, a pooling layer PL may be provided for each of a plurality of convolution layers CL. In other words, the number of pooling layers PL included in the neural network 30a may be less than the number of convolution layers CL. Furthermore, if it is desired to preserve as much position information of extracted features as possible, it is not necessary to provide a pooling layer PL.

[0095] The neural network 30a can optimize the filter values ​​of the weight filters, the weight coefficients of the fully connected layer FCL, and the like by performing learning.

[0096] Next, an example of the convolution processing performed in the convolution layer CL and the pooling processing performed in the pooling layer PL will be described with reference to Fig. 4. In Fig. 4, it is assumed that the data input to the convolution layer CL has input data values ​​arranged in three rows and three columns (input data value i11, input data value i12, input data value i13, input data value i21, input data value i22, input data value i23, input data value i31, input data value i32, and input data value i33). It is also assumed that the weighting filter has filter values ​​arranged in two rows and two columns (filter value f11, filter value f12, filter value f21, and filter value f22).

[0097] Here, for example, the data input to the convolution layer CL[1] may be image data, and in this case, the input data value may be a pixel value included in the image data.

[0098] In this specification, a pixel value refers to a value that represents the luminance gradation of light emitted by a pixel. For example, if a pixel value is an 8-bit value, the pixel can emit light with 256 luminance gradations. Image data can be said to include a set of pixel values, and can include, for example, the same number of pixel values ​​as there are pixels. For example, if the number of pixels in an image is 2 x 2, image data representing the image can be said to include 2 x 2 pixel values.

[0099] Furthermore, for example, the input data value input to the convolutional layer CL[2] can be the output value of the pooling layer PC[1], and the input data value input to the convolutional layer CL[m] can be the output value of the pooling layer PC[m-1].

[0100] Convolution is performed by multiplying and adding input data values ​​and filter values. The filter values ​​can be data indicating predetermined features (called feature data). In this case, by comparing the input data values ​​with the filter values, feature extraction can be performed on the image data input to the neural network 30a.

[0101] 4 shows how the convolutional layer CL performs a filter process on input data value i11, input data value i12, input data value i21, and input data value i22 to obtain a convolutional value C11 contained in the data output from the convolutional layer CL. It also shows how the convolutional layer CL performs a filter process on input data value i12, input data value i13, input data value i22, and input data value i23 to obtain a convolutional value C12 contained in the data output from the convolutional layer CL. It also shows how the convolutional layer CL performs a filter process on input data value i21, input data value i22, input data value i31, and input data value i32 to obtain a convolutional value C21 contained in the data output from the convolutional layer CL. Furthermore, the convolution layer CL performs filtering on the input data value i22, the input data value i23, the input data value i32, and the input data value i33 to obtain the convolution value C22 contained in the data output from the convolution layer CL. From the above, it can be said that the stride of the convolution processing shown in Figure 4 is 1.

[0102] The convolution value C11, the convolution value C12, the convolution value C21, and the convolution value C22 can be obtained by the product-sum calculation shown in the following equations.

[0103] (Number 1) C11=i11·f11+i12·f12+i21·f21+i22·f22 (1)

[0104] (Number 2) C12=i12·f11+i13·f12+i22·f21+i23·f22 (2)

[0105] (Number 3) C21=i21·f11+i22·f12+i31·f21+i32·f22 (3)

[0106] (Number 4) C22=i22·f11+i23·f12+i32·f21+i33·f22 (4)

[0107] The convolution value C11, the convolution value C12, the convolution value C21, and the convolution value C22 acquired by the convolution layer CL are arranged in a matrix according to the addresses and then output to the pooling layer PL. Specifically, the convolution value C11 is arranged in the first row and the first column, the convolution value C12 is arranged in the first row and the second column, the convolution value C21 is arranged in the second row and the first column, and the convolution value C22 is arranged in the second row and the second column.

[0108] FIG. 4 shows how convolution values ​​C11, C12, C21, and C22 are input to the pooling layer PL, and one value is determined as a pooling value P based on the four convolution values. For example, the maximum value among the convolution values ​​C11, C12, C21, and C22 can be determined as the pooling value P. Alternatively, the average value of the convolution values ​​C11, C12, C21, and C22 can be determined as the pooling value P. The pooling value P is an output value output from the pooling layer PL.

[0109] While FIG. 4 shows an example in which data input to the convolutional layer CL is processed by one weight filter, it may also be processed by two or more weight filters. In this case, it is possible to extract multiple features contained in the image data input to the neural network 30a. When data input to the convolutional layer CL is processed by two or more weight filters, the processing shown in FIG. 4 is performed for each filter. Also, as described above, the stride is set to 1 in FIG. 4, but the stride may be set to 2 or more.

[0110] 5 is a diagram showing an example of the configuration of the convolution layer CL and the pooling layer PL included in the neural network 30a. In FIG. 5, an example is shown in which the convolution layer CL and the pooling layer PL perform the operations shown in FIG.

[0111] FIG. 5 shows neurons 32. Specifically, neurons 32a, 32b, and 32c are shown as neurons 32. In FIG. 5, a value output from a neuron 32 is written inside the neuron 32. The value is output in the direction of the arrow. Furthermore, when the value is multiplied by a weighting factor, the weighting factor is written near the arrow. In FIG. 5, filter values ​​f11, f12, f21, and f22 are used as weighting factors.

[0112] Neuron 32a is a neuron 32 included in layer L, which is a layer preceding convolutional layer CL shown in Fig. 5. Layer L can be, for example, an input layer when convolutional layer CL shown in Fig. 5 is convolutional layer CL[1], a pooling layer PL[1] when convolutional layer CL[2], or a pooling layer PL[m-1] when convolutional layer CL[m].

[0113] 5 shows neurons 32a[1] to 32a[9] as neurons 32a. In the example shown in FIG. 5, neuron 32a[1] outputs an input data value i11, neuron 32a[2] outputs an input data value i12, neuron 32a[3] outputs an input data value i13, neuron 32a[4] outputs an input data value i21, neuron 32a[5] outputs an input data value i22, neuron 32a[6] outputs an input data value i23, neuron 32a[7] outputs an input data value i31, neuron 32a[8] outputs an input data value i32, and neuron 32a[9] outputs an input data value i33.

[0114] The neuron 32b is a neuron 32 included in the convolutional layer CL shown in Fig. 5. In Fig. 5, the neurons 32b are shown as neurons 32b[1] to 32b[4].

[0115] 5, neuron 32b[1] receives the following inputs: input data value i11 multiplied by filter value f11; input data value i12 multiplied by filter value f12; input data value i21 multiplied by filter value f21; and input data value i22 multiplied by filter value f22. Neuron 32b[1] then outputs a convolution value C11, which is the sum of these values.

[0116] Neuron 32b[2] receives the following inputs: input data value i12 multiplied by filter value f11; input data value i13 multiplied by filter value f12; input data value i22 multiplied by filter value f21; and input data value i23 multiplied by filter value f22. Neuron 32b[2] outputs a convolution value C12, which is the sum of these values.

[0117] Neuron 32b[3] receives the following inputs: input data value i21 multiplied by filter value f11; input data value i22 multiplied by filter value f12; input data value i31 multiplied by filter value f21; and input data value i32 multiplied by filter value f22. Neuron 32b[3] outputs a convolution value C21, which is the sum of these values.

[0118] Furthermore, neuron 32b[4] receives the following inputs: input data value i22 multiplied by filter value f11; input data value i23 multiplied by filter value f12; input data value i32 multiplied by filter value f21; and input data value i33 multiplied by filter value f22. Neuron 32b[4] outputs the sum of these values, a convolution value C22.

[0119] 5, each of neurons 32b[1] to 32b[4] is connected to a part of neurons 32a[1] to 32a[9]. Therefore, the convolutional layer CL can be said to be a partially connected layer.

[0120] The neuron 32c is the neuron 32 included in the pooling layer PL shown in FIG. 5. In the case shown in FIG. 5, the neuron 32c receives the convolution values ​​C11, C12, C21, and C22. The neuron 32c outputs the pooling value P. The convolution value output from the neuron 32b is not multiplied by a weighting factor. As described above, the weighting factor is a parameter that is optimized by learning of a neural network. Therefore, the parameters used by the pooling layer PL during calculations can be configured so that no parameter that is optimized by learning exists.

[0121] From the above, the database image data GD DB is input to the neural network 30, the database image data GD DB Database image feature data GFD representing the feature values DB For example, as shown in FIG. 3A, the data output from the layer 31[m] can be converted into database image feature data GFD DB Alternatively, as shown in FIG. 3B, the data output from the pooling layer PL[m] can be converted into database image feature data GFD DB The database image feature data GFD DB may contain two or more layers of output data. DB By including the output data of many layers, the database image feature data GFD DB , the database image data GD DB The database image feature quantity data GFD acquired by the processing unit 13 can more accurately represent the features of the image. DB can be stored in the database 17.

[0122] [Step S03] Next, the database image data GD DB Database tag associated with TAG DB The processing unit 13 acquires the database image data GD DB The database tag TAG represents the concept, technical content, and points of interest of the image corresponding to the image. DB So, the database tag TAG DB It is preferable to obtain the database tag TAG DB 6A is a diagram showing an example of a method for acquiring the data. Note that the illustration of each piece of data shown in FIG. 6A is an example, and is not limited to this. Furthermore, the illustration of each piece of data, vectors, etc. shown in other figures is also an example, and is not limited to the contents shown.

[0123] In this specification, when the same reference numeral is used for multiple elements, and when it is particularly necessary to distinguish between them, an identifying symbol such as [1] or [2] will be added to the reference numeral.

[0124] In the method shown in FIG. 6A, as an example, the database image data GD DB [1] to database image data GD DB

[0100] is associated with a tag. DB Document data TD corresponding to DB are stored in advance in the database 17. Furthermore, the database image data GD DB Each figure is associated with a figure number.

[0125] Document Data TD DB For example, the database image data GD DB The data may be data corresponding to documents described in publications such as patent documents, utility model publications, design publications, and papers in which drawings represented by the database image data GD DB If the publication in which the drawing represented by is published is a patent document or a utility model publication, the data corresponding to the specification is stored in document data TD.DB Alternatively, data corresponding to the claims of a patent, the claims of a utility model registration, or the abstract can be stored as document data TD. DB In addition, the database image data GD DB If the publication in which the application is published is a design gazette, the data corresponding to the application will be stored in document data TD. DB It can be said that:

[0126] For example, document data TD DB When the data corresponds to a specification or paper, the database tag TAG DB is the database image data GD DB In FIG. 6A, the database image data GD DB The figure number of the image corresponding to [1] is "Figure 1", and the database image data GD DB Document data TD linked to [1] DB This shows an example where "Figure 1 is" ​​is written in paragraph [0xx0] of the document represented by [1]. Therefore, for example, in paragraph [0xx0], the database image data GD DB By performing morphological analysis on the sentence in paragraph [0xx0], the database tag TAG DB [1] can be obtained. Also, in FIG. 6A, the image data GD DB The figure number of the image corresponding to

[0100] is "Figure 15", and the database image data GD DB Document data TD linked to

[0100] DB This shows an example in which "Figure 15 is" is written in paragraph [0xx7] of the document represented by

[0100] . Therefore, for example, in paragraph [0xx7], the database image data GD DB By assuming that the description of the drawing represented by

[0100] is described and performing morphological analysis on the sentence described in paragraph [0xx7], the database tag TAG DB

[0100] can be obtained.

[0127] In addition, all document data TD DB For example, the database image data GD DB [1] corresponds to "Figure 1" in the specified publication, and the database image data GD DB If [2] corresponds to "Figure 2" in the same publication, then Document Data TD DB [1] represents the document and document data TD DB The documents represented by [2] can be the same.

[0128] Morphological analysis divides a sentence written in a natural language into morphemes (the smallest unit of meaning in language) and determines the part of speech of each morpheme. For example, this allows you to extract only the nouns in paragraph [0xx0] and create a database tag TAG DB In the case shown in FIG. 6A, words such as "circuit diagram," "aaa," "bbb," "ccc," and "ddd" are included in the database tag TAG DB [1]. In addition, words such as "block diagram", "ggg", "aaa", "ccc", and "hhh" are included in the database tag TAG DB It is assumed to be

[0100] .

[0129] As mentioned above, the database tag TAG DB For example, the database image data GD DB Document data TD linked to DB This method can be used to obtain the database tag TAG DB By obtaining the database image data GD DB It is possible to comprehensively obtain tags that represent the concept, technical content, points of interest, etc. of the corresponding image.

[0130] In this specification, one tag means, for example, one word. For example, in the case shown in FIG. 6A, the database tag TAG DB The number of [1] can be 5 or more. Also, the database tag TAG DB The number of

[0100] can be five or more.

[0131] For example, all words extracted by morphological analysis are stored in the database tag TAG DB For example, a predetermined number of words may be extracted from the extracted words, and the extracted words may be used as a database tag TAG DB For example, a predetermined number of words with high TF-IDF (Term Frequency-Inverse Document Frequency) can be extracted from the extracted words, and the extracted words can be used as a database tag TAG DB The database image data GD DB Database tag associated with TAG DB The number of database image data GD DB By making each of the above equal, the image search method using the image search system 10 can be simplified.

[0132] TF-IDF is calculated based on two metrics: word frequency (TF) and inverse document frequency (IDF). Therefore, a word that appears frequently throughout a document will have a high TF but a low IDF. Therefore, the database tag TAG DB A word that appears frequently in the paragraph from which it was extracted will have a lower TF-IDF than a word that appears less frequently in other paragraphs. For example, a word that appears frequently throughout a document may not be a word that strongly represents the characteristics of an image, such as a concept, technical content, or a point of interest. Therefore, using TF-IDF, the database tag TAG DB Once you have obtained the database tag TAG using only TF, for example DB The database tag TAG strongly represents the characteristics of the database image. DB Therefore, the image search system 10 can perform a search with high accuracy. It should be noted that the database tag TAG can be obtained without calculating the TF-IDF and using only the TF. DB In this case, the calculation performed by the processing unit 13 can be simplified.

[0133] Note that morphological analysis may be performed on sentences of two or more paragraphs. For example, DB In addition to the paragraph that is considered to contain the description of the drawing represented by the paragraph, the morphological analysis may also be performed on the paragraphs before and after the paragraph. For example, in the document data TD shown in FIG. 6A, DB When performing morphological analysis on [1], in addition to paragraph [0xx0], morphological analysis may also be performed on the next paragraph, paragraph [0xx1]. In this case, for example, the word "eee" in paragraph [0xx1] is added to the database tag TAG DB [1]. For example, the document data TD shown in FIG. 6A can be DB When performing morphological analysis on

[0100] , morphological analysis may be performed on the previous paragraph [0xx6] in addition to the paragraph [0xx7]. In this case, for example, the word "fff" written in the paragraph [0xx6] is added to the database tag TAG DB It can be set to

[0100] .

[0134] Or database image data GD DB Alternatively, morphological analysis may be performed on all paragraphs containing figure numbers linked to the database image data GD DB Morphological analysis may be performed on a paragraph that contains a figure number associated with the paragraph and does not contain any other figure numbers.

[0135] Alternatively, morphological analysis may be performed only on some sentences included in a given paragraph. For example, in the case shown in FIG. 6A, morphological analysis may be performed only on the sentences in paragraph [0xx0] that include "Figure 1." In this case, the word "ddd" is included in the database tag TAG DB [1] is not the case.

[0136] In addition, document data TD DB Not only the words themselves that appear in the documents represented by the database tag TAG DBFor example, thesaurus data may be stored in advance in the storage unit 15 or the database 17, and a word extracted by morphological analysis and a word registered in thesaurus as a synonym of the extracted word may be stored in a database tag TAG. DB The synonyms used here may be those extracted using a commonly available thesaurus, or may be those extracted using embedded representations of words. Furthermore, synonyms extracted using embedded representations may be extracted using a database containing other documents in the same field as the document to be searched.

[0137] Document Data TD DB Not only the words themselves that appear in the documents represented by the tag, but also synonyms of those words are included in the database tag TAG DB By doing so, the database tag TAG DB , the database image data GD DB It can be something that strongly expresses the characteristics such as the concept, technical content, and points of interest.

[0138] In addition, the database tag TAG DB For example, the database image feature data GFD DB Based on the database tag TAG DB may be obtained.

[0139] FIG. 6B shows the database image data GD DB 6B shows an example of a method for linking a figure number to a publication data PD. DB [1] and image data GD DB [2], and Document Data TD DB Assume that the publication represented by the publication data PD contains the text "Figure 1 xxx" and the text "Figure 2 yyy". The data representing the text "Figure 1 xxx" and the data representing the text "Figure 2 yyy" are stored in the document data TD. DBIt is assumed not to be included. Also, the "x1", "x2", "x1 < x2", and the dashed lines, arrows, etc. shown in FIG. 6B are added for convenience of explanation, and it is assumed that they are not actually described in the publication represented by the publication data PD.

[0140] In the method shown in FIG. 6B, for example, if the text "FIG. N" is described within a predetermined distance from the drawing, the figure number of the drawing provided at the closest distance to the text "FIG. N" can be set as "N". Here, for example, the distance between the coordinates representing the center of the text (center coordinates) and the center coordinates of the drawing can be regarded as the distance from the text to the drawing. Note that "N" is not limited to an integer and may include letters, for example. For example, N may be "1(A)".

[0141] In the case shown in FIG. 6B, the distance x1 between the center coordinates of the text "FIG. 1 xxx" and the center coordinates of the drawing corresponding to the database image data GD DB [1] is shorter than the distance x2 between the center coordinates of the text "FIG. 1 xxx" and the center coordinates of the drawing corresponding to the database image data GD DB [2]. Therefore, the drawing provided at the closest distance to the text "FIG. 1 xxx" can be the database image data GD DB [1]. Therefore, the figure number associated with the database image data GD DB [1] can be set as "1".

[0142] Also, in FIG. 6B, an example is shown where in the paragraph [0zz3] of the document represented by the document data TD DB it is described as "FIG. 1 is," and in the paragraph [0zz4] it is described as "FIG. 2 is." From the above, in the case shown in FIG. 6B, the database tag TAG DB [1] associated with the database image data GD DB [1] can be obtained, for example, by performing morphological analysis on the text described in the paragraph [0zz3]. In FIG. 6B, words such as "block diagram", "iii", "kkk", "hhh", "ppp", etc. described in the paragraph [0zz3] are the database tag TAGDB [1]

[0143] For example, the coordinates of the centers of all drawings may be arranged as a first one-dimensional array, and the coordinates of the centers of all text "Figure N" may be arranged as a second one-dimensional array. Then, the coordinates contained in the first one-dimensional array may be compared with the coordinates contained in the second one-dimensional array, and the text "Figure N" written at the closest coordinates may be linked to each drawing. In other words, the figure number of the drawing located at the coordinates closest to the coordinates representing the position of the text "Figure N" may be designated as "N." The comparison of the coordinates contained in the first one-dimensional array with the coordinates contained in the second one-dimensional array may be performed, for example, by calculating the sum of the square of the difference between the x coordinates and the square of the difference between the y coordinates. When performing the comparison using this method, the element with the smallest sum may be designated as the element located at the closest coordinates.

[0144] By the above method, the processing unit 13 obtains the database tag TAG DB The database tag TAG acquired by the processing unit 13 can be acquired. DB can be stored in the database 17.

[0145] [Step S04] Next, the database tag TAG DB is represented by a vector. DB The vector representing the database tag vector TAGV DB FIG. 7A shows the database tag TAG DB is expressed by a vector.

[0146] Database tag vector TAGV DB is the database tag TAG DB can be obtained by inputting the database tag vector TAGV DBcan be, for example, a distributed representation vector. A distributed representation vector is a vector that represents a word with a quantified continuous value for each feature element (dimension). Words with similar meanings will have similar vectors.

[0147] The neural network used to acquire the distributed representation vectors can have a different configuration from the neural network used to acquire the image feature vectors. Fig. 7B shows an example of the configuration of neural network 40, which is a neural network used to acquire the distributed representation vectors.

[0148] In this specification, for example, a neural network used to acquire image feature data may be referred to as a first neural network, and a neural network used to acquire a distributed representation vector may be referred to as a second neural network. Note that the ordinal numbers are merely an example, and for example, a neural network used to acquire a distributed representation vector may be referred to as a first neural network, and a neural network used to acquire image feature data may be referred to as a second neural network. Furthermore, for example, a neural network used to acquire image feature data may be referred to as a third neural network, or a neural network used to acquire a distributed representation vector may be referred to as a third neural network, or the like.

[0149] As shown in FIG. 7B, the neural network 40 has an input layer IL, a hidden layer ML, and an output layer OL. Here, the neural network 40 can be configured to have one hidden layer ML. The neural network 40 can obtain a distributed representation vector representing a word input to the input layer IL using, for example, Word2Vec, an open-source algorithm. In the following, it will be described how the neural network 40 configured as shown in FIG. 7B obtains a distributed representation vector representing a word input to the input layer IL. DB A database tag vector TAGV representing DBAn example of a method for obtaining the above information will be described below.

[0150] The input layer IL contains the database tag TAG DB is input as a one-hot vector. Here, one component of a one-hot vector represents one word, and the component corresponding to the word input to the input layer IL can be set to 1, and the other components can be set to 0. In other words, a one-hot vector can be said to be a vector in which one component is 1 and all other components are 0. The number of neurons in the input layer IL can be set to the same number as the number of components that make up the one-hot vector.

[0151] The hidden layer ML has a function of generating a distributed representation vector based on the one-hot vector input to the input layer IL. For example, the hidden layer ML can generate a distributed representation vector by multiplying the one-hot vector by a predetermined weight. Since the weight can be expressed by a matrix, the neural network 40 can generate a distributed representation vector by performing a product-sum operation between the one-hot vector and the weight matrix.

[0152] The number of neurons in the hidden layer ML can be the same as the number of dimensions of the distributed representation vector. For example, if the number of dimensions of the distributed representation vector is 300, the hidden layer ML can be configured to have 300 neurons.

[0153] The weight matrix can be obtained by learning, for example, supervised learning. Specifically, a word represented by a one-hot vector is input to the input layer IL, and neighboring words of the word input to the input layer IL are input to the output layer OL, represented by a one-hot vector. Here, for each word input to the input layer IL, multiple neighboring words are input to the output layer OL. Then, the values ​​of the weight matrix of the neural network 40 are adjusted so that the output layer OL can output the probability that the word will be a neighboring word of the word input to the input layer IL. Note that, for example, one neuron in the output layer OL corresponds to one word. The above is an example of a learning method for the neural network 40.

[0154] As described above, in both the input layer IL and the output layer OL, one neuron can correspond to one word. Therefore, the number of neurons in the input layer IL can be the same as the number of neurons in the output layer OL.

[0155] Furthermore, the number of neurons in the hidden layer ML can be smaller than the number of neurons in the input layer IL. For example, the number of words that can be processed by the neural network 40, i.e., the number of neurons in the input layer IL, can be set to 10,000, and the number of dimensions of the distributed representation vector, i.e., the number of neurons in the hidden layer ML, can be set to 300. Therefore, with distributed representation, the number of dimensions can be kept small even if the number of representable words increases, so the amount of calculation is unlikely to increase even if the number of representable words increases. Therefore, the image retrieval system 10 can perform searches in a short time.

[0156] By the above method, the processing unit 13 obtains the database tag vector TAGV DB The database tag vector TAGV obtained by the processing unit 13 can be obtained. DB can be stored in the database 17.

[0157] As described above, in steps S01 to S04, the processing unit 13 generates the database image feature amount data GFD DB, database tag TAG DB , and the database tag vector TAGV DB is acquired and stored in the database 17. This enables the image retrieval system 10 to search for database images similar to the query image. DB does not need to be stored in the database 17.

[0158] In the method shown in FIG. 2, the processing unit 13 calculates the database image feature data GFD in steps S01 and S02. DB After acquiring the database tag TAG DB , and the database tag vector TAGV DB However, the present invention is not limited to this. For example, the processing unit 13 acquires the database tag TAG DB , and the database tag vector TAGV DB After obtaining the database image feature data GFD DB may be obtained.

[0159] In addition, in the method shown in Figure 2, the database tag TAG DB is input to the neural network 40, the vector itself output from the neural network 40 is used as the database tag vector TAGV DB However, one aspect of the present invention is not limited to this. DB A modified example of the method for obtaining the above will be described.

[0160] First, the processing unit 13 calculates the database tag TAG DB Get candidate words for database tag TAG DB Candidate words can be obtained by morphological analysis, for example, as shown in FIGS. 6A and 6B.

[0161] Next, the acquired words are represented by vectors. For example, the acquired words can be input to the neural network 40 to be represented by distributed representation vectors.

[0162] Then, a predetermined number of clusters are generated by clustering the distributed representation vectors. For example, DB The same number of clusters as the number of objects are generated. Clustering can be performed using the K-means method, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, etc.

[0163] In FIG. 8A, the database image data GD DB As candidates for tags to be linked to [1], 20 words are acquired by the processing unit 13, and these words are respectively converted into database word vectors WORDV DB In addition, in Fig. 8A, 20 database word vectors WORDV DB 8A shows an example of generating five clusters (cluster CST1, cluster CST2, cluster CST3, cluster CST4, and cluster CST5) based on the database word vector WORDV. For convenience of explanation, the vectors shown in FIG. 8A are two-dimensional vectors, with the horizontal axis representing one component of the two-dimensional vector and the vertical axis representing the other component of the two-dimensional vector. DB etc. can be, for example, a 300-dimensional vector.

[0164] Next, a vector representing a representative point is calculated for each of the clusters CST1 to CST5. The vector representing the representative point is then used as a database tag vector TAGV DB In Figure 8A, the vector representing the representative point of cluster CST1 is the database tag vector TAGV1 DB [1], and the vector representing the representative point of cluster CST2 is the database tag vector TAGV2 DB[1], and the vector representing the representative point of cluster CST3 is the database tag vector TAGV3 DB [1], and the vector representing the representative point of cluster CST4 is the database tag vector TAGV4 DB [1], and the vector representing the representative point of cluster CST5 is the database tag vector TAGV5 DB [1] shows an example.

[0165] Each component of the vector representing the representative point is, for example, the database word vector WORDV included in the cluster. DB For example, if a cluster has five database word vectors WORDV (0.1,0.7), (0.2,0.5), (0.3,0.5), (0.4,0.2), and (0.5,0.1), DB In this case, the vector representing the representative point of the cluster can be, for example, (0.3, 0.4). DB [1] can be obtained.

[0166] Database tag vector TAGV DB [2] and subsequent steps can be obtained in the same manner. DB [1] to database image data GD DB For each of

[0100] , the database tag vector TAGV DB Five of each (database tag vector TAGV1 DB , database tag vector TAGV2 DB , database tag vector TAGV3 DB , database tag vector TAGV4 DB , and the database tag vector TAGV5 DB 8B is a table showing the components of each vector when acquiring the vectors. Note that the components shown in FIG. 8B are an example for convenience of explanation.

[0167] As shown in Figure 8B, the database tag vector TAGV DBThe weights can be assigned to the database word vectors WORDV included in one cluster. DB The number of database image data GD DB 8A and 8B, the database image data GD DB In this example, the processing unit 13 acquires 20 words as candidates for tags to be linked to [1]. In addition, the cluster CST1 contains 8 database word vectors WORDV DB Cluster CST2 contains four database word vectors WORDV DB Cluster CST3 contains two database word vectors WORDV DB Cluster CST4 contains three database word vectors WORDV DB Cluster CST5 contains three database word vectors WORDV DB Therefore, as shown in FIG. 8B, for example, the database image data GD DB For [1], the database tag vector TAGV1 included in cluster CST1 DB The weight of [1] is 8 / 20, and the database tag vector TAGV2 included in cluster CST2 DB The weight of [1] is 4 / 20, and the database tag vector TAGV3 included in cluster CST3 DB The weight of [1] is set to 2 / 20, and the database tag vector TAGV4 included in cluster CST4 DB The weight of [1] is 3 / 20, and the database tag vector TAGV5 included in cluster CST5 DB The weight of [1] can be set to 3 / 20.

[0168] By the above method, for example, database image data GD DB The weight of tag vectors that strongly express features such as concepts, technical content, and points of interest can be increased, thereby enabling the image retrieval system 10 to perform retrieval with high accuracy.

[0169] Next, we will explain an image search method using the image search system 10. Fig. 9 is a flowchart showing an example of the processing method.

[0170] [Step S11] First, a user of the image search system 10 inputs query image data GD Q Input the query image data GD Q is supplied from the input unit 11 to the processing unit 13 via the transmission path 12. Alternatively, the query image data GD Q may be stored in the storage unit 15 or the database 17 via the transmission path 12 and supplied to the processing unit 13 from the storage unit 15 or the database 17 via the transmission path 12.

[0171] Query image data GD Q may include, for example, an image illustrating a pre-filing invention, device or design, a pre-release industrial product, technical information, or a technical idea.

[0172] [Step S12] Next, the query image data GD Q is input to the neural network included in the processing unit 13. For example, the query image data GD Q As a result, the processing unit 13 inputs the query image data GD Q Query image feature data GFD representing the feature Q For example, the data output from the layer 31[m] shown in FIG. 3A can be used as the query image feature data GFD Q Alternatively, the data output from the pooling layer PL[m] shown in FIG. 3B can be used as the query image feature data GFD Q The query image feature data GFD Q is the database image feature data GFD DB Similarly, the query image feature data GFD may contain two or more layers of output data. QSince the output data of many layers is included, the query image feature data GFD Q Query image data GD Q It can be made to more accurately represent the characteristics of

[0173] [Step S13] Next, the database image data GD DB Query image data GD Q The processing unit 13 calculates the similarity to

[0174] FIG. 10 shows the database image data GD DB Query image data GD Q 10 is a diagram showing how to calculate the similarity for one query image data GD Q and 100 database image data GD DB 3B is input to the neural network 30a. In addition, in FIG. 10, the query image feature data GFD Q , and database image feature data GFD DB , each of which has a pooling value P of x rows and y columns (x and y are integers equal to or greater than 1).

[0175] In this specification, the query image feature data GFD Q The pooling value P Q and the database image feature data GFD DB The pooling value P DB For example, the database image feature data GFD DB The pooling value of [1] is pooling value P1 DB and the database image feature data GFD DB The pooling value of

[0100] is pooled as P100 DB It is written as follows.

[0176] In the case shown in FIG. 10, the database image feature data GFD DB [1] or database image feature data GFDDB For each of

[0100] , the query image feature data GFD Q Then, the similarity is calculated based on the database image data GD DB [1] to database image data GD DB

[0100] , the query image data GD Q It should be noted that the similarity to all the database image feature data GFD stored in the database 17 can be used. DB Regarding the query image feature data GFD Q Alternatively, the similarity may be calculated based on the database image feature data GFD stored in the database 17. DB For part of the query image feature data GFD Q Alternatively, the similarity to the

[0177] The similarity is preferably, for example, a cosine similarity. Alternatively, it may be a Euclidean similarity or a Minkowski similarity. For example, the database image feature data GFD DB [1] Query image feature data GFD Q The cosine similarity for the database image data GD can be calculated using the following formula: DB is the query image data GD Q It can be said to be similar to

[0178] (Number 5) JPEG0007771299000001.jpg22167

[0179] Database image feature data GFD DB [2] or database image feature data GFD DB Query image feature data GFD of

[0100] Q The cosine similarity for the database image data GD DB [1] to database image data GD DB

[0100] , the query image data GD QThe similarity to the

[0180] By calculating the cosine similarity as the similarity, the image retrieval system 10 can perform a search with high accuracy. Furthermore, the cosine similarity can be calculated by a simple calculation. Therefore, if the processing unit 13 has a GPU, the similarity can be calculated by the GPU. Therefore, the similarity can be calculated in a short time, and the image retrieval system 10 can perform a search in a short time.

[0181] [Step S14] Next, the database image data GD DB Query image data GD Q Based on the calculation result of the similarity to the query image data GD Q Query tag TAG that is associated with Q The processing unit 13 acquires the above.

[0182] 11A and 11B show the query tag TAG Q 11A is a diagram showing an example of a method for acquiring the database image data GD based on the similarity calculated in step S13. DB [1] to database image data GD DB For example, the most query image data GD Q Database image data GD with high similarity to DB In the case shown in FIG. 11A, the database image data GD DB [2] has the highest similarity of 0.999, and the database image data GD DB

[31] has the second highest similarity score of 0.971, and the database image data GD DB

[73] has the third highest similarity score of 0.964, and the database image data GD DB

[52] has the fourth highest similarity score of 0.951, and the database image data GD DB

[28] has the fifth highest similarity score of 0.937.

[0183] Next, the database image data GD DB Database tag associated with TAG DB In the case shown in FIG. 11A, the database image data GD DB Database tag associated with TAG DB Specifically, the database image data GD DB [2] The tags "aaa", "bbb", "ccc", "ddd", and "eee" associated with the database image data GD DB

[31] and the tags “aaa”, “ccc”, “fff”, “ggg”, and “hhh” associated with the database image data GD DB

[73] and the tags "aaa", "bbb", "fff", "iii", and "kkk" associated with the database image data GD DB

[52] and the tags "aaa", "ccc", "ggg", "ppp", and "qqq" associated with the database image data GD DB The tags “aaa,” “kkk,” “rrr,” “sss,” and “ttt” associated with

[28] are extracted. As shown in FIG. 11A, the extracted tags may overlap.

[0184] In the above, the database tag TAG DB Extracting database image data GD DB However, the number of database image data GD whose similarity is equal to or greater than a predetermined value is not limited to this. DB You can also extract the database tag associated with the database tag TAG DB Extracting database image data GD DB The number of the elements does not need to be fixed.

[0185] Then, as shown in FIG. 11B, the number of occurrences of each of the extracted tags is calculated. For example, the tag “aaa” is DB [2], Database Image Data GD DB

[31] , Database Image Data GD DB

[73] , Database Image Data GD DB

[52] , and database image data GD DB

[28] , so the number of occurrences is 5. The tag "ddd" is linked to the database image data GD DB [2], Database Image Data GD DB

[31] , Database Image Data GD DB

[73] , Database Image Data GD DB

[52] , and database image data GD DB In

[28] , the database image data GD DB Since it is linked only to [2], the number of occurrences is 1.

[0186] Next, as shown in FIG. 11B, a predetermined number of tags are further extracted, for example, in descending order of the number of times of appearance, and the extracted tags are referred to as query tags TAG Q In the case shown in FIG. 11B, the query tag TAG Q Specifically, the most frequently occurring tag "aaa" (5 times) and the second most frequently occurring tag "ccc" (3 times) are extracted.

[0187] If there are multiple tags with the same number of occurrences, but it is not possible to extract all of the multiple tags, for example, a database image data GD with a higher similarity DB For example, the database image data GD DB The similarity ranking is expressed numerically. Then, the database image data GD DB The sums of the numbers representing the similarity rankings are compared, and tags with the smallest sums can be extracted in order.

[0188] In the case shown in FIG. 11B, the query tag TAG QThe number of tags is set to 5, the number of occurrences of the tag "aaa" is set to 5, and the number of occurrences of the tag "ccc" is set to 3. Therefore, it is necessary to extract three tags from the tags with an occurrence count of 2 or less. However, there are four tags with an occurrence count of 2: "bbb", "fff", "ggg", and "kkk". Therefore, it is necessary to select three tags from these four tags. Here, the database image data GD linked to the tag "bbb" is DB The similarity ranking of [2] is 1, and the database image data GD DB The similarity rank of

[73] is 3. Therefore, the sum of the similarity ranks for the tag "bbb" is 4. Similarly, the sum of the similarity ranks for the tag "fff" is 5, the sum of the similarity ranks for the tag "ggg" is 6, and the sum of the similarity ranks for the tag "kkk" is 8. From the above, the tag with the smallest sum of the similarity ranks for tags with an appearance count of 2 is the tag "bbb", followed in ascending order by the tags "fff", "ggg", and "kkk". Therefore, the tags "bbb", "fff", and "ggg" are used as the query tag TAG Q It can be said that:

[0189] In summary, in the case shown in FIG. 11B, the tag “aaa” with the number of occurrences of 5, the tag “ccc” with the number of occurrences of 3, and the tags “bbb”, “fff”, and “ggg” with the first to third smallest total similarity rankings among the tags with the number of occurrences of 2 are used as the query tag TAG Q It can be said that:

[0190] In addition, the database tag TAG DB Not only the words themselves, but also the database tag TAG DB Search for synonyms of words in the query tag TAG Q For example, the thesaurus data may be stored in advance in the storage unit 15 or the database 17, and the database tag TAG DB The words included in and the words registered in the thesaurus as synonyms of the words are used as query tags. Q can be included in

[0191] In the case shown in FIG. 11B, the extracted database tag TAG DB The processing unit 13 selects the query tag TAG Q However, one aspect of the present invention is not limited to this. For example, the database tag TAG extracted by the user of the image search system 10 is automatically selected. DB and select the query tag TAG from the presented tags. Q The user of the image retrieval system 10 may select a tag that matches the criteria. Alternatively, for example, database images with high similarity may be presented to the user of the image retrieval system 10, and the user may select one of the presented database images. Then, database image data GD representing the selected database image may be generated. DB Database tag associated with TAG DB All or part of the above can be used as a query tag. Q It may also be possible to use the following.

[0192] In the method shown in FIGS. 11A and 11B, the query tag TAG Q , the database tag TAG DB However, one aspect of the present invention is not limited to this. For example, the database tag TAG DB Generate a new tag based on the query tag TAG Q In the following, the database tag TAG DB A database tag vector TAGV representing DB Using the query tag TAG Q A query tag vector TAGV Q An example of a method for the processing unit 13 to acquire the information will be described below.

[0193] The method illustrated in FIG. 11A uses the query tag vector TAGV Q This can also be applied when acquiring the database tag TAG DB The extraction can be performed in a manner similar to that shown in FIG. 11A.

[0194] Database tag TAG DBAfter extraction, the extracted database tag TAG DB A database tag vector TAGV representing DB For example, a query tag TAG is generated by clustering the query tag TAG. Q The same number of clusters as the number of objects are generated. Clustering can be performed using the K-means method, DBSCAN method, etc.

[0195] In FIG. 12A, the 25 database tags TAG shown in FIG. DB is acquired by the processing unit 13. In addition, in FIG. 12A, the database tag TAG DB The database tag vector TAGV corresponding to DB 12A shows an example in which five clusters (cluster CST1, cluster CST2, cluster CST3, cluster CST4, and cluster CST5) are generated based on the above. For ease of explanation, the vectors shown in FIG. 12A are two-dimensional vectors, with the horizontal axis representing one component of the two-dimensional vector and the vertical axis representing the other component of the two-dimensional vector, but in reality they can be vectors of, for example, 300 dimensions. Also, the numbers in parentheses shown in FIG. 12A represent the extracted database tags TAG DB For example, "aaa(5)" indicates that the tag "aaa" appears 5 times.

[0196] Next, a vector representing a representative point is calculated for each of the clusters CST1 to CST5. Then, the vector representing the representative point is used as a query tag vector TAGV Q In FIG. 12A, the vector representing the representative point of the cluster CST1 is the query tag vector TAGV1 Q The vector representing the representative point of cluster CST2 is the query tag vector TAGV2 DB The vector representing the representative point of cluster CST3 is the query tag vector TAGV3 Q The vector representing the representative point of cluster CST4 is the query tag vector TAGV4 QThe vector representing the representative point of cluster CST5 is the query tag vector TAGV5 Q This shows an example where

[0197] Each component of the vector representing the representative point is, for example, a database tag vector TAGV included in the cluster. DB By the above procedure, the processing unit 13 calculates the query tag vector TAGV Q can be obtained.

[0198] Figure 12B shows the query tag vector TAGV1 Q or query tag vector TAGV5 Q 12B is a table showing the components of the signal sigma 12. Note that the components shown in FIG. 12B are an example for the sake of convenience of explanation.

[0199] As shown in Figure 12B, the query tag vector TAGV Q The weight can be assigned to the database tag vector TAGV DB The number of database tags TAG DB For example, in FIG. 12A and FIG. 12B, the total number of 25 database tags TAG DB In addition, cluster CST1 contains 11 database tag vectors TAGV DB Cluster CST2 contains four database tag vectors TAGV DB Cluster CST3 contains five database tag vectors, TAGV DB Cluster CST4 contains two database tag vectors TAGV DB Cluster CST5 contains three database tag vectors, TAGV DB Therefore, as shown in FIG. 12B, the query tag vector TAGV1 included in the cluster CST1 is Q The weight of the query tag vector TAGV2 in cluster CST2 is set to 11 / 25. QThe weight of the query tag vector TAGV3 in cluster CST3 is set to 4 / 25. Q The weight of the query tag vector TAGV4 in cluster CST4 is set to 5 / 25. Q The weight of the query tag vector TAGV5 in cluster CST5 is set to 2 / 25. Q The weight of can be set to 3 / 25.

[0200] By the above method, for example, query image data GD Q The weight of tag vectors that strongly express features such as concepts, technical content, and points of interest can be increased, thereby enabling the image retrieval system 10 to perform retrieval with high accuracy.

[0201] The query tag TAG shown in steps S13 and S14 Q For example, the database tag TAG DB Query without tagging Q This is a simpler method than the method of acquiring the query tag TAG shown in steps S13 and S14. Therefore, the image search system 10 can perform a search in a short time. Q For example, the user of the image search system 10 acquires all the query tags TAG Q Specify and query tag TAG to the user Q Compared with the case where no candidates are presented, the query image data GD Q It is possible to comprehensively acquire tags that represent the concept, technical content, points of interest, etc. of the image corresponding to the image. Therefore, the image search system 10 can perform searches easily and with high accuracy.

[0202] [Step S15] Next, the processing unit 13 extracts the database image feature data GFD DB and the database tag vector TAGV DB and data D including DB Furthermore, the processing unit 13 acquires the query image feature amount data GFD Q and the query tag vector TAGV Q and data D includingQ Get.

[0203] Figure 13 shows the data D DB , and Data D Q FIG. 1 is a diagram illustrating an example of the configuration of database image feature data GFD. DB , and query image feature data GFD Q The database tag vector TAGV can be configured similarly to the configuration shown in FIG. DB is the component VC DB [1] to component VC DB [h] (h is an integer equal to or greater than 2). Q is the component VC Q [1] to component VC Q Here, for example, if five tags expressed by a 300-dimensional vector are linked to one image data, h is 1500.

[0204] In this specification, for example, a database tag vector TAGV DB The component of [1] is component VC1 DB and the database tag vector TAGV DB The component contained in

[0100] is component VC100 DB It is written as follows.

[0205] As mentioned above, the term "component" can sometimes be replaced with the term "value." In this case, both the image feature data and the tag vector can be considered to be a set of multiple values. Therefore, the terms "data" and "vector" can sometimes be used interchangeably.

[0206] [Step S16] Next, Data D DB Data D Q In the case shown in FIG. 13, the processing unit 13 calculates the similarity between the data D DB [1] Data D DB For each of

[0100] , data D QThen, the similarity is calculated based on the database image data GD DB [1] to database image data GD DB

[0100] , the query image data GD Q Therefore, the similarity to the database image data GD DB Query image data GD Q The similarity to the

[0207] Here, when weights are assigned to tag vectors as shown in FIG. 8B and FIG. 12B, weighting can be performed by multiplying the components of the tag vector by the weights, for example.

[0208] Data D DB Data D Q It is preferable that the similarity for data D be the same type as the similarity calculated by the processing unit 13 in step S13. For example, if the cosine similarity is calculated in step S13, DB Data D Q It is preferable to calculate the cosine similarity as the similarity to

[0209] For example, data D DB [1] Data D Q The cosine similarity for can be calculated using the following formula:

[0210] (Number 6) JPEG0007771299000002.jpg23167

[0211] Data D DB [2] Data D DB

[0100] Data D Q The cosine similarity for data D can be calculated in a similar way. DB [1] Data D DB

[0100] Data D QAs a result, the similarity to the database image data GD DB [1] to database image data GD DB

[0100] , the query image data GD Q The similarity to the

[0212] The search results can be changed by adjusting the ratio between the number of values ​​in the image feature data and the number of components in the tag vector. For example, if the query image feature data GFD Q and the database image feature data GFD DB or increasing the number of values ​​that the query tag vector TAGV has. Q The number of components that DB If the number of components in is reduced, the similarity after correction will be a result that emphasizes image features. For example, for database image data GD DB The feature quantity of the query image data GD Q If the feature is similar to the database tag TAG DB is the query tag TAG Q Even if it is slightly different from the database image data GD DB Query image data GD Q On the other hand, the similarity after correction for the query image feature data GFD Q The number of values ​​that the database image feature data GFD has DB or the query tag vector TAGV Q The number of components that DB If the number of components in is increased, the similarity after correction will be a result that emphasizes tags. For example, if the database tag TAG DB is the query tag TAG Q If it is similar to the database image data GD DB The feature quantity of the query image data GD Q Even if the feature amount is slightly different from that of the database image data GD DBQuery image data GD Q The similarity after correction for

[0213] To increase or decrease the number of components in a tag vector, the number of tags associated with image data can be increased or decreased. Furthermore, by using only some of the values ​​contained in the image feature data to calculate the similarity, it is possible to calculate a similarity that emphasizes tags. For example, by not using values ​​representing the feature amounts of parts of an image that do not make a strong impression when viewed, it is possible to calculate a similarity that emphasizes tags while preventing the similarity between a query image and a database image that has a significantly different visual impression from becoming too high. This allows the image search system 10 to perform searches with high accuracy.

[0214] In addition, the search results can be changed by multiplying the value of the image feature data or the component of the tag vector by a predetermined coefficient. For example, Q and the database image feature data GFD DB By multiplying the value of by a real number greater than 1, the corrected similarity can be a result that emphasizes image features. Q and the database tag vector TAGV DB By multiplying the components of the query image feature data GFD by a real number between 0 and 1, the corrected similarity can be a result that emphasizes the image feature. Q and the database image feature data GFD DB By multiplying the value of by a real number between 0 and 1, the similarity after correction can be a result that emphasizes tags. Q and the database tag vector TAGV DB By multiplying the component of by a real number greater than 1, the similarity after correction can be a result that places importance on the tag.

[0215] [Step S17] Next, the processing unit 13 generates ranking data including information about the corrected similarity ranking calculated in step S16, and outputs the ranking data to the outside of the image search system 10 as a search result.

[0216] The processing unit 13 can supply the ranking data to the storage unit 15 or the database 17 via the transmission path 12. The processing unit 13 can also supply the ranking data to the output unit 19 via the transmission path 12. This allows the output unit 19 to supply the ranking data to the outside of the image retrieval system 10.

[0217] The ranking data may include the ranking of the similarity of each database image to the query image, the similarity value, etc. Preferably, the ranking data includes a file path to the database image. This allows a user of the image retrieval system 10 to easily access a target image from the ranking data. Tags linked to the query image and the output database image may also be viewable. Furthermore, for example, if publication data representing publications in which database images are published is stored in the database 17, a user of the image retrieval system 10 can easily access publications in which database images linked to the ranking data are published. The above is an example of an image retrieval method using the image retrieval system 10.

[0218] In the image search method using the image search system 10, first, the database image data GD is searched without being associated with tags. DB Query image data GD Q The similarity is calculated for each image. Then, tags are linked and the similarity is corrected. This makes it possible to prevent searches for database images that have similar features to the query image but different concepts, technical content, or points of interest.

[0219] For example, query image data GD Q The database image data GD with the first to fifth highest similarity to DBDatabase tag associated with TAG DB Based on this, the processing unit 13 calculates the query tag TAG Q In this case, the database image data GD DB This prevents image data that differs from the query image in terms of concept, technical content, focus, etc. from being mixed in with the query image. This prevents images that become noise from being mixed into the search results, preventing the desired image from being output. As a result, the image search system 10 can perform searches with high accuracy.

[0220] In addition, in the image search method using the image search system 10, the database tag TAG DB Based on the query tag TAG Q The acquisition method is to acquire the database tag TAG DB Query without tagging Q This is a simpler method than the method of acquiring the database tag TAG. Therefore, the image search system 10 can perform a search in a short time. DB Based on the query tag TAG Q For example, a method for acquiring all query tags TAG Q Specify and query tag TAG to the user Q Compared with the case where no candidates are presented, the query image data GD Q It is possible to comprehensively acquire tags that represent the concept, characteristics, technical content, points of interest, etc. of the image corresponding to the image. Therefore, the image search system 10 can perform searches easily and with high accuracy.

[0221] <1-3. Image search method-2> In the image search method shown in FIG. 9 etc., a user of the image search system 10 inputs a query tag TAG Q 14 shows a case where a user of the image search system 10 inputs a query tag TAG Q14 is a flowchart showing an example of an image retrieval method using the image retrieval system 10 when a part of the image is manually input. Even when operating the image retrieval system 10 using the method shown in Fig. 14, it is preferable to perform the process shown in Fig. 2 in advance, as in the case where the image retrieval system 10 is operated using the image retrieval method shown in Fig. 9.

[0222] [Step S21] First, a user of the image retrieval system 10 inputs query image data GD Q In addition, query tags Q is input to the input unit 11. The query tag TAG input by the user of the image search system 10 Q and the query tag TAG Q The content of can be set by the user. Also, the query tag TAG that will be automatically acquired in a later step Q Query tags, including Q The number of the items may be set by the user.

[0223] FIG. 15 shows the query image data GD Q , and query tag TAG Q 15 is a diagram showing input to the input unit 11. In the case shown in FIG. 15, the user of the image retrieval system 10 inputs query image data GD Q In addition, query image data GD Q Two query tags, "circuit diagram" and "semiconductor", represent Q is being entered.

[0224] Here, the query tag TAG input to the input unit 11 Q By changing the database image data GD DB Query image data GD Q For example, the similarity calculation result for the query tag "capacitor" can be changed. Q is input to the input unit 11, the similarity of database image data representing a circuit diagram in which no capacitive element is drawn can be reduced.

[0225] [Step S22] Next, the query image data GD Q is input to the neural network included in the processing unit 13. For example, the query image data GD Q As a result, the processing unit 13 inputs the query image data GD Q Query image feature data GFD representing the feature Q can be obtained.

[0226] [Step S23] Next, the processing unit 13 extracts the database image feature data GFD DB and the database tag vector TAGV DB and data D including DB Furthermore, the processing unit 13 acquires the query image feature amount data GFD Q and the query tag vector TAGV Q and data D including Q Get.

[0227] Here, one database image data GD DB Database tag associated with TAG DB The number of the query tags TAG Q If the number of data is greater than the number of data D DB The tags included in the database image data GD DB For example, select from tags associated with one database image data GD DB 5 database tags in TAG DB The query tag TAG input to the input unit 11 is Q In this case, there are five database tags TAG DB For example, the tag with the highest TF-IDF and the tag with the second highest TF-IDF are grouped into data D DB The tag may be a tag that

[0228] [Step S24] Next, data GD DBData GD Q The processing unit 13 calculates the similarity to the above-mentioned. The similarity can be calculated by the same method as shown in FIG.

[0229] [Step S25] Next, Data D DB Data D Q Based on the calculation result of the similarity to the query tag TAG Q Add and modify.

[0230] 16A and 16B show the query tag TAG Q 16A is a diagram showing an example of a method for adding data D DB In Figure 16A, 100 data D DB For example, the most data D Q Data D that has high similarity to DB In the case shown in FIG. 16A, data D DB The similarity of [2] is 0.999, the highest, and data D DB

[41] has the second highest similarity score of 0.971, and data D DB The similarity score of

[53] is 0.964, which is the third highest. DB The similarity score of

[22] is 0.951, which is the fourth highest. DB

[88] has the fifth highest similarity score of 0.937.

[0231] Next, data D with high similarity DB The database image data GD DB Database tag associated with TAG DB In the case shown in FIG. 16A, the data D DB The database image data GD DB Database tag associated with TAG DB Specifically, the database image data GD DB[2] The tags "aaa", "bbb", "ccc", "ddd", and "eee" associated with the database image data GD DB

[41] and the database image data GD DB

[53] and the tags "aaa", "bbb", "fff", "iii", and "kkk" associated with the database image data GD DB

[22] and the database image data GD DB The tags “aaa,” “kkk,” “rrr,” “sss,” and “ttt” associated with

[88] are extracted. As in the case shown in FIG. 11A, the extracted tags may overlap.

[0232] Thereafter, as shown in FIG. 16B, the number of occurrences of each of the extracted tags is calculated.

[0233] Next, as shown in FIG. 16B, a predetermined number of tags are further extracted from the extracted tags, and the extracted tags are used as a new query tag TAG Q In the case shown in FIG. 16B, two tags ("circuit diagram" and "semiconductor") have already been added to the query tag TAG Q By adding three tags, the query tag TAG Q The number of images in one database GD DB Database tag associated with TAG DB Let's say there are 5, which is the same number as above.

[0234] New query tag TAG Q The extraction of tags that satisfy the condition can be performed in the same manner as the method shown in FIG. 11B. For example, tags can be extracted in order starting from the tag that appears most frequently. In addition, if there are multiple tags that appear the same number of times but it is not possible to extract all of the multiple tags, data D with a higher similarity can be extracted. DB The database image data GDDB In the case shown in FIG. 16B, the tags “aaa”, “bbb”, and “ccc” can be extracted as a new query tag TAG Q It can be extracted as:

[0235] In summary, in the case shown in FIG. 16B, the tags "circuit diagram" and "semiconductor" input by the user of the image search system 10 to the input unit 11 in step S21, as well as the five additional tags "aaa", "bbb", and "ccc", are used as a new query tag TAG Q It can be said that:

[0236] Note that a part or all of the information input by the user of the image search system 10 to the input unit 11 is referred to as a query tag TAG Q For example, you can delete the tags "circuit diagram" and "semiconductor" from the tag TAG Q 16B, and five tags are extracted from the tags shown in FIG. 16B to create a new tag TAG Q In this case, for example, the tags "aaa", "bbb", "ccc", "fff", and "ggg" can be used as a new tag TAG Q It can be said that:

[0237] [Step S26] Next, the query tag TAG Q In response to additions and modifications, data D DB Add or modify tags that belong to one data D. DB The database tag vector TAGV DB The number of query tags TAG Q Make it equal to the number of .

[0238] [Step S27] Next, data GD DB Data GD Q The processing unit 13 again calculates the similarity to the data GD. The similarity can be calculated by the same method as shown in step S24. DB Data GD Q The similarity to the

[0239] [Step S28] Next, processing unit 13 generates ranking data including information about the corrected similarity ranking calculated in step S27, and outputs the ranking data as a search result to the outside of image retrieval system 10. This allows the user of image retrieval system 10 to check, for example, the similarity ranking of each database image with respect to the query image, the similarity value, the retrieved database images, tags, etc.

[0240] [Step S29, Step S30] Next, the user of the image search system 10 checks whether the ranking data is the expected result. If the expected result is obtained, the search ends. If the expected result is not obtained, the user of the image search system 10 checks whether the ranking data is the expected result. Q After adding or correcting, the process returns to step S23. The above is an example of an image search method using the image search system 10.

[0241] This embodiment mode can be combined with other embodiment modes as appropriate. In addition, in this specification, when a plurality of configuration examples are shown in one embodiment mode, the configuration examples can be combined as appropriate.

[0242] (Embodiment 2) In the first embodiment, the image retrieval system 10 includes a database of image data GD DB and the entire region of the query image data GD Q By comparing the entire area of ​​with the database image data GD DB Query image data GD Q However, the present invention is not limited to this. For example, the similarity of the database image data GD DB and the query image data GD Q By comparing the entire area of ​​with the database image data GD DB Query image data GD Q Alternatively, the similarity to the database image data GD DBand the entire region of the query image data GD Q By comparing some areas of the image data GD DB Query image data GD Q Alternatively, the similarity to the

[0243] <2-1. Image search method-3> FIG. 17 shows the database image data GD DB and the query image data GD Q By comparing the entire area of ​​with the database image data GD DB Query image data GD Q 14. This is an example of an image retrieval method using the image retrieval system 10 when calculating the similarity to the image. First, the image retrieval system 10 performs step S11 shown in FIG. 9 or step S21 shown in FIG.

[0244] [Step S31] Next, the processing unit 13 calculates the query image data GD Q and database image data GD DB Compare with the query image data GD Q Database image data GD containing areas with high matching scores DB Here, the extracted database image data GD DB The extracted image data GD Ex Query image data GD Q and database image data GD DB The comparison can be performed, for example, by region-based matching.

[0245] An example of the operation of step S31 will be described in detail with reference to Figs. 18 to 20. In step S31, as shown in Fig. 18A, the query image data GD Q n (n is an integer greater than or equal to 1) database image data GD DB where n is the database image data GD stored in the database 17. DBThe number n may be equal to or less than the number of database image data GD stored in the database 17. DB In this case, the number of database image data GD stored in the database 17 may be larger than the number of DB In addition to the above, image data stored in the storage unit 15 or image data input to the processing unit 13 via the input unit 11 from outside the image search system 10, and query image data GD Q It should be noted that n is the database image data GD DB Even if the number of images is less than the number of images stored in the storage unit 15 or image data input to the processing unit 13 via the input unit 11 from outside the image search system 10, the query image data GD Q may be compared with.

[0246] When n is small, the operation of step S31 can be performed in a short time. On the other hand, when n is large, the query image data GD Q Database image data GD containing areas with high matching scores DB can be extracted with high accuracy.

[0247] FIG. 18B shows the query image data GD Q and database image data GD DB 1 is a diagram illustrating a procedure for comparing query image data GD Q The number of pixels of the corresponding image is 2 × 2, and the database image data GD DB The number of pixels of the image corresponding to the query image data GD Q has 2x2 pixel values, and the database image data GD DB has 4x4 pixel values.

[0248] In FIG. 18B, the query image data GD Q The pixel values ​​of the 2×2 pixels in the query image data GD are respectively set to vq11, vq12, vq21, and vq22. QIn this example, the pixel value corresponding to the pixel in the first row and first column is pixel value vq11, the pixel value corresponding to the pixel in the first row and second column is pixel value vq12, the pixel value corresponding to the pixel in the second row and first column is pixel value vq21, and the pixel value corresponding to the pixel in the second row and second column is pixel value vq22. DB The 4×4 pixel values ​​of the database image data GD are respectively set to pixel values ​​vdb11 to vdb44. DB In this example, the pixel value corresponding to the pixel in the first row and first column is pixel value vdb11, the pixel value corresponding to the pixel in the first row and fourth column is pixel value vdb14, the pixel value corresponding to the pixel in the fourth row and first column is pixel value vdb41, and the pixel value corresponding to the pixel in the fourth row and fourth column is pixel value vdb44.

[0249] First, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb11, vdb12, vdb21, and vdb22. Q and database image data GD DB 18B, the degree of coincidence between the database image data GD DB Among the pixel values ​​of the query image data GD Q The pixel values ​​to be compared are indicated as a comparison data area 21 surrounded by a dotted line.

[0250] Next, the comparison data area 21 is DB The pixel values ​​are shifted by one column for the pixel values ​​of the query image data GD, and the pixel values ​​are compared in the same way to calculate the degree of match. Specifically, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb12, vdb13, vdb22, and vdb23. Q and database image data GD DB It is possible to calculate the degree of coincidence between the pixel values ​​vdb12, vdb13, vdb22, and vdb23 and the region formed by the pixel values ​​vdb14, vdb15, vdb16, vdb17, and vdb18.

[0251] After that, the comparison data area 21 is also DB The pixel values ​​are shifted by one column for the pixel values ​​of the query image data GD, and the pixel values ​​are compared in the same way to calculate the degree of match. Specifically, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb13, vdb14, vdb23, and vdb24. Q and database image data GD DB It is possible to calculate the degree of coincidence between the pixel values ​​vdb13, vdb14, vdb23, and vdb24 and the region formed by the pixel values ​​vdb15, vdb16, vdb17, vdb18, and vdb29.

[0252] Next, the comparison data area 21 is DB The pixel values ​​of the database image data GD DB The pixel values ​​of the second and third rows of the query image data GD Q As described above, the pixel values ​​constituting the database image data GD DB The area consisting of the pixel values ​​of the second and third rows of the query image data GD Q The degree of agreement can be calculated for each column in the same manner as above.

[0253] Then, the comparison data area 21 is stored in the database image data GD DB The pixel values ​​of the database image data GD DB The pixel values ​​in the third and fourth rows of the query image data GD Q As described above, the pixel values ​​constituting the database image data GD DB The pixel values ​​of the third and fourth rows of the query image data GD Q The degree of agreement can be calculated for each column in the same manner as above.

[0254] After the above operations are performed, for example, the highest degree of match is selected as the database image data GDDB Query image data GD Q The above is the degree of match for n database image data GD DB Then, n pieces of database image data GD DB From among them, query image data GD Q Database image data GD with high degree of match DB The extracted image data GD Ex For example, the database image data GD DB Extract the specified number of image data GD Ex Alternatively, for example, the query image data GD Q Database image data GD that matches with the specified value or more DB The extracted image data GD Ex It may be extracted as

[0255] Also, the database image data GD DB This is because the query image data GD Q The database image data GD was compared with DB All of the extracted image data GD Ex This can be rephrased as:

[0256] FIG. 19 shows the database image data GD DB 19 is a diagram illustrating the extraction of database image data GD DB [1] to database image data GD DB [3] Extract one image data from image data GD Ex An example of extracting the following is shown.

[0257] The query image data GD shown in FIG. Q The image corresponding to the database image data GD shown in FIG. DB The image corresponding to [2] contains a transistor symbol, but the database image data GD DB[1] and database image data GD DB The image corresponding to [3] does not contain a transistor symbol. In this case, the database image data GD DB [2] Query image data GD Q The degree of match is calculated using the database image data GD DB [1] and database image data GD DB [3] Query image data GD Q Therefore, the degree of match is higher than that for the database image data GD DB [2] is extracted image data GD Ex It can be extracted as:

[0258] In addition, the query image data GD Q and database image data GD DB Comparison and calculation of the degree of agreement can be performed using SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), NCC (Normalized Cross Correlation), ZNCC (Zero-mean Normalized Cross Correlation), POC (Phase-Only Correlation), etc.

[0259] In addition, in FIG. 18B, the comparison data area 21 is the database image data GD DB However, the present invention is not limited to this. DBmay be shifted by two or more columns or two or more rows relative to the pixel values ​​of vdb11, vdb12, vdb21, and vdb22. For example, immediately after comparing pixel values ​​vq11, vq12, vq21, and vq22 with pixel values ​​vdb11, vdb12, vdb21, and vdb22, pixel values ​​vq11, vq12, vq21, and vq22 with pixel values ​​vdb13, vdb14, vdb23, and vdb24. In this case, comparisons of pixel values ​​vq11, vq12, vq21, and vq22 with pixel values ​​vdb12, vdb13, vdb22, and vdb23 are not performed. Furthermore, immediately after comparing pixel values ​​vq11, vq12, vq21, and vq22 with pixel values ​​vdb13, vdb14, vdb23, and vdb24, pixel values ​​vq11, vq12, vq21, and vq22 may be compared with pixel values ​​vdb31, vdb32, vdb41, and vdb42.

[0260] By increasing the movement range of the comparison data area 21, the query image data GD Q and the database image data GD DB This reduces the number of comparisons between the pixel values ​​of and the database image data GD DB Query image data GD Q The degree of match can be calculated in a short time.

[0261] In FIG. 18A, one query image data GD Q n database image data GD DB 20A, the query image data GD Q Based on this, multiple query image data GD with different numbers of pixel values ​​are Q 20A shows the query image data GD QBased on this, query image data GD Q [1], Query image data GD Q [2], and query image data GD Q As shown in FIG. 20A, the query image data GD Q [1] The number of pixels of the image corresponding to the query image data GD Q [2] The number of pixels of the image corresponding to the query image data GD Q The number of pixels of the image corresponding to [3] is different. That is, the query image data GD Q [1] to query image data GD Q The image corresponding to [3] is the query image data GD Q It can be said that the image is an enlarged or reduced version of the image corresponding to the image.

[0262] Multiple query image data GD Q When generating the query image data GD Q For each of these, the database image data GD DB [1] to database image data GD DB [n]. This compares the database image data GD DB [1] to database image data GD DB For each of [n], multiple query image data GD Q Then, for example, the degree of match for each of the plurality of query image data GD Q The highest match score among the scores for the database image data GD DB The query image data GD input to the processing unit 13 Q It can be the degree of agreement with

[0263] For example, in the case shown in FIG. 20A, the query image data GD Q [1] is the database image data GD DB [1] to database image data GD DB [n] and compare it with the query image data GD Q[2] is the database image data GD DB [1] to database image data GD DB [n] and compare it with the query image data GD Q [3] is used to create a database of image data. DB [1] to database image data GD DB [n]. This compares the database image data GD DB [1] to database image data GD DB For each of [n], the query image data GD Q [1] Matching score for query image data GD Q [2] and query image data GD Q The degree of match to [3] can be calculated.

[0264] For example, the query image data GD Q [1] Matching score for query image data GD Q [2] and query image data GD Q The highest match score for [3] is calculated as the database image data GD DB The query image data GD input to the processing unit 13 Q For example, the database image data GD DB [1] Query image data GD Q [1] Matching score for query image data GD Q [2] and query image data GD Q The highest match score for [3] is calculated as the database image data GD DB [1] The query image data GD input to the processing unit 13 Q It can be the degree of agreement with

[0265] Query image data GD Q and the database image data GD DB Even if the image corresponding to and the image corresponding to have the same elements are shown in different sizes, the database image data GD DBQuery image data GD Q In the case shown in FIG. 20B1, it may be determined that the query image data GD Q and the database image data GD DB Both the image corresponding to and the image corresponding to show the same element, the transistor symbol. However, the query image data GD Q The size of the transistor symbol shown in the corresponding image and the database image data GD DB In this case, the size of the transistor symbol shown in the corresponding image is different from the database image data GD DB Query image data GD Q The degree of match may be judged to be low.

[0266] On the other hand, in the case shown in FIG. 20B2, the query image data GD Q and the database image data GD DB The image corresponding to the database image data GD DB Query image data GD Q The processing unit 13 can determine that the region includes a region with a high degree of match with the region.

[0267] As shown in FIG. 20A, a plurality of query image data GD Q By generating the query image data GD Q Therefore, the size of the elements shown in the image corresponding to the query image data GD Q and the database image data GD DB Even if the same elements are shown in different sizes in the image corresponding to the query image data GD shown in FIG. 20B1, the degree of match between the two image data can be made high. Q When the query image data GD QThe query image data GD shown in FIG. 20B2 is generated by varying the number of pixel values ​​of Q By generating the database image data GD DB Query image data GD Q As a result, the degree of match with the database image data GD DB The query image data GD input to the processing unit 13 Q The degree of match for the above can be calculated with high accuracy.

[0268] [Step S32] Next, the extracted image data GD Ex From the query image data GD Q Partial image data GD part For example, the processing unit 13 extracts the database image data GD DB Query image data GD for each region Q When the degree of match is calculated for each of the partial image data GD part Therefore, the partial image data GD part The number of pixel values ​​that the query image data GD Q can be equal to the number of pixel values ​​that .

[0269] 21A and 21B are diagrams showing an example of the operation of step S32. Ex [1] to extracted image data GD Ex In [4], the query image data GD Q As shown in FIGS. 21A and 21B, the hatched areas are extracted and the partial image data GD part [1] to partial image data GD part [4]. In FIG. 21A and FIG. 21B, the extracted image data GD Ex [1] to extracted image data GD Ex The image data extracted from [4] are divided into partial image data GD part[1] to partial image data GD part [4]

[0270] FIG. 21A shows one query image data GD Q On the other hand, the database image data GD DB In this example, partial image data GD part The number of pixels of the images corresponding to the above can all be made equal.

[0271] FIG. 21B shows a plurality of query image data GDs with different numbers of pixel values ​​as shown in FIG. 20A. Q On the other hand, the database image data GD DB In this example, partial image data GD part The number of pixels of the image corresponding to the query image data GD Q Therefore, the number of pixels of the image corresponding to the partial image data GD part If there are multiple partial image data GD part The number of pixels in the image corresponding to the partial image data GD part In FIG. 21B, the partial image data GD part [1] to partial image data GD part [4] shows an example where the number of pixels in the corresponding images is different.

[0272] In addition, the partial image data GD part In this case, the partial image data GD part Extract image data GD appropriately Ex The following explanation can be applied by reading it as: Ex The entire image data GD part For example, the query image data GD Q and extracted image data GD Ex By increasing or decreasing the number of pixel values ​​of one or both of the image data of the partial image data GD partThe image retrieval method using the image retrieval system 10 can be performed without extracting the above.

[0273] [Step S33] Next, the query image data GD Q is input to the neural network of the processing unit 13, the processing unit 13 outputs the query image feature data GFD Q Also, partial image data GD part is input to the neural network of the processing unit 13, the processing unit 13 generates the database image feature data GFD DB Get the query image data GD Q , and partial image data GD part can be input to the neural network 30 having the configuration shown in FIG. 3A or 3B. When the image retrieval system 10 is operated in the manner shown in FIG. 17, step S02 shown in FIG. 2 does not need to be performed. DB Database image feature data GFD that represents the features of the entire region DB does not need to be obtained.

[0274] As explained in the first embodiment, the database image data GD DB Here, it is preferable that the number of pixel values ​​in the image data used for the learning data is equal to the number of pixel values ​​in the image data input to the neural network 30. Therefore, when the neural network 30 performs learning, the database image data GD DB It is preferable to adjust the number of pixel values ​​by increasing or decreasing the number of pixel values ​​as necessary. Q , or partial image data GD part When entering the query image data GD Q , or partial image data GD partPreferably, the number of pixel values ​​in the image is increased or decreased. Here, the increase in pixel value is preferably achieved by padding, for example, by zero padding.

[0275] FIG. 22A shows the database image data GD DB 22A is a diagram illustrating adjustment of the number of pixel values ​​in the database image data GD DB [1] to database image data GD DB In this case, the number of pixel values ​​in the database image data GD DB [1] to database image data GD DB When [4] is used as training data for the neural network 30, it is preferable to make the number of pixel values ​​of these image data uniform as shown in FIG. 22A.

[0276] FIG. 22B shows the partial image data GD part 10 is a diagram illustrating adjustment of the number of pixel values ​​in the partial image data GD. part It is preferable that the number of pixel values ​​of the query image data GD be equal to the number of pixel values ​​of the image data used for training the neural network 30. Similarly, Q When entering the query image data GD Q It is preferable that the number of pixel values ​​contained in the image data used for training the neural network 30 be equal to the number of pixel values ​​contained in the image data used for training the neural network 30.

[0277] After performing step S33, the image retrieval system 10 performs step S13 shown in Fig. 9 or step S23 shown in Fig. 14. Specifically, if step S11 is performed before step S31, step S13 is performed after step S33, and if step S21 is performed before step S31, step S23 is performed after step S33. DB and the query image data GD Q By comparing the entire area of ​​with the database image data GDDB Query image data GD Q 10 is an example of an image retrieval method using the image retrieval system 10 when calculating the similarity between the images.

[0278] In the method shown in FIG. 17, the query image data GD Q and database image data GD DB and the query image data GD Q Database image data GD containing areas with high matching scores DB The extracted image data GD Ex Then, the extracted image data GD Ex The area with the highest degree of match is then extracted as partial image data GD part Extract the query image data as GD Q and partial image data GD part is input to the neural network of the processing unit 13. In this way, the database image data GD DB By extracting the query image data GD Q Database image data GD representing database images that do not contain images that are highly similar to the corresponding images DB can be prevented from being input to the neural network of the processing unit 13. Q It is possible to search for database images that partially contain images similar to the image corresponding to the query image data GD Q Database image data GD to compare with DB If the number of images is small, the database image data GD DB The above search can be performed with high accuracy in a short time without extracting the above.

[0279] <2-2. Image search method-4> FIG. 23 shows the database image data GD DB and the entire region of the query image data GD Q By comparing some areas of the image data GD DB Query image data GDQ 14. This is an example of an image retrieval method using the image retrieval system 10 when calculating the similarity to the image. First, the image retrieval system 10 performs step S11 shown in FIG. 9 or step S21 shown in FIG.

[0280] [Step S41] Next, the processing unit 13 calculates the query image data GD Q and database image data GD DB Compare with the query image data GD Q Database image data GD with high matching degree to part of DB The extracted image data GD Ex Query image data GD Q and database image data GD DB The comparison with can be performed by, for example, region-based matching, as in step S31.

[0281] An example of the operation of step S41 will be described in detail with reference to Fig. 24 and Fig. 25. In step S41, as shown in Fig. 24A, the query image data GD Q n database image data GD DB Compare with each of the above.

[0282] FIG. 24B shows the query image data GD Q and database image data GD DB 1 is a diagram illustrating a procedure for comparing query image data GD Q The pixel size of the corresponding image is 4x4, and the database image data GD DB The number of pixels of the image corresponding to the query image data GD Q has 4x4 pixel values, and the database image data GD DB has 2x2 pixel values.

[0283] In FIG. 24B, the query image data GD Q The 4×4 pixel values ​​of the query image data GD are respectively set to pixel values ​​vq11 to vq44.Q In this example, the pixel value corresponding to the pixel in the first row and first column is pixel value vq11, the pixel value corresponding to the pixel in the first row and fourth column is pixel value vq14, the pixel value corresponding to the pixel in the fourth row and first column is pixel value vq41, and the pixel value corresponding to the pixel in the fourth row and fourth column is pixel value vq44. DB The 2×2 pixel values ​​of the database image data GD are respectively set to pixel values ​​vdb11, vdb12, vdb21, and vdb22. DB In this example, the pixel value corresponding to the pixel in the first row and first column is pixel value vdb11, the pixel value corresponding to the pixel in the first row and second column is pixel value vdb12, the pixel value corresponding to the pixel in the second row and first column is pixel value vdb21, and the pixel value corresponding to the pixel in the second row and second column is pixel value vdb22.

[0284] First, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq11, vq12, vq21, and vq22. DB and the query image data GD Q 24B, the degree of coincidence between the query image data GD Q Among the pixel values ​​of the database image data GD DB The pixel values ​​to be compared are indicated as a comparison data area 21 surrounded by a dotted line.

[0285] Next, the comparison data area 21 is Q The pixel values ​​are shifted by one column and compared in the same way to calculate the degree of match. Specifically, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq12, vq13, vq22, and vq23. This results in the database image data GD DB and the query image data GD QIt is possible to calculate the degree of coincidence between the pixel value vq12, the pixel value vq13, the pixel value vq22, and the pixel value vq23.

[0286] After that, the comparison data area 21 is also Q The pixel values ​​are shifted by one column and compared in the same way to calculate the degree of match. Specifically, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq13, vq14, vq23, and vq24. As a result, the database image data GD DB and the query image data GD Q It is possible to calculate the degree of coincidence between the pixel value vq13, the pixel value vq14, the pixel value vq23, and the pixel value vq24.

[0287] Next, the comparison data area 21 is Q The pixel values ​​of the query image data GD Q The pixel values ​​of the second and third rows of the database image data GD DB As described above, the pixel values ​​constituting the query image data GD Q The area consisting of the pixel values ​​in the second and third rows of the database image data GD DB The degree of agreement can be calculated for each column in the same manner as above.

[0288] Then, the comparison data area 21 is Q The pixel values ​​of the query image data GD Q The pixel values ​​in the third and fourth rows of the database image data GD DB As described above, the pixel values ​​constituting the query image data GD Q The area consisting of the pixel values ​​in the third and fourth rows of the database image data GD DB The degree of agreement can be calculated for each column in the same manner as above.

[0289] After the above operations are performed, for example, the highest degree of match is selected as the database image data GD DB Query image data GD Q The above is the degree of match for n database image data GD DB Then, in the same manner as in step S31, the n pieces of database image data GD DB From among them, query image data GD Q Database image data GD with high degree of match DB The extracted image data GD Ex As in step S31, the database image data GD DB It is not necessary to extract the above.

[0290] FIG. 25 shows the database image data GD DB 25 is a diagram illustrating the extraction of database image data GD DB [1] to database image data GD DB [3] Extract one image data from image data GD Ex An example of extracting the following is shown.

[0291] The query image data GD shown in FIG. Q The image corresponding to the database image data GD shown in FIG. 25 includes, for example, a symbol of a transistor and a symbol of a capacitor. DB The image corresponding to [2] contains a transistor symbol, but the database image data GD DB [1] and database image data GD DB The image corresponding to [3] does not include any symbols of transistors or capacitor elements. In this case, the database image data GD DB [2] Query image data GD Q The degree of match is calculated using the database image data GD DB [1] and database image data GD DB [3] Query image data GD QTherefore, the degree of match is higher than that for the database image data GD DB [2] is extracted image data GD Ex It can be extracted as:

[0292] In addition, the query image data GD Q and database image data GD DB The comparison and calculation of the degree of match can be performed using the same method as that used in step S31. Q In step S31, the comparison data area 21 is shifted by one column or one row for the pixel values ​​of the query image data GD Q 20A, the pixel values ​​of the query image data GD Q Based on this, multiple query image data GD with different numbers of pixel values ​​are Q may be generated.

[0293] [Step S42] Next, the query image data GD Q Extracted image data GD Ex Partial image data GD part-Q For example, the processing unit 13 extracts the query image data GD Q Database image data GD DB When the degree of match is calculated for each of the partial image data GD part-Q Therefore, the partial image data GD part-Q The number of pixel values ​​in the extracted image data GD Ex can be equal to the number of pixel values ​​that .

[0294] 26 is a diagram showing an example of the operation of step S42. In the case shown in FIG. 26, the query image data GD Q The upper left part of the image corresponding to the extracted image data GD ExThis is the area with the highest degree of match for [1]. Q The data corresponding to the upper left area is called partial image data GD part-Q [1]. On the other hand, the query image data GD Q The lower right part of the image corresponding to the extracted image data GD Ex This is the area with the highest degree of match for [2]. Q The data corresponding to the lower right area is called partial image data GD part-Q [2]. In other words, one query image data GD Q Multiple partial image data GD part-Q is extracted.

[0295] In step S42, the extracted image data GD Ex The same number of image data is used as the query image data GD Q Partial image data GD part-Q Alternatively, the extracted image data GD Ex The smaller number of image data is used as the query image data GD Q Partial image data GD part-Q For example, multiple extracted image data GD Ex For the query image data GD Q If the regions are the same, the query image data GD Q Partial image data GD extracted from part-Q In other words, the number of the same partial image data GD part-Q Query image data GD Q There is no need to extract multiple items from the same table.

[0296] In addition, the partial image data GD part-Q In this case, the partial image data GD part-Q Query image data GD accordingly Q Alternatively, the query image data GD Q The entire image data GD part-QFor example, the query image data GD Q and extracted image data GD Ex By increasing or decreasing the number of pixel values ​​of one or both of the image data of the partial image data GD part-Q The image retrieval method using the image retrieval system 10 can be performed without extracting the above.

[0297] [Step S43] Next, the partial image data GD part-Q and extracted image data GD Ex are input to the neural network of the processing unit 13.

[0298] The operation in step S43 is to appropriately use the query image data GD Q Partial image data GD part-Q Partial image data GD part Extract image data GD Ex The explanation of step S33 can be referred to by replacing the query image data GD Q Extract image data GD Ex Partial image data GD part Partial image data GD part-Q It may also be read as:

[0299] The above is the database image data GD DB and the entire region of the query image data GD Q By comparing a part of the area with the database image data GD DB Query image data GD Q 10 is an example of an image retrieval method using the image retrieval system 10 when calculating the similarity between the images.

[0300] In the method shown in FIG. 23, the query image data GD Q and database image data GD DB and the query image data GD Q Database image data GD with high matching degree to part of DBThe extracted image data GD Ex Then, the query image data GD Q The area with the highest degree of match is then extracted as partial image data GD part-Q The partial image data GD part-Q and extracted image data GD Ex is input to the neural network of the processing unit 13. In this way, the database image data GD DB By extracting the query image data GD Q Database image data GD representing database images that do not contain images that are highly similar to the corresponding images DB can be prevented from being input to the neural network of the processing unit 13. Q It is possible to search for database images similar to a part of the image corresponding to the query image data GD Q Database image data GD to compare with DB If the number of images is small, the database image data GD DB The above search can be performed with high accuracy in a short time without extracting the above.

[0301] This embodiment mode can be combined with other embodiment modes as appropriate. In addition, in this specification, when a plurality of configuration examples are shown in one embodiment mode, the configuration examples can be combined as appropriate. [Example]

[0302] In this embodiment, an image is input into an image search system, and the results of searching for database images similar to the input image will be described.

[0303] In this example, one image was input as a query image into the image search system. Then, 100 database images similar to the image were searched for under conditions 1 and 2. In both conditions 1 and 2, the query image was a schematic diagram of a semiconductor manufacturing device. Furthermore, the database images included schematic diagrams of semiconductor manufacturing devices, as well as circuit diagrams, circuit layout diagrams, block diagrams, and the like. Furthermore, the database images were drawings published in patent documents.

[0304] 9 were performed to calculate the similarity of the database image to the query image. Then, step S17 was performed to generate ranking data representing the database images with the highest similarity from 1 to 100.

[0305] Under Condition 2, first, database tags were obtained from the specification of the same patent document as the patent document in which the database image was published, in steps S01 to S04 shown in Fig. 2. Next, steps S11 to S13 shown in Fig. 9 were performed to calculate the similarity of the database image to the query image, and then steps S14 to S16 were performed to correct the similarity. Then, step S17 was performed to generate ranking data representing the database images ranked 1st to 100th in descending order of similarity after the correction.

[0306] Under condition 1, of the database images ranked 1st to 100th in similarity to the query image, 14 images represented semiconductor manufacturing equipment, just like the query image, and the remaining 86 images represented circuit diagrams, circuit layout diagrams, block diagrams, etc. On the other hand, under condition 2, all 100 images represented semiconductor manufacturing equipment.

[0307] From the above, it was confirmed that under condition 2, it was possible to retrieve more database images that are conceptually similar to the query image than under condition 1. [Explanation of symbols]

[0308] 10: Image retrieval system, 11: Input unit, 12: Transmission path, 13: Processing unit, 15: Memory unit, 17: Database, 19: Output unit, 21: Comparison data area, 30: Neural network, 30a: Neural network, 31: Layer, 32: Neuron, 32a: Neuron, 32b: Neuron, 32c: Neuron, 40: Neural network

Claims

1. A database, a processing unit, and an input unit, the database has a function of storing document data and a plurality of database image data; the processing unit has a function of acquiring database image feature amount data representing feature amounts of the database image data for each of the plurality of database image data; the processing unit has a function of generating a plurality of database tags using the document data and linking the database tags to the database image data; the processing unit has a function of acquiring a database tag vector representing the database tag for each of the plurality of database tags; the processing unit has a function of acquiring query image feature amount data representing feature amounts of the query image data when query image data is input to the input unit; the processing unit has a function of calculating a first similarity, which is a similarity of the database image data to the query image data, for each of the plurality of database image data; the processing unit has a function of acquiring a query tag associated with the query image data by using a part of the database tag based on the first similarity; the processing unit has a function of obtaining a query tag vector representing the query tag; the processing unit has a function of acquiring first data including the database image feature data and the database tag vector; the processing unit has a function of acquiring second data including the query image feature amount data and the query tag vector; the processing unit has a function of calculating a second similarity that is a similarity of the first data to the second data, the processing unit has a function of extracting a predetermined number of words from words extracted by performing a morphological analysis on the document data, and generating the database tags. Image search system.

2. A database, a processing unit, and an input unit, the database has a function of storing document data and a plurality of database image data; the processing unit has a function of acquiring database image feature amount data representing feature amounts of the database image data for each of the plurality of database image data; the processing unit has a function of generating a plurality of database tags using the document data and linking the database tags to the database image data; the processing unit has a function of acquiring a database tag vector representing the database tag for each of the plurality of database tags; the processing unit has a function of acquiring query image feature amount data representing feature amounts of the query image data when query image data is input to the input unit; the processing unit has a function of calculating a first similarity, which is a similarity of the database image data to the query image data, for each of the plurality of database image data; the processing unit has a function of acquiring a query tag associated with the query image data by using a part of the database tag based on the first similarity; the processing unit has a function of obtaining a query tag vector representing the query tag; the processing unit has a function of acquiring first data including the database image feature data and the database tag vector; the processing unit has a function of acquiring second data including the query image feature amount data and the query tag vector; the processing unit has a function of calculating a second similarity that is a similarity of the first data to the second data, the processing unit has a function of extracting a predetermined number of words based on an evaluation by TF-IDF from among words extracted by performing a morphological analysis on the document data, and generating the database tags; Image search system.

3. In claim 1 or 2, the processing unit includes a first neural network and a second neural network; the database image feature data and the query image feature data are obtained using the first neural network; The image retrieval system, wherein the database tag vector and the query tag vector are obtained using the second neural network.

4. In claim 3, the first neural network has a convolutional layer and a pooling layer; The database image feature data and the query image feature data are output from the pooling layer.

5. In claim 3 or 4, The image retrieval system, wherein the database tag vector and the query tag vector are distributed representation vectors.

6. In any one of claims 1 to 5, An image retrieval system, wherein the first similarity and the second similarity are cosine similarities.

Images