Image search system

Abstract

To provide an image retrieval system and method with high retrieval accuracy.SOLUTION: The database has a function of storing a plurality of pieces of database image data, and a database tag is associated with each of the plurality of pieces of database image data. The processing unit has a function of acquiring database image feature amount data representing a feature amount of database image data for each piece of database image data, a function of acquiring query image feature amount data representing a feature amount of query image data, a function of calculating a first degree of similarity that is a degree of similarity of the database image data to the query image data for each piece of database image data, and a function of acquiring a query tag that is a tag associated with the query image data using a part of the database tag.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] One aspect of the present invention relates to an image search system and an image search method.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. Examples of the semiconductor device include a semiconductor device, a display device, a light-emitting device, a power storage device, a memory device, an electronic device, a lighting device, Examples include a method for driving them or a method for manufacturing them. [Background technology]

[0003] By conducting a prior art search on a pre-filing invention, you can determine whether or not relevant intellectual property rights exist. The patent documents and papers both in Japan and overseas obtained through prior art searches can be searched. Prior art documents such as these are useful for confirming the novelty and inventive step of an invention and for deciding whether to apply for a patent. In addition, by conducting a search of prior art documents, you can make your own judgment. There is no risk of your patent rights being invalidated, or you are unable to invalidate patent rights owned by others. You can investigate whether this is the case.

[0004] For example, searching for prior art documents containing drawings similar to those embodying the pre-filing technology Specifically, for example, an image search system By inputting drawings into the system, the user can search for prior art documents containing drawings similar to the input drawings. You can search the literature.

[0005] The search for images similar to the input image can be performed using, for example, a neural network. For example, Patent Document 1 discloses a method for calculating the similarity between images using a neural network. A method for determining whether the [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-207947 Summary of the Invention [Problem to be solved by the invention]

[0007] When the similarity between the input image and the image to be searched is calculated using only the image data, For example, images with different concepts from the input image may be retrieved. As a result, noise images may be mixed in, and the image you are looking for may not be output. , the accuracy of searching for similar images may be reduced.

[0008] Therefore, one aspect of the present invention aims to provide an image search system with high search accuracy. Another aspect of the present invention is an image search system that can perform searches in a short time. Another object of the present invention is to provide a method for easily performing a search. Another object of the present invention is to provide an image search system that can One of the objectives is to provide a novel image search system.

[0009] Another object of one aspect of the present invention is to provide an image retrieval method with high retrieval accuracy. Another aspect of the present invention is to provide an image search method that allows searches to be performed in a short time. Another object of the present invention is to provide an image search system that can be easily searched. Another object of the present invention is to provide a novel image search method. One of the objectives is to provide a method.

[0010] Note that the description of these problems does not preclude the existence of other problems. It is not necessary to solve all of these problems. From the description of the section, it is possible to extract other issues. [Means for solving the problem]

[0011] One aspect of the present invention is a system including a database, a processing unit, and an input unit, wherein the database includes a sentence. The processing unit has a function of storing document data and a plurality of database image data. The database image feature data representing the feature of the database image data is then extracted from multiple databases. The processing unit has a function of acquiring data for each of the document data. It is a function that generates multiple database tags and links the database tags to database image data. The processing unit may include a function of converting a database tag vector representing a database tag into a plurality of databases. The processing unit has a function of acquiring the query image data for each of the database tags. When the query image data is input, the query image feature data representing the feature of the query image data is acquired. The processing unit has a function of obtaining a classification of the database image data with respect to the query image data. A first similarity is calculated for each of the plurality of database image data. The processing unit performs a query using a portion of the database tag based on the first similarity. The processing unit has a function of acquiring a query tag associated with the image data, and the processing unit acquires the query tag. The processing unit has a function of acquiring a query tag vector representing the database image feature data. a processing unit having a function of acquiring first data including a database tag vector and a database tag vector; acquires second data including query image feature data and a query tag vector. The processing unit has a function of calculating a second similarity which is a similarity of the first data to the second data. This is an image search system that has the function of calculating the degree of

[0012] Alternatively, in the above aspect, the database tag may include a word.

[0013] Alternatively, in the above aspect, the processing unit performs a morphological analysis on the document data, It may also have the ability to generate database tags.

[0014] Alternatively, in the above aspect, the processing unit may include a first neural network and a second neural network. and a network for generating database image feature data and query image feature data. The data is obtained using a first neural network, and a database tag vector and The query tag vector may be obtained using a second neural network.

[0015] Alternatively, in the above aspect, the first neural network may include a convolution layer and a pooling layer. and a layer, and the database image feature data and the query image feature data are It may be output from the encoding layer.

[0016] Alternatively, in the above aspect, the database tag vector and the query tag vector are distributed It may be a representation vector.

[0017] Alternatively, in the above aspect, the first similarity and the second similarity are cosine similarities, Good too.

[0018] Alternatively, one aspect of the present invention is a database in which document data and a plurality of database images are stored. An image retrieval method using an image retrieval system having a database and an input unit, The database image feature data representing the feature of the database image data is extracted from multiple databases. The database tags are generated using the document data. Generate and associate database tags with database image data, and represent the database tags. A database tag vector is obtained for each of the multiple database tags, and the input Query image data is input to the input section, and query image feature data representing the feature of the query image data is generated. and obtain a first similarity of the database image data to the query image data. A similarity is calculated for each of the plurality of database image data, and a first similarity is calculated based on the first similarity. Based on this, a part of the database tag is used to generate a query tag that can be linked to the query image data. The query tag vector representing the query tag is obtained, and the database image feature data is and a first data including a database tag vector, a query image feature data, and a query and second data including an ERI tag vector, and This is an image retrieval method for calculating a second similarity, which is a similarity to the first similarity.

[0019] Alternatively, in the above aspect, the database tag may include a word.

[0020] Alternatively, in the above aspect, a morphological analysis is performed on the document data to generate a database. You may also generate a tag.

[0021] Alternatively, in the above aspect, the database image feature amount data and the query image feature amount data is obtained using a first neural network, and the database tag vector and The ERI tag vector may be obtained using a second neural network.

[0022] Alternatively, in the above aspect, the first neural network may include a convolution layer and a pooling layer. and a layer, and the database image feature data and the query image feature data are It may be output from the encoding layer.

[0023] Alternatively, in the above aspect, the database tag vector and the query tag vector are distributed It may be a representation vector.

[0024] Alternatively, in the above aspect, the first similarity and the second similarity are cosine similarities, Good too. [Effects of the Invention]

[0025] According to one aspect of the present invention, it is possible to provide an image search system with high search accuracy. According to one aspect of the present invention, an image search system capable of performing searches in a short time is provided. Alternatively, according to one aspect of the present invention, an image search system that allows for easy search can be provided. Alternatively, according to one aspect of the present invention, a novel image retrieval system can be provided. can be provided.

[0026] According to one aspect of the present invention, an image search method with high search accuracy can be provided. According to one aspect of the present invention, there is provided an image search method that allows searches to be performed in a short time. Alternatively, according to one aspect of the present invention, there is provided an image search method that allows for easy search. Alternatively, according to one aspect of the present invention, a novel image search method is provided. It is possible.

[0027] The description of these effects does not preclude the existence of other effects. However, it is not necessary to have all of these effects. , it is possible to extract effects other than these. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image search system. [Figure 2] FIG. 2 is a flowchart showing an example of a search data generation method. [Figure 3] 3A and 3B are diagrams showing examples of the configuration of a neural network. [Figure 4] FIG. 4 is a diagram illustrating an example of the convolution process and the pooling process. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of a neural network. [Figure 6] 6A and 6B are diagrams showing an example of a method for generating search data. [Figure 7] Fig. 7A is a diagram showing an example of a method for generating search data, and Fig. 7B is a diagram showing an example of the configuration of a neural network. [Figure 8] 8A and 8B are diagrams showing an example of a method for generating search data. [Figure 9] FIG. 9 is a flowchart showing an example of an image search method. [Figure 10] FIG. 10 is a diagram showing an example of an image search method. [Figure 11] 11A and 11B are diagrams showing an example of an image search method. [Figure 12] 12A and 12B are diagrams showing an example of an image search method. [Figure 13] FIG. 13 is a diagram showing an example of an image search method. [Figure 14] FIG. 14 is a flowchart showing an example of an image search method. [Figure 15] FIG. 15 is a diagram showing an example of an image search method. [Figure 16] 16A and 16B are diagrams showing an example of an image search method. [Figure 17] FIG. 17 is a flowchart showing an example of an image search method. [Figure 18] 18A and 18B are diagrams showing an example of an image search method. [Figure 19] FIG. 19 is a diagram showing an example of an image search method. [Figure 20] 20A, 20B1, and 20B2 are diagrams showing an example of an image search method. [Figure 21] 21A and 21B are diagrams showing an example of an image search method. [Figure 22] 22A and 22B are diagrams showing an example of an image search method. [Figure 23] FIG. 23 is a flowchart showing an example of an image search method. [Figure 24] 24A and 24B are diagrams showing an example of an image search method. [Figure 25] FIG. 25 is a diagram showing an example of an image search method. [Figure 26] FIG. 26 is a diagram showing an example of an image search method. DETAILED DESCRIPTION OF THE INVENTION

[0029] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description. The present invention is not limited to the above embodiments, and various changes and modifications may be made in the form and details thereof without departing from the spirit and scope of the present invention. It will be readily understood by those skilled in the art that the present invention can be achieved by the following embodiments. It should not be construed as being limited to the contents described.

[0030] (Embodiment 1) In this embodiment, an image search system and an image search method according to one embodiment of the present invention will be described. This will be explained using a surface.

[0031] An image search system according to one aspect of the present invention includes an input unit, a database, and a processing unit. The processing unit includes a first neural network and a second neural network. The first and second neural networks are provided with layers having neurons.

[0032] In this specification, a neural network is a network that imitates the neural circuit network of a living organism and is capable of learning. This refers to a general model that determines the strength of connections between neurons and gives them problem-solving ability.

[0033] In this specification, when discussing neural networks, Determining the connection strength (also called weight coefficient) between neurons is called "learning." .

[0034] In this specification, the neural network is constructed using the connection strengths obtained by learning. The process of constructing a theory and deriving a new conclusion from it is called "inference."

[0035] The image data is stored in the database. When a user inputs image data into the input unit, the image search system according to one aspect of the present invention Image data similar to the input image data is searched for in the database and output.

[0036] In this specification, image data stored in a database is referred to as database image data. The image data input to the input unit is called query image data. The database image data and the query image data are collectively referred to simply as image data. There is a match.

[0037] An image search method using an image search system according to one embodiment of the present invention will be described below.

[0038] The image data is input to a first neural network included in the processing unit, thereby generating an image. Image feature data can be obtained.

[0039] In this specification, data representing the feature amounts of image data is referred to as image feature amount data. For example, data representing the feature quantities of database image data is called database image feature quantity data. The data representing the feature amount of the query image data is called query image feature amount data.

[0040] The first neural network may be a convolutional network having, for example, a convolutional layer and a pooling layer. The first neural network can be a convolutional neural network. In the case of a neural network, the image data is fed to a first neural network. The data output from the pooling layer is used as image feature data. This can be done.

[0041] In addition, tags are attached to the database image data. The document data linked to the data is stored in a database, and By performing morphological analysis, tags can be linked. Keywords that represent the concept, technical content, and points of interest of the image corresponding to the base image data For example, one tag can represent one word. Database image data can be associated with multiple tags.

[0042] In this specification, a tag linked to database image data is referred to as a database A tag associated with query image data is called a query tag.

[0043] The tag is input to a second neural network in the processing unit to base the tag. For example, a tag can be represented by a 300-dimensional distributed representation vector. It is possible.

[0044] In this specification, a vector representing a tag is called a tag vector. The vector representing the database tag is called the database tag vector, and the vector representing the query tag is called the We call it a query tag vector. A tag vector is a set of tags corresponding to a single tag. Denotes a vector.

[0045] In this specification, the term "vector" refers to a set of multiple values. The number of values ​​that make up a matrix is ​​called the number of dimensions. For example, A vector can be said to be a five-dimensional vector. The values ​​that make up a vector are are sometimes referred to as components.

[0046] In the image search method using the image search system according to one aspect of the present invention, database image data In addition, database image feature data representing the feature of the database image is stored in advance. The image data is stored in the database. The database tag and the database tag vector representing the database tag are also The database tag itself is stored in the database in advance. It does not have to be stored in the base.

[0047] In the image search method using the image search system according to one aspect of the present invention, When a user inputs query image data to the input unit, the query image data is input to the first neural network. The query image feature data is then generated. The query image of the database image data is extracted using the feature data and the query image feature data. For example, cosine similarity is calculated. The calculation of similarity for each of the database image data is performed, for example, It can be done.

[0048] Then, based on the calculation result of the similarity, the query tag is obtained using the database tag. For example, database tags linked to highly similar database image data Among these, the database tag with the highest frequency of appearance can be used as the query tag. The number of query tags is, for example, the number of database images linked to one database image data. The number of tags can be the same as the number of tags.

[0049] In this specification, one piece of image data is, for example, one piece of image data displayed during one frame period. 1 shows image data representing an image.

[0050] Next, a first data set including database image feature data and database tag vectors is generated. Also, a second data including the query image feature data and the query tag vector is obtained. Then, the similarity between the first data and the second data is calculated. By doing so, the similarity of the database image data to the query image data is corrected. For example, by calculating the cosine similarity between the first data and the second data, Make corrections.

[0051] Here, one piece of first data includes, for example, one piece of database image feature data and the corresponding Data linked to database image data corresponding to database image feature data and a database tag vector corresponding to the base tag. The number of pieces of data can be the same as the number of pieces of database image feature data. In addition, one piece of second data includes query image feature data and one piece of first data. It may contain as many query tag vectors as database tag vectors.

[0052] Next, ranking data including information on the ranking of similarity after the correction is generated, and the search results are The results are output to the outside of the image search system according to one embodiment of the present invention. By correcting the similarity of the source image data to the query image data, for example, suppresses the retrieval of database images that are similar to the query image but have different concepts. This allows you to avoid unwanted images from being mixed into the search results. Therefore, the image search system according to one embodiment of the present invention can prevent the image from being input. The system can perform searches with high accuracy.

[0053] In addition, in the image search method using the image search system according to one aspect of the present invention, a database tag The query tag is acquired using the following method. For example, the acquisition method is based on the query image feature data. This is a simpler method than the method of acquiring a query tag by using the above method. The image search system can perform searches in a short time. The method of acquiring the query tag in the image search system according to one embodiment of the present invention is, for example, Compared with the method of specifying all query tags, the concept of images corresponding to the query image data, It is possible to comprehensively acquire tags that represent technical content, points of interest, etc. The image search system of the embodiment can perform searches easily and with high accuracy.

[0054] <1-1. Image search system> FIG. 1 is a block diagram showing an example of the configuration of an image search system 10. In the diagram, the components are classified by function and are presented as independent blocks in a block diagram. However, it is difficult to completely separate the components into functions, and it is difficult to create a single structure. A component may be involved in multiple functions. Also, one function may be involved in multiple components. For example, the plurality of processes performed by the processing unit 13 may be performed by different servers. This is sometimes executed.

[0055] The image retrieval system 10 includes at least a processing unit 13. The image retrieval system shown in FIG. The system 10 further includes an input unit 11, a transmission path 12, a storage unit 15, a database 17, and an output It has 19 parts.

[0056] [Input section 11] Image data and the like are supplied to the input unit 11 from outside the image search system 10. The image data etc. supplied to the image processing unit 1 are transmitted via a transmission path 12 to a processing unit 13, a storage unit 15, or a digital As described above, the image data input to the input unit 11 is sent to the database 17. This is called image data.

[0057] [Transmission path 12] The transmission path 12 has a function of transmitting image data, etc. The transmission and reception of information between the 15, the database 17, and the output unit 19 is carried out via the transmission line 12. It is possible to do so.

[0058] [Processing section 13] The processing unit 13 processes image data supplied from the input unit 11, the storage unit 15, the database 17, etc. The processing unit 13 has a function of performing calculations, inferences, etc. using the neural network. The neural network can be used to perform calculations, inferences, etc. The processing unit 13 can perform calculations without using a neural network. , calculation results, inference results, etc. are supplied to the memory unit 15, database 17, output unit 19, etc. can be done.

[0059] The processing section 13 preferably uses a transistor having a metal oxide in a channel forming region. Since the off-state current of the transistor is extremely low, the transistor can be used as a memory element. It is used as a switch to hold the charge (data) that has flowed into the capacitance element that functions as a This makes it possible to ensure a long data retention period. By using it as at least one of the register and cache memory of the processing unit 13, The processing unit 13 is operated only when necessary, and in other cases, the information of the immediately preceding processing is stored in the memory element The processing unit 13 can be turned off by evacuating it to the normally-on state. This makes it possible to achieve low power consumption in image search systems. do.

[0060] In this specification, metal oxide refers to a metal oxide in a broad sense. Metal oxides are oxides. Metal oxides are oxide insulators and oxide conductors (including transparent oxide conductors). , oxide semiconductors (also referred to as OS), etc. For example, when a metal oxide is used in the semiconductor layer of a transistor, the metal oxide In other words, a material in which a metal oxide has amplifying, rectifying, and and switching action, the metal oxide is called a metal oxide semiconductor. The term "OS" is used for short. can.

[0061] In this specification and the like, a transistor using an oxide semiconductor or a metal oxide for a channel formation region Transistors are called oxide semiconductor transistors, or OS transistors. It's called Transista.

[0062] The metal oxide contained in the channel formation region preferably contains indium (In). When the metal oxide in the panel formation region contains indium, The carrier mobility (electron mobility) of the semiconductor is increased. The oxide is preferably an oxide semiconductor containing element M. Element M is preferably aluminum. M can be aluminum (Al), gallium (Ga), or tin (Sn), etc. Other elements M can also be used. Possible elements include boron (B), silicon (Si), titanium (Ti), iron (Fe), Nickel (Ni), Germanium (Ge), Yttrium (Y), Zirconium (Zr), Molybdenum (Mo), lanthanum (La), cerium (Ce), neodymium (Nd), hafnium Hf, tantalum (Ta), tungsten (W), etc. However, the element M is In some cases, a combination of the above elements may be used. The element M may be, for example, a bond with oxygen. For example, the bond energy with oxygen is higher than that with indium. In addition, the metal oxide in the channel formation region is a metal containing zinc (Zn). An oxide is preferred, as zinc-containing metal oxides may be prone to crystallization.

[0063] The metal oxide contained in the channel formation region is not limited to a metal oxide containing indium. The semiconductor layer is indium-free, such as zinc tin oxide or gallium tin oxide, Even if it is a metal oxide containing zinc, a metal oxide containing gallium, a metal oxide containing tin, etc. It's okay.

[0064] The processing unit 13 is, for example, an arithmetic circuit or a central processing unit (CPU). It also has a Recessing Unit.

[0065] The processing unit 13 includes a DSP (Digital Signal Processor), a GPU, (Graphics Processing Unit) or other microprocessors The microprocessor may be a field programmable gate array (FPGA). e Gate Array), FPAA(Field Programmable An PLDs (Programmable Logic Devices) such as Analog Arrays The processing unit 13 may be configured to perform various operations by a processor. By interpreting and executing commands from a program, various data processing and program control are performed. The program that can be executed by the processor is stored in the memory of the processor. The data is stored in at least one of the storage area and the storage unit 15.

[0066] The processing unit 13 may have a main memory. The main memory may be a RAM (Random Access Memory). Volatile memory such as ROM (Read Only Access Memory) The memory includes at least one of a non-volatile memory such as a memory for storing data and a non-volatile memory.

[0067] Examples of RAM include DRAM (Dynamic Random Access Memory) mory), SRAM (Static Random Access Memory), etc. is used, and a virtual memory space is allocated and used as a working space for the processing unit 13. The operating system, application programs, and programs stored in the storage unit 15 Program modules, program data, and lookup tables are used for execution. These data, programs, and programs loaded into RAM are Each RAM module is directly accessed and operated by the processing unit 13.

[0068] The ROM contains a BIOS (Basic Input / Output) It can store the ROM (System) and firmware. SCROM, OTPROM (One Time Programmable Read Only Memory), EPROM (Erasable Programmable EPROM is a type of memory that can be read and written by ultraviolet light. UV-EPROM (Ultra-Violet) Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmability) e Read Only Memory), flash memory, etc.

[0069] [Storage section 15] The storage unit 15 has a function of storing a program executed by the processing unit 13. 15 is a calculation result and an inference result generated by the processing unit 13, and a calculation result and an inference result input to the input unit 11. It may also have a function to store image data and the like.

[0070] The storage unit 15 includes at least one of a volatile memory and a non-volatile memory. The storage unit 15 may include a volatile memory such as a DRAM or an SRAM. For example, ReRAM (Resistive Random Access Memory) ry, also known as resistive random access memory), PRAM (Phase change Random Access Memory) m Access Memory), FeRAM (Ferroelectric Ran dom Access Memory), MRAM (Magnetoresistive Random Access Memory (also known as magnetoresistive memory), or flash memory The storage unit 15 may have a nonvolatile memory such as a hard disk drive. Hard Disc Drive (HDD) and Solid State Drive It has a recording media drive such as a Solid State Drive (SSD). It's fine.

[0071] [Database 17] The database 17 has a function of storing image data to be searched. Image data stored in the database is called database image data. The database 17 has a function of storing the calculation results and inference results generated by the processing unit 13. Furthermore, the input unit 11 may have a function of storing image data and the like input thereto. The storage unit 15 and the database 17 do not have to be separated from each other. The image retrieval system 10 is a storage unit having the functions of both the storage unit 15 and the database 17. It may have a slot.

[0072] [Output section 19] The output unit 19 has a function of supplying information to the outside of the image search system 10. For example, The calculation results or inference results in the logic unit 13 can be supplied to the outside.

[0073] <1-2. Image search method-1> First, a process to be performed in advance to perform a search using the image search system 10 will be described. FIG. 2 is a flow chart showing an example of this processing method.

[0074] [Step S01] First, the database image data GD DB But The database image data GD DB The intellectual property information has Here, the intellectual property information may be, for example, patent documents. (Unexamined Patent Gazettes, Patent Gazettes, etc.), Utility Model Gazettes, Design Gazettes, and publications such as papers. Not only publications published in Japan, but also publications published around the world are subject to intellectual property protection. It can be used as information.

[0075] Intellectual property information is not limited to publications. For example, the user or user group of an image search system Various files such as image files that the body has uniquely are also stored in the database image data GD DB and Furthermore, intellectual property information may include information describing an invention, device, or design. Examples include drawings that clarify the invention.

[0076] Also, the database image data GD DB are described, for example, in patent documents of certain applicants. data representing drawings in patent documents in a particular technical field The data may include:

[0077] The image retrieval system 10 retrieves database image data GD similar to the query image data. DB Therefore, by using the image search system 10, for example, It is possible to search for patent documents, papers, or industrial products that are similar to the invention before filing. This allows you to conduct a prior art search for your invention before filing an application. By doing so, you can strengthen your invention and make it into a strong patent that is difficult for other companies to circumvent. do.

[0078] In addition, by using the image search system 10, it is possible to search for features similar to, for example, industrial products that have not yet been released. You can search patent documents, papers, or industrial products. Database Image Data GD DB If the company has data corresponding to the images in its patent documents, it can It is possible to check whether the technology related to the product has been sufficiently patented within the company. Database image data GD DBThe data corresponding to the images described in patent documents of other companies If you have a patent, you can check whether your industrial products before release are infringing on the intellectual property rights of other companies. By understanding and reviewing related prior art, you can discover new inventions and improve your business. It is possible to make an invention that can be a strong patent that contributes to the future. You can also search for industrial products after they have been released.

[0079] In addition, for example, the image search system 10 can be used to search for patent documents, papers, and other documents similar to a specific patent. You can search for industrial products, especially by searching based on the filing date of the patent. It is possible to easily and accurately investigate whether the patent in question contains grounds for invalidation.

[0080] [Step S02] Next, the database image data GD DB The neural network included in the processing unit 13 Enter.

[0081] FIG. 3A shows a neural network 3 which is a neural network included in the processing unit 13. 3 is a diagram showing an example of the configuration of layer 31[1] to layer 31[0]. m] (m is an integer of 1 or greater).

[0082] Layers 31[1] to 31[m] have neurons, and the neurons provided in each layer For example, the neurons in layer 31[1] are connected to the neurons in layer 31[ 2]. Also, the neurons in layer 31[2] The neurons are those in layer 31[1] and those in layer 31[3]. In other words, layers 31[1] to 31[m] are connected to the neurons that are connected to the A hierarchical neural network is constructed.

[0083] Database image data GD DB is input to layer 31[1], and layer 31[1] is input The data corresponding to the image data is input to layer 31[2], and the layer 31 [2] outputs data corresponding to the input data. Layer 31[m] has layer 31[m- 1] is input, and layer 31[m] is the layer corresponding to the input data. The data is output. From the above, layer 31[1] is the input layer, and layers 31[2] to 31[m-1 ] can be the hidden layer and layer 31[m] can be the output layer.

[0084] The neural network 30 is configured to generate a set of data output from layers 31[1] to 31[m]. The data is arranged to represent the feature quantity of the image data input to the neural network 30. It is pre-trained. Learning can be done by unsupervised learning, supervised learning, etc. In particular, unsupervised learning is preferable because it does not require training data (also known as correct labels). In addition, whether learning is performed using unsupervised or supervised learning methods, The algorithm may be an error backpropagation method or the like.

[0085] Here, the database image data GD DB It is preferable to use This allows the data output from layers 31[1] to 31[m] to be processed by neural networks. It is possible to accurately represent the feature quantities of the image data input to the network 30. For example, the database image data GD stored in the database 17 DB All of Using the training data, the neural network 30 can be trained. For example, Database image data GD DB A part of the data is used as training data to create a neural network. 30 can perform learning. For example, the database image data GD DB In addition, memory Image data stored in the image search system 10 and image data stored in the image search system 10 are input from the input unit 11. The image data input to the processing unit 13 via the Q30 can learn.

[0086] As training data, the database image data GD DB For example, Image data input from outside the image search system 10 to the processing unit 13 via the input unit 11 The neural network 30 can perform learning using only the data as training data.

[0087] Neural network 30 is a convolutional neural network (CNN). Fig. 3B shows the neural network. Neural network 30 when CNN is applied as neural network 30 FIG. 1 is a diagram showing an example of the configuration of a neural network 30 to which CNN is applied. Let us call this neural network 30a.

[0088] The neural network 30a includes a convolutional layer CL, a pooling layer PL, and a fully connected layer F In FIG. 3B, a neural network 30a has a convolutional layer CL and a pulley Each layer has m layers (m is an integer greater than or equal to 1) of coupling layers PL and one fully connected layer FCL. The neural network 30a has two or more fully connected layers FCL. You may do so.

[0089] The convolution layer CL has the function of performing convolution on the data input to the convolution layer CL. For example, the convolution layer CL[1] performs the following on the image data input to the processing unit 13: The convolution layer CL[2] has the function of performing convolution using the pooling layer PL[1 ] has the function of performing convolution on the data output from the convolution layer CL[ m] is a function that performs convolution on the data output from the pooling layer PL[m-1] It has.

[0090] Convolution is performed by repeating the multiplication and accumulation of the data input to the convolution layer CL and the weight filter. The convolution in the convolution layer CL is used to generate the neural network. The features of the image corresponding to the image data input to the work 30a are extracted.

[0091] The convolved data is transformed by the activation function and then output to the pooling layer PL. The activation function is ReLU (Rectified Linear Unit ts) can be used. ReLU outputs "0" if the input value is negative, If the input value is "0" or greater, the function outputs the input value as is. As the number, a sigmoid function, a tanh function, etc. can also be used.

[0092] The pooling layer PL performs pooling on the data input from the convolution layer CL. Pooling divides data into multiple regions and allocates a predetermined amount of data to each region. The pooling process extracts the data and arranges them in a matrix. This allows the amount of data to be reduced while retaining the extracted features. This can improve robustness against small deviations in the Large pooling, average pooling, Lp pooling, etc. can be used.

[0093] The fully connected layer FCL uses the data output from the pooling layer PL[m] to determine the image. The fully connected layer FCL has the function of connecting all the nodes in a layer to all the nodes in the next layer. The data output from the convolution layer CL or the pooling layer PL is The data is a two-dimensional feature map, which is expanded to one dimension when input to the fully connected layer FCL. Then, the vector obtained by inference by the fully connected layer FCL is output from the fully connected layer FCL. will be done.

[0094] The configuration of the neural network 30a is not limited to the configuration shown in FIG. A neural network layer PL may be provided for each of a plurality of convolution layers CL. The number of pooling layers PL included in the network 30a is less than the number of convolutional layers CL. Also, if you want to retain as much position information of the extracted features as possible, you can use the pooling layer PL. It does not have to be provided.

[0095] The neural network 30a learns the filter values ​​of the weighting filters, The weighting coefficients of the coupling layer FCL can be optimized.

[0096] Next, the convolution process is performed in the convolution layer CL and the pooling process is performed in the pooling layer PL. An example of the pooling process will be described with reference to FIG. 4. The data input to layer CL is the input data value of 3 rows and 3 columns (input data value i11, input data input data value i12, input data value i13, input data value i21, input data value i22, input data value i31, input data value i32, input data value i33 input data value i23, input data value i31, input data value i32, input data value i33 In addition, the weight filter is a 2-row, 2-column filter value (filter value f11, filter value f12, filter value f21, filter value f22).

[0097] Here, for example, the data input to the convolution layer CL[1] can be image data. In this case, the input data value can be a pixel value included in the image data. .

[0098] In this specification, the pixel value indicates a value that represents the gradation of the brightness of the light emitted by a pixel. For example, if a pixel value is an 8-bit value, the pixel can emit light with 256 levels of brightness. Image data can be said to include a set of pixel values, for example, a set of pixel values ​​equal to the number of pixels. For example, if the number of pixels in an image is 2x2, the image data representing the image may include The data can be said to contain 2x2 pixel values.

[0099] In addition, for example, the input data value input to the convolution layer CL[2] is input to the pooling layer PC The input data value can be the output value of [1] and input to the convolution layer CL[m]. can be the output value of the pooling layer PC[m-1].

[0100] The convolution is performed by multiplying and adding the input data value and the filter value. In this case, the input data can be data showing a predetermined feature (called feature data). By comparing the data value with the filter value, the image input to the neural network 30a is Feature extraction can be performed on the image data.

[0101] In Figure 4, the convolutional layer CL receives input data value i11, input data value i12, and input data value By filtering the input data value i21 and the input data value i22, the convolution layer C The figure shows how the convolution value C11 of the data output from L is obtained. The convolution layer CL receives input data value i12, input data value i13, input data value i22, and By filtering the input data value i23 and the input data value i24, the output from the convolution layer CL is The figure shows how the convolutional value C12 of the data is obtained. CL is the input data value i21, the input data value i22, the input data value i31, and the input data By filtering the data value i32, the data output from the convolution layer CL is The figure shows how the convolution layer CL obtains the convolution value C21 held by the data. Input data value i22, input data value i23, input data value i32, and input data value i3 By filtering 3, the data output from the convolution layer CL has The convolution process shown in Figure 4 is performed as follows: We can say that the stride is 1.

[0102] The convolution value C11, the convolution value C12, the convolution value C21, and the convolution value C22 are These can be obtained by the product-sum calculation shown in the following equations.

[0103] (Number 1) C11=i11·f11+i12·f12+i21·f21+i22·f22 (1)

[0104] (Number 2) C12=i12·f11+i13·f12+i22·f21+i23·f22 (2)

[0105] (Number 3) C21=i21·f11+i22·f12+i31·f21+i32·f22 (3)

[0106] (Number 4) C22=i22·f11+i23·f12+i32·f21+i33·f22 (4)

[0107] The convolutional values ​​C11, C12, C21, and The convolution value C22 is arranged in a matrix according to the address, and then passed through the pooling layer P Specifically, the convolution value C11 is placed in the first row and first column, and the convolution value C 12 is placed in the first row, second column, and the convolution value C21 is placed in the second row, first column, and the convolution value C 22 is placed in the second row and second column.

[0108] In Figure 4, the pooling layer PL has convolutional values ​​C11, C12, and C21. , and the convolution value C22 are input, and one value is pooled based on the four convolution values. For example, the convolution value C11, the convolution value C12, the convolution value P, The maximum value of the value C21 and the convolution value C22 can be set as the pooling value P. Or, the convolution value C11, the convolution value C12, the convolution value C21, and the convolution value C22 The pooling value P can be calculated by dividing the pooling layer PL by the average value of the pooling value P. This is the output value that is output from the

[0109] Figure 4 shows an example in which data input to the convolution layer CL is processed by one weight filter. Although the figure shows the weighting of the neural network, it may be processed using two or more weighting filters. It is possible to extract multiple features contained in the image data input to the network 30a. When the data input to the convolution layer CL is processed by two or more weight filters, The process shown in Figure 4 is performed for each filter. As mentioned above, the stride is set to 1 in Figure 4. However, the stride may be two or more.

[0110] FIG. 5 shows the convolution layer CL and the pooling layer PL 5 is a diagram illustrating an example of the configuration of the convolution layer CL and the pooling layer PL shown in FIG. An example of how the operation works is shown.

[0111] FIG. 5 shows a neuron 32. Specifically, the neuron 32 is In FIG. 5, neurons 32a, 32b, and 32c are shown. The value output from the neuron 32 is written inside the neuron 32. The value is output in the direction of the arrow. If the value is multiplied by a weighting factor, the weighting factor is displayed near the arrow. In FIG. 5, the filter values ​​f11, f12, and f21 are listed. , and the filter value f22 is used as a weighting factor.

[0112] Neuron 32a is a neuron included in layer L, which is the layer before convolution layer CL shown in FIG. 32. For example, when the convolution layer CL shown in FIG. 5 is the convolution layer CL[1], If it is a convolutional layer CL[2], it can be the input layer, and if it is a pooling layer PL[1 ], and if it is a convolutional layer CL[m], it is a pooling layer PL[m-1]. It is possible.

[0113] In FIG. 5, neurons 32a[1] to 32a[9] are shown as neurons 32a. In the case shown in FIG. 5, neuron 32a[1] receives input data value i1 neuron 32a[1] outputs the input data value i12, and neuron 32a[2] outputs the input data value i12. [3] outputs the input data value i13, and neuron 32a[4] outputs the input data value i21. neuron 32a[5] outputs input data value i22, and neuron 32a[6] outputs input data value i23. ] outputs input data value i23, and neuron 32a[7] outputs input data value i31. Then, neuron 32a[8] outputs input data value i32, and neuron 32a[9] outputs Outputs input data value i33.

[0114] The neuron 32b is the neuron 32 included in the convolution layer CL shown in FIG. indicates neurons 32b[1] to 32b[4] as neurons 32b. is doing.

[0115] In the case shown in FIG. 5, neuron 32b[1] has a filter for input data value i11. the value obtained by multiplying the input data value i12 by the filter value f11, and the value obtained by multiplying the input data value i12 by the filter value f12. The input data value i21 multiplied by the filter value f21 and the input data value i22 multiplied by the filter value f22 are The sum of these values, the convolution value C11, is input. It is output from Ron 32b[1].

[0116] Also, neuron 32b[2] receives the input data value i12 multiplied by the filter value f11. The input data value i13 multiplied by the filter value f12 and the input data value i22 multiplied by the filter value f13 are The input data value i23 is multiplied by the filter value f22. Then, the sum of these values, the convolution value C12, is output from neuron 32b[2]. It is output from

[0117] Also, neuron 32b[3] receives the input data value i21 multiplied by the filter value f11. The input data value i22 multiplied by the filter value f12 and the input data value i31 multiplied by the filter value f12 are The input data value i32 is multiplied by the filter value f21, and the input data value i32 is multiplied by the filter value f22. Then, the sum of these values, the convolution value C21, is output from neuron 32b[3]. It is output from

[0118] Furthermore, neuron 32b[4] is set to the input data value i22 multiplied by the filter value f11. The input data value i23 multiplied by the filter value f12 and the input data value i32 multiplied by the filter value f13 are The input data value i33 is multiplied by the filter value f21, and the input data value i33 is multiplied by the filter value f22. The sum of these values, the convolution value C22, is then input to neuron 32b[4]. is output from

[0119] As shown in FIG. 5, each of neurons 32b[1] to 32b[4] has the following functions: It is connected to some of the neurons 32a[1] to 32a[9]. The convolutional layer CL can be said to be a partially connected layer.

[0120] The neuron 32c is the neuron 32 included in the pooling layer PL shown in FIG. In the case shown in FIG. 1, the neuron 32c has a convolution value C11, a convolution value C12, and a convolution value C13. The pooling value P is input as the new The convolution value output from the neuron 32b includes the overlapping As mentioned above, the weighting coefficient is determined by the neural network learning. These are the parameters to be optimized. Therefore, the parameters used by the pooling layer PL during calculations are The parameter optimization algorithm may be configured so that there are no parameters that are optimized by learning.

[0121] From the above, the database image data GD DB By inputting this into the neural network 30, By this, the database image data GD DB Image feature data representing the feature values ​​of TaGFD DB For example, as shown in FIG. 3A, The data output from 31[m] is used as the database image feature data GFD DB Toshiko Alternatively, as shown in FIG. 3B, the data output from the pooling layer PL[m] can be The database image feature data GFD DB It is possible to do so. Image feature data GFD DB The database may contain more than one layer of output data. Image feature data GFD DB contains the output data of many layers, Image feature data GFD DB , the database image data GD DB More accurately represent the characteristics of The database image feature amount data GFD acquired by the processing unit 13 can be DB can be stored in the database 17.

[0122] [Step S03] Next, the database image data GD DB Database tag associated with TAG DB of, The processing unit 13 acquires the database image data GD DB Image concept, technical Tags that indicate content, points of interest, etc. are database tags TAG DB So, the database type GuTAG DB It is preferable to obtain the database tag TAG DB Get 6A is a diagram showing an example of a method for determining whether or not the data is to be used. In addition, the illustrations of the data, vectors, etc. shown in other figures are also examples. The content is not limited to the above.

[0123] In this specification and the like, when the same reference numeral is used for a plurality of elements, it is not necessary to particularly distinguish between them. In some cases, the symbol should be followed by an identifying symbol such as [1] or [2].

[0124] In the method shown in FIG. 6A, as an example, the database image data GD DB [1] Data Base image data GD DB Let's say that a tag is attached to each of

[0100] . Database image data GD DB Document data TD corresponding to DB However, in database 17 It is assumed that the database image data GD DB The drawing number The numbers are assumed to be linked.

[0125] Document Data TD DB For example, the database image data GD DB The drawing shown was published Data corresponding to documents described in publications such as patent documents, utility model publications, design publications, and papers. For example, the database image data GD DB The drawings shown are If the publication is a patent document or utility model publication, the data corresponding to the specification shall be Data TD DB Or, the scope of claims, the scope of claims for utility model registration, Or the data corresponding to the abstract is document data TD DB It is also possible to Database image data GD DB If the publication in which the design is published is the Design Gazette, The data is document data TD DB It can be said that:

[0126] For example, document data TD DB When the data corresponds to a specification or a paper, the data Base tag DB is the database image data GD DB In the paragraph describing the drawing represented by In Figure 6A, the database image is Image data GD DB The figure number of the image corresponding to [1] is "Figure 1" and the database image data is Data GD DB Document data TD linked to [1] DB [1] represents the paragraph [0xx 0] shows an example where "Figure 1 is," so for example, paragraph [0xx0] , database image data GD DB The description of the drawing represented by [1] is deemed to be included. By performing morphological analysis on the sentence in paragraph [0xx0], Database tag DB [1] can be obtained. Also, in Figure 6A, image data GD DB The figure number of the image corresponding to

[0100] is "Figure 15", and the database image data Ta GD DB Document data TD linked to

[0100] DB

[0100] represents the paragraph [ 0xx7] shows an example where "Figure 15 is" is written. Therefore, for example, paragraph [0 xx7], database image data GD DB The description of the drawing indicated by

[0100] is provided. Conduct a morphological analysis of the sentence in paragraph [0xx7], assuming that By database tag TAG DB

[0100] can be obtained.

[0127] In addition, all document data TD DB For example, the database image Image data GD DB [1] corresponds to "Figure 1" of the specified publication, and the database image data G D DB If [2] corresponds to "Figure 2" in the same publication, then Document Data TD DB [1] is shown Document and Document Data TD DB The documents represented by [2] can be the same.

[0128] Morphological analysis analyzes text written in natural language into morphemes (the smallest units that have meaning in language). This allows us to distinguish between parts of speech, etc., of morphemes. ] and extract only the nouns listed in the database tag TAG DB [1] In the case shown in Figure 6A, "circuit diagram", "aaa", "bbb", "ccc", Words such as "ddd" are included in the database tag TAG DB [1] He also states that Words such as "rock figure," "ggg," "aaa," "ccc," and "hhh" are included in the database. Stag TAG DB It is assumed to be

[0100] .

[0129] As mentioned above, the database tag TAG DB For example, the database image data GD DB Document data TD linked to DB It can be obtained by performing morphological analysis on This method can be used to create a database tag DB By acquiring the database image Image data GD DB The tags that represent the concept, technical content, and points of interest of the corresponding image are comprehensively collected. You can gain.

[0130] In this specification, one tag means, for example, one word. For example, In this case, the database tag TAG DB The number of [1] can be five or more. Also, the database tag TAG DB The number of

[0100] can be five or more.

[0131] For example, all words extracted by morphological analysis are stored in the database tag TAG D B For example, it is not necessary to extract a predetermined number of words from the extracted words and The extracted words are tagged in the database. DB For example, the extracted unit From the words, TF-IDF (Term Frequency-Inverse Document A predetermined number of words with high ment frequency are extracted and the extracted words are Database tag TAG DB The database image data GDDB Nihimo Database tag that can be attached DB The number of database image data GD D B By making the image search method using the image search system 10 equal to each of the The law can be simplified.

[0132] TF-IDF is based on two metrics: term frequency (TF) and inverse document frequency (IDF). Therefore, words that appear frequently throughout the document will have a high TF but a low IDF. Therefore, the database tag TAG DB In paragraphs where candidate words are extracted, A word that appears frequently in a sentence will have a lower TF-IDF than a word that appears less frequently in other paragraphs. For example, frequently occurring words in a document can be used to identify image features such as concepts, technical content, and points of interest. Therefore, the database tag T AG DB Once you have obtained the database tag TAG using only TF, for example DB Get In some cases, database tags that strongly represent the characteristics of database images are used. DB To obtain Therefore, the image retrieval system 10 can perform a highly accurate search. In addition, without calculating TF-IDF, for example, the database tag TAG DB of In this case, the calculation performed by the processing unit 13 can be simplified.

[0133] Note that morphological analysis may be performed on sentences of two or more paragraphs. For example, Data GD DB In addition to the paragraph in which the description of the drawing represented by Morphological analysis may also be performed on the subsequent paragraphs. For example, in the document data TD shown in FIG. D B When morphological analysis is performed on [1], in addition to paragraph [0xx0], the next paragraph, You can also perform morphological analysis on paragraph [0xx1]. The word "eee" is listed in the database tag TAG DB [1] can be For example, the document data TD shown in FIG. DB Perform morphological analysis on

[0100] In this case, morphological analysis is performed on paragraph [0xx7] as well as the previous paragraph [0xx6]. In this case, for example, the word "fff" in paragraph [0xx6] Database tag TAG DB It can be set to

[0100] .

[0134] Or database image data GD DB All paragraphs with figure numbers associated with Alternatively, a morphological analysis may be performed on the database image data GD DB Linked to Morphological analysis is performed on paragraphs that contain the figure number and no other figure numbers. It is also possible.

[0135] Or, morphological analysis is performed only on some sentences contained in a given paragraph. For example, in the case shown in FIG. 6A, the sentence in paragraph [0xx0] is Morphological analysis may be performed only on sentences containing the word "dd" in this case. 'd' is the database tag TAG DB [1] is not the case.

[0136] In addition, document data TD DBNot only the words themselves that appear in the document represented by the Synonyms of words, database tag TAG DB For example, the storage unit 15 or Thesaurus data is stored in advance in the database17, and extracted through morphological analysis. The data is then compiled by combining the input word and the words registered in the thesaurus as synonyms of the input word. Base tag DB The synonyms used here are generally available Alternatively, synonyms extracted using embedded representations of words may be used. In addition, synonym extraction using distributed representations is based on the search results of other documents in the same field as the document to be searched. This may be done using a database containing

[0137] Document Data TD DB Not only the words themselves that appear in the document represented by the Synonym database tag TAG DB By doing so, the database tag TAG DB of, Database image data GD DB It strongly expresses the characteristics of the concept, technical content, and points of interest. It can be said that.

[0138] In addition, the database tag TAG DB For example, Database image feature data GFD DB Based on the database tag TAG DB Get Good too.

[0139] FIG. 6B shows the database image data GD DB 10 is a diagram showing an example of a method for linking a figure number to a In FIG. 6B, the publication data PD is combined with the image data GD. DB [1] and image data GD D B [2] and document data TD DB is included. Also, it is assumed that the publication data PD the publication represented by includes the text "Figure 1 xxx" and the text "Figure 2 yyy" described. Note that the data representing the text "Figure 1 xxx" and the text "Figure 2 yyy " are not included in the document data TD DB The "x 1", "x2", "x1 < x2", and the dashed lines, arrows, etc. shown in Fig. 6B are added for convenience of explanation and are not actually described in the publication represented by the publication data PD.

[0140] In the method shown in Fig. 6B, for example, if the text "Figure N" is described within a predetermined distance from the drawing the drawing number of the drawing provided at the closest distance to the text "Figure N" can be set as "N". Here, for example, the distance between the coordinates representing the center of the text (center coordinates) and the center coordinates of the drawing can be defined as the distance from the text to the drawing. Note that "N" is not limited to an integer and may include letters, for example. For example, N may be "1(A)". .

[0141] In the case shown in Fig. 6B, the distance x1 between the center coordinates of the text "Figure 1 xxx" and the center coordinates of the drawing corresponding to the database image data GD DB [1] is shorter than the distance x2 between the center coordinates of the text "Figure 1 x xx" and the center coordinates of the drawing corresponding to the database image data DB [2]. Therefore, it can be said that the drawing provided at the closest distance to the text "Figure 1 xxx" is the database image data GD [1]. Thus DB . Database image data GD DB The figure number associated with [1] should be "1". can.

[0142] In addition, in FIG. 6B, document data TD DB The paragraph [0zz3] of the document represented by In paragraph [0zz4], it is stated that "Figure 2 is" In the case shown in FIG. 6B, the database image data GD DB Data linked to [1] Base tag DB [1] is, for example, the sentence in paragraph [0zz3] This can be obtained by performing morphological analysis. "Block diagram," "iii," "kkk," "hhh," "ppp," etc. The word is a database tag TAG DB [1]

[0143] For example, the center coordinates of all drawings are arranged as a first one-dimensional array, and all texts are arranged as a first one-dimensional array. The center coordinates of the first one-dimensional array may be arranged to form a second one-dimensional array. For each drawing, compare the coordinates in the column with the coordinates in the second one-dimensional array. , the text "Figure N" written in the nearest coordinate may be linked. The figure number of the drawing located at the coordinates closest to the coordinates representing the location of "Figure N" can be set to "N". The ratio of the coordinates contained in the first one-dimensional array to the coordinates contained in the second one-dimensional array is The comparison is performed by, for example, calculating the sum of the square of the difference in the x coordinate and the square of the difference in the y coordinate. When the comparison is performed by this method, for example, the element with the smallest sum value is It can be the element located at the nearest coordinates.

[0144] By the above method, the processing unit 13 obtains the database tag TAG DB can be obtained. The database tag TAG acquired by the processing unit 13 DB be stored in the database 17 can be done.

[0145] [Step S04] Next, the database tag TAG DB is represented by a vector. DB The vector representing the database tag vector TAGV DB Figure 7A is a diagram of the Shows the database tag TAG DB is expressed by a vector. .

[0146] Database tag vector TAGV DB is the database tag TAG DB For example, This can be obtained by inputting the data into the neural network of the unit 13. In this case, the database tag vector TAGV DB can be, for example, a distributed representation vector A distributed representation vector is a quantified association of words for each feature element (dimension). It is a vector expressed as a sequence of values. Words with similar meanings have similar vectors.

[0147] The neural network used to obtain the distributed representation vector is the image feature vector mentioned above. The neural network used to acquire the data can be configured differently. Figure 7B shows the neural network used to obtain the distributed representation vectors. 1 is a diagram showing an example of the configuration of a neural network 40. FIG.

[0148] In this specification, for example, a neural network used to acquire image feature data The network is called the first neural network and is used to obtain the distributed representation vector. The neural network that uses this method is sometimes called the second neural network. The numbers are just examples. For example, the neural network used to obtain the distributed representation vector The network is called the first neural network, and is used to obtain image feature data. The neural network used may be called the second neural network. For example, the neural network used to acquire image feature data is called the third neural network. It can be called a neural network, or for example, a distributed representation vector. The neural network may also be called a third neural network, etc.

[0149] As shown in FIG. 7B, the neural network 40 includes an input layer IL, a hidden layer ML, and an output layer ML. Here, the neural network 40 has one hidden layer ML. The neural network 40 can be configured to perform the simple input to the input layer IL. The distributed representation vectors representing words are generated using, for example, the open-source algorithm W The neural network configuration shown in Figure 7B is used below. The input layer IL of the network 40 receives the database tag TAG DB Representing the date Database tag vector TAGV DB An example of a method for obtaining the above information will be described below.

[0150] The input layer IL contains the database tag TAG DB is expressed as a one-hot vector. Here, one-hot vectors are input, each of which represents one word. The component corresponding to the word to be input to L can be set to 1, and the other components can be set to 0. In other words, on An e-hot vector is a vector in which one component is 1 and all other components are 0. The number of neurons in the input layer IL is determined by the number of neurons that compose the one-hot vector. The number of components may be the same as the number of components that make up the compound.

[0151] The hidden layer ML generates a distributed representation vector based on the one-hot vector input to the input layer IL. For example, a one-hot vector is multiplied by a predetermined weight. By doing so, the hidden layer ML can generate a distributed representation vector. Since it can be expressed by a column, a multiply-and-add operation can be performed between the one-hot vector and the weight matrix. By performing the calculation, the neural network 40 can generate a distributed representation vector. can.

[0152] The number of neurons in the hidden layer ML is the same as the number of dimensions of the distributed representation vector. For example, if the dimension of the distributed representation vector is 300, the hidden layer ML is It can be configured with 300 neurons.

[0153] The weight matrix can be obtained by learning, for example, supervised learning. ,A word is represented as a one-hot vector and input to the input layer IL. The surrounding words of the input word are represented as one-hot vectors and input to the output layer OL. Here, for each word input to the input layer IL, multiple surrounding words are input to the output layer OL. Then, the output layer OL can output the probability that the word input to the input layer IL is a neighboring word. The values ​​of the weight matrix of the neural network 40 are adjusted so that Each neuron in the input layer OL corresponds to one word. This is an example of how to learn 0.

[0154] As described above, in both the input layer IL and the output layer OL, one neuron corresponds to one word. Therefore, the number of neurons in the input layer IL and the number of neurons in the output layer OL can be The number of neurons can be the same as the number of neurons.

[0155] The number of neurons in the hidden layer ML is greater than the number of neurons in the input layer IL. For example, the processing can be performed by a neural network 40. The number of words that can be used, that is, the number of neurons in the input layer IL, is set to 10,000. The number of dimensions of the representation vector, that is, the number of neurons in the hidden layer ML, is set to 300. Therefore, with distributed representation, the number of dimensions remains small even if the number of expressible words increases. Therefore, the amount of calculation does not increase even if the number of words that can be expressed increases. The image retrieval system 10 can perform searches in a short time.

[0156] By the above method, the processing unit 13 generates the database tag vector TAGV DB to obtain The database tag vector TAGV acquired by the processing unit 13 can be DB is a database The data can be stored in the storage device 17.

[0157] As described above, in steps S01 to S04, the processing unit 13 calculates the database image feature quantity Data GFD DB , database tag TAG DB , and the database tag vector TAG V DB are acquired and stored in the database 17. As a result, the image retrieval system 10 It is possible to search for database images similar to the query image. Database tag TAG DB does not need to be stored in the database 17.

[0158] In the method shown in FIG. 2, the processing unit 13 creates a database in steps S01 and S02. Image feature data GFD DB After obtaining the above, steps S03 and S04 are performed. The processing unit 13 generates the database tag TAG DB , and the database tag vector TAGV DB However, one aspect of the present invention is not limited to this. Stag TAG DB , and the database tag vector TAGV DB After obtaining the data, Source image feature data GFD DB may be obtained.

[0159] In addition, in the method shown in Figure 2, the database tag TAG DB Neural Network 40 The vector itself output from the neural network 40 is input to Database tag vector TAGV DB However, one embodiment of the present invention is not limited to this. In the following, the database tag vector TAGV DB A modified example of the method for obtaining the above will be described.

[0160] First, the processing unit 13 calculates the database tag TAG DB Get candidate words for data Base tag DB The candidate words are, for example, in the form shown in Figs. 6A and 6B. It can be obtained by elementary analysis.

[0161] Next, the acquired words are represented by vectors. For example, the acquired words are input to a neural network. By inputting the data into the network 40, it can be represented as a distributed representation vector.

[0162] Then, clustering is performed on the distributed representation vectors to obtain a predetermined number of clusters. For example, generate the database tag you want to get. DB The same number of classes as there are Clustering is performed using the K-means method, DBSCAN (Density- Based on Spatial Clustering of Applications This can be done by the (with noise) method or the like.

[0163] In FIG. 8A, the database image data GD DB [1] Possible tags to associate with this are: The 20 words are acquired by the processing unit 13, and these words are each added to the database word base. Kutol WORDV DB In addition, in Fig. 8A, 20 databases are used. Source word vector WORDV DB Based on this, five clusters (cluster CST1, cluster Cluster CST2, Cluster CST3, Cluster CST4, and Cluster CST5) For convenience of explanation, the vectors shown in FIG. 8A are two-dimensional vectors. The horizontal axis is one component of the two-dimensional vector, and the vertical axis is the other component of the two-dimensional vector. , but in reality it is the database word vector WORDV DB For example, 3 It can be a 0-dimensional vector.

[0164] Next, for each of the clusters CST1 to CST5, a vector representing a representative point is calculated. Then, the vector representing the representative point is called the database tag vector TAGV DB In Figure 8A, the vector representing the representative point of cluster CST1 is The database tag vector TAGV1 DB [1] and represent the representative points of cluster CST2. The vector is stored in the database tag vector TAGV2 DB [1] and cluster CST3 The vector representing the representative point is the database tag vector TAGV3 DB [1] and the cluster The vector representing the representative point of CST4 is the database tag vector TAGV4 DB [1] The vector representing the representative point of cluster CST5 is the database tag vector TAGV5. DB [1] shows an example.

[0165] Each component of the vector representing the representative point is, for example, a database word vector included in the cluster. Kutol WORDV DB For example, in a cluster, (0.1,0.7), (0.2,0.5), (0.3,0.5), (0.4,0.2), Five database word vectors WORDV (0.5,0.1) DB Contains In this case, the vector representing the representative point of the cluster should be (0.3, 0.4), for example. By the above procedure, the processing unit 13 can obtain the database tag vector TAGV DB [1 ] can be obtained.

[0166] Database tag vector TAGV DB [2] After that, you can obtain it in the same way. FIG. 8B shows the database image data GD DB [1] or database image data GD DB For each of

[0100] , the database tag vector TAGV DB 5 pieces Each (database tag vector TAGV1 DB , database tag vector TAGV2 DB , database tag vector TAGV3 DB , database tag vector TAGV4 DB , and the database tag vector TAGV5 DB ) when obtaining the result of each vector 8B is a table showing components. Note that the components shown in FIG. 8B are an example for convenience of explanation.

[0167] As shown in Figure 8B, the database tag vector TAGV DB The weighting The weights can be calculated by, for example, RDV DB The number of database image data GD DB As a candidate for tags to be linked to It can be a value obtained by dividing by the total number of words acquired by the processing unit 13. For example, in FIG. In FIG. 8B, the database image data GD DB As a candidate for tags to be linked to [1] In this example, the processing unit 13 acquires 20 words. Database word vectors WORDV DB Cluster CST2 contains four devices. Database word vector WORDV DB Cluster CST3 contains two databases. Source word vector WORDV DB Cluster CST4 contains three database units. Word Vector WORDV DB Cluster CST5 contains three database word vectors. Tor WORDV DB Therefore, as shown in FIG. 8B, Database image data GD DB Regarding [1], the database included in cluster CST1 Tag vector TAGV1 DB [1] weight is 8 / 20, included in cluster CST2 Database tag vector TAGV2 DB [1] weighted to 4 / 20, cluster CST3 Included database tag vector TAGV3 DB [1] weight 2 / 20, Cluster C Database tag vector TAGV4 included in ST4 DB [1] weight 3 / 20, Database tag vector TAGV5 contained in raster CST5 DB [1] weight 3 / It can be 20.

[0168] By the above method, for example, database image data GD DB Concept, technical content, and highlights This allows us to increase the weight of tag vectors that strongly represent features such as The image retrieval system 10 is capable of performing retrieval with a high degree of accuracy.

[0169] Next, an image search method using the image search system 10 will be described. 1 is a flowchart illustrating an example of the method.

[0170] [Step S11] First, a user of the image search system 10 inputs query image data GD Q Enter Query image data GD Q is supplied from the input unit 11 to the processing unit 13 via the transmission line 12. Or, the query image data GD Q is transmitted to the storage unit 15 or the database via the transmission line 12. The data is stored in the storage unit 15 or the database 17, and is transmitted to the processing unit 1 via the transmission path 12. 3 may be supplied.

[0171] Query image data GD Q For example, inventions, devices or designs before application, industrial products before release It may have images that explain the product, technical information, or technical idea.

[0172] [Step S12] Next, the query image data GD Q is input to the neural network of the processing unit 13. For example, a query image is input to the neural network 30 having the configuration shown in FIG. 3A or 3B. Data GD Q As a result, the processing unit 13 inputs the query image data G D Q Query image feature data GFD representing the feature Q For example, The data output from layer 31[m] shown in FIG. 3A is the query image feature data GFD Q and Alternatively, the data output from the pooling layer PL[m] shown in FIG. , Query image feature data GFD Q The query image feature data GFD Q is the database image feature data GFD DB Similarly, output data of two or more layers The query image feature data GFD may include: Q contains the output data of many layers. By this, the query image feature data GFD Q Query image data GD Q More accurately characterizes It can be said to accurately represent.

[0173] [Step S13] Next, the database image data GD DB Query image data GD Q Calculate the similarity to The calculation is performed by the science department 13.

[0174] FIG. 10 shows the database image data GD DB Query image data GD Q Similarity to 10 is a diagram showing the calculation of one query image data GD Q and 100 database image data GD DB and the neural network shown in Figure 3B. 10 shows an example in which the query image feature amount data G FD Q , and database image feature data GFD DB are x rows and y columns (x and y are 1. An example is shown in which the pooling value P is an integer equal to or greater than 1.

[0175] In this specification, the query image feature data GFD Q The pooling value of Value P Q and the database image feature data GFD DB The pooling value of Ring value P DB For example, the database image feature data GFD DB [ 1] pooled value P1 DB and the database image feature data Data GFD DBThe pooling value of

[0100] is pooled as P100 DB and write .

[0176] In the case shown in FIG. 10, the database image feature data GFD DB [1] or database Image feature data GFD DB For each of

[0100] , query image feature data GFD Q The similarity is calculated based on the database image data. GD DB [1] to database image data GD DB

[0100] , query image data G D Q It is possible to use the similarity to all the Database of image feature data GFD DB Regarding the query image feature data GFD Q Alternatively, the similarity to the database stored in the database 17 may be calculated. Image feature data GFD DB For part of the query image feature data GFD Q against Alternatively, the similarity may be calculated.

[0177] The similarity is preferably, for example, a cosine similarity. Alternatively, the similarity may be a Euclidean similarity. , Minkowski similarity. For example, the database image feature data GFD D B [1] Query image feature data GFD Q The cosine similarity for The larger the cosine similarity value, the more likely it is that the database image data GD D B is the query image data GD Q It can be said to be similar to

[0178] (Number 5) JPEG2026027343000002.jpg22167

[0179] Database image feature data GFD DB [2] or database image feature data GF D DB Query image feature data GFD of

[0100] Q The cosine similarity to The above method can be used to calculate the database image data GD DB [1]~ Database image data GD DB

[0100] , the query image data GD Q Similarity to can be calculated.

[0180] By calculating the cosine similarity as the similarity, the image retrieval system 10 can Cosine similarity can be calculated easily. Therefore, if the processing unit 13 has a GPU, the similarity can be calculated by the GPU. Therefore, the similarity can be calculated in a short time, and the image search system 10 can perform a search in a short time. This can be done.

[0181] [Step S14] Next, the database image data GD DB Query image data GD Q Calculating similarity to Based on the results, the query image data GD Q Query tag TAG that is associated with Q of The processing unit 13 acquires it.

[0182] 11A and 11B show the query tag TAG Q FIG. 1 is a diagram illustrating an example of a method for acquiring 11A, based on the similarity calculated in step S13, the database image Data GD DB [1] to database image data GD DB Rearrange

[0100] . Example For example, the most query image data GD Q Database image data GD with high similarity to D B In the case shown in FIG. 11A, the database image data GD DB [ 2] has the highest similarity of 0.999, and the database image data GD DB Similar to

[31] The degree of accuracy is 0.971, the second highest, and the database image data GD DB Similarity of

[73] is 0 .964, the third highest, and database image data GD DB The similarity of

[52] is 0.95 1, the fourth highest, database image data GD DB The similarity of

[28] is 0.937 and 5 It is said to be the second highest.

[0183] Next, the database image data GD DB The database type associated with GuTAG DB In the case shown in FIG. 11A, the data with the first to fifth highest similarities are extracted. Base image data GD DB Database tag associated with TAG DB Extracting Specifically, the database image data GD DB [2] The tag "aa a”, “bbb”, “ccc”, “ddd”, and “eee” and the database image data Ta GD DB

[31] The tags "aaa", "ccc", "fff", "g" gg" and "hhh" and database image data GD DB Linked to

[73] The tags "aaa", "bbb", "fff", "iii", and "kkk" are used, and the database Source image data GD DB

[52] The tags "aaa", "ccc", and "g" are linked to gg", "ppp", and "qqq" and the database image data GD DB

[28] Attached tags: "aaa", "kkk", "rrr", "sss", and "ttt" As shown in FIG. 11A, the extracted tags may overlap.

[0184] In the above, the database tag TAG DB Extracting database image data GD D B However, one aspect of the present invention is not limited to this. Database image data GD with a similarity greater than a predetermined value DB Database tags associated with In other words, the database tag TAG DB Extract the database image data Data GD DB The number of the elements does not need to be fixed.

[0185] Then, as shown in FIG. 11B, the number of occurrences of each extracted tag is calculated. For example, the tag "aaa" is a database image data GD DB [2], Database Image Image data GD DB

[31] , Database Image Data GD DB

[73] , Database Image Image data GD DB

[52] , and database image data GD DB

[28] Since they are linked, the number of occurrences is 5. The tag "ddd" is Ta GD DB [2], Database Image Data GD DB

[31] ,Database image data. GD DB

[73] , Database Image Data GD DB

[52] , and database image data Data GD DB In

[28] , the database image data GD DB [2] only Therefore, the number of occurrences is 1.

[0186] Next, as shown in FIG. 11B, a predetermined number of tags are sorted, for example, in descending order of the number of times of appearance. The extracted tags are then used as query tags. Q In the case shown in Figure 11B So, let's start with the most frequently appearing tag and then the query tag TAG Q Extract five tags as follows: Specifically, the most frequently occurring tag is "aaa" with 5 occurrences, and the second most frequently occurring tag is "aaa" with 3 occurrences. It extracts the frequently seen tag "ccc".

[0187] There are multiple tags that appear the same number of times, but it is not possible to extract all of the multiple tags. For example, if the database image data GD DB The tags associated with For example, the database image data GD DB The similarity ranking of Then, the database image data G that are linked to tags with the same number of occurrences are D DB The sum of the numbers representing the similarity ranking of the tags is compared, and the tags with the smallest sum are extracted first. It is possible.

[0188] In the case shown in FIG. 11B, the query tag TAGQ The number of occurrences of tag "aaa" is set to 5. The number of occurrences is 5, and the number of occurrences of the tag "ccc" is 3. Therefore, It is necessary to extract three tags from the tag. However, the tag with the number of occurrences of 2 is "b There are four tags: "bb", "fff", "ggg", and "kkk". You need to select three tags from the tag. Here, the tag "bbb" is linked. Database image data GD DB The similarity ranking of [2] is 1, and the database image data GD DB The similarity ranking of

[73] is 3. Therefore, the similarity ranking of the tag "bbb" is The total ranking of similarities for the tag "fff" is 4. Similarly, the total ranking of similarities for the tag "fff" is 5. The total similarity ranking for the tag "ggg" is 6, and the similarity ranking for the tag "kkk" is The total of the similarity rankings for the tag with the number of occurrences of 2 is 8. The smallest tag is "bbb", and the tags in ascending order are "fff", "ggg", and "kkk". Therefore, the tags "bbb", "fff", and "ggg" are used as the query tag TAG Q Let's say It is possible.

[0189] In summary, in the case shown in FIG. 11B, the tag “aaa” appears 5 times, and the tag “aaa” appears 5 times. Among the tags with the number of occurrences of 3, "ccc" and the tags with the number of occurrences of 2, the total similarity ranking is 1st to 3rd. The small tags "bbb", "fff", and "ggg" are used as the query tag. Q Tosu It is possible.

[0190] In addition, the database tag TAG DB Not only the words themselves, but also the database types GuTAGDB Search for synonyms of words in the query tag TAG Q For example, Thesaurus data is stored in advance in the storage unit 15 or the database 17. Database tag TAG DB and the words contained in it, and register them in the thesaurus as synonyms of the word. The word and the query tag TAG Q can be included in

[0191] In the case shown in FIG. 11B, the extracted database tag TAG DB The processing unit 13 selects Query tag TAG Q However, one embodiment of the present invention is not limited to this. ,database tag TAG extracted by users of image retrieval system 10 DB Present and present Query tag TAG from the tags Q The user of the image search system 10 Alternatively, for example, a database image with a high degree of similarity may be selected from the list of images in the image retrieval system 10. The database images are presented to the user, and the user of the image retrieval system 10 selects one of the presented database images. Then, database image data GD representing the selected database image may be D B Database tag associated with TAG DB All or part of the above can be used as a query tag. Q It may also be possible to use the following.

[0192] In the method shown in FIGS. 11A and 11B, the query tag TAG Q , the database tag TA G DB However, one aspect of the present invention is not limited to this. Stag TAG DBGenerate a new tag based on the query tag TAG Q Even if In the following, the database tag TAG DB A database tag vector TAGV representing D B Using the query tag TAG Q A query tag vector TAGV Q The processing unit 13 takes An example of how to obtain this will be described below.

[0193] The method illustrated in FIG. 11A uses the query tag vector TAGV Q Get In other words, it can be applied even when the database tag TAG DB Extraction of can be performed in a manner similar to that shown in FIG. 11A.

[0194] Database tag TAG DB After extraction, the extracted database tag TAG DB Representing the date Database tag vector TAGV DB By performing clustering on Generate a number of clusters. For example, the query tag you want to get is TAG Q The same number of classes as there are Clustering is performed using the K-means method, DBSCAN method, etc. can be done.

[0195] In FIG. 12A, the 25 database tags TAG shown in FIG. DB is processed by the processing unit 13. 12A shows an example of how the database tag TA G DB The database tag vector TAGV corresponding to DB Based on this, five clusters ( Cluster CST1, Cluster CST2, Cluster CST3, Cluster CST4, and Cluster CST5 For convenience of explanation, the vector shown in FIG. 12A is generated. A vector is a two-dimensional vector, with the horizontal axis being one component of the two-dimensional vector and the vertical axis being one component of the vector. represents the other component of a two-dimensional vector, but in reality, for example, a 300-dimensional vector The numbers in parentheses in FIG. 12A represent the extracted database tags. TAG DB For example, "aaa(5)" indicates the number of occurrences of the tag "aaa". This indicates that the current count is 5.

[0196] Next, for each of the clusters CST1 to CST5, a vector representing a representative point is calculated. Then, the vector representing the representative point is called the query tag vector TAGV Q Tosu In Figure 12A, the vector representing the representative point of cluster CST1 is used as the query tag. Vector TAGV1 Q The vector representing the representative point of cluster CST2 is the query tag vector. Tor TAGV2 DB The vector representing the representative point of cluster CST3 is defined as the query tag vector. TAGV3 Q The vector representing the representative point of cluster CST4 is the query tag vector T AGV4 Q The vector representing the representative point of cluster CST5 is the query tag vector TAG V5 Q This shows an example where

[0197] Each component of the vector representing the representative point is, for example, a database tag vector included in the cluster. Kutlu TAGV DB By the above procedure, the processing unit 1 3 is the query tag vector TAGV Q can be obtained.

[0198] Figure 12B shows the query tag vector TAGV1 Q or query tag vector TAGV5 Q The growth of 12B is a table showing components. Note that the components shown in FIG. 12B are an example for convenience of explanation.

[0199] As shown in Figure 12B, the query tag vector TAGV Q can be weighted The weight is, for example, the weight of the database tag vector TAGV included in one cluster. D B The number of database tags TAG DB Total number of For example, in FIG. 12A and FIG. 12B, 25 pieces of data Base tag DB In addition, cluster CST1 contains 11 Database tag vector TAGV DB Cluster CST2 contains four databases. Tag Vector TAGV DB Cluster CST3 contains five database tags. Vector TAGV DB Cluster CST4 contains two database tag vectors TAGV DB Cluster CST5 contains three database tag vectors, TAGV DB Therefore, as shown in FIG. 12B, the cluster CST1 contains Query tag vector TAGV1 Q The weight of 11 / 25 is included in cluster CST2. Query tag vector TAGV2 Q The weight of the query in cluster CST3 is set to 4 / 25. Retag Vector TAGV3 Q The weight of 5 / 25, the query tag included in cluster CST4 Vector TAGV4 Q The weight of the query tag vector in cluster CST5 is set to 2 / 25. TAGV5 Q The weight of can be set to 3 / 25.

[0200] By the above method, for example, query image data GD Q Concept, technical content, and points of interest The weight of the tag vector that expresses the feature strongly can be increased. Stem 10 can perform searches with high accuracy.

[0201] The query tag TAG shown in steps S13 and S14 Q For example, the method of obtaining data is Database tag DB Query without tagging Q Compared to the method of obtaining This is a convenient method, and the image search system 10 can therefore perform searches in a short time. Also, the query tag TAG according to the method shown in steps S13 and S14 Q To obtain For example, if a user of the image search system 10 searches all query tags TAG Q Specify and User query tag TAG Q Compared with the case where no candidate is presented, the query image data Ta GD Q By comprehensively acquiring tags that represent the concept, technical content, and points of interest of the corresponding image, Therefore, the image search system 10 can perform a search easily and with high accuracy. Cut.

[0202] [Step S15] Next, the processing unit 13 extracts the database image feature data GFDDB and database tag base Kutlu TAGV DB and data D including DB Also, the processing unit 13 acquires the query image. Image feature data GFD Q and the query tag vector TAGV Q and data D including Q Get do.

[0203] Figure 13 shows the data D DB , and Data D Q FIG. 1 is a diagram illustrating an example of the configuration of a database image special GFD data DB , and query image feature data GFD Q is the same as the configuration shown in Figure 10. The database tag vector TAGV DB is the component VC DB [1] to component VC DB [h] (h is an integer of 2 or more). Query Tag Vector TAGV Q is the component VC Q [1] to component VC Q Configuration with [h] Here, for example, one image data is represented by a 300-dimensional vector. If five tags are linked, h will be 1500.

[0204] In this specification, for example, a database tag vector TAGV DB [1] contains The component VC1 DB and the database tag vector TAGV DB

[0100] has Ingredients: VC100 DB It is written as follows.

[0205] As mentioned above, the term component can sometimes be replaced with the term value. In this case, both the image feature data and the tag vector are a set of multiple values. Therefore, the terms data and vector can be used interchangeably. It may be possible to do this.

[0206] [Step S16] Next, Data D DB Data D Q The processing unit 13 calculates the similarity between the In this case, data D DB [1] Data D DB For each of

[0100] , TaD Q Then, the similarity is calculated based on the database image data G D DB [1] to database image data GD DB

[0100] , the query image data GD Q Therefore, the similarity calculated by the processing unit 13 in step S13 can be used as the similarity to , database image data GD DB Query image data GD Q Correct the similarity to It is possible.

[0207] Here, when weights are assigned to tag vectors as shown in FIG. 8B and FIG. 12B, for example, Weighting can be performed by multiplying the components of the tag vector by weights.

[0208] Data D DB Data D Q The similarity to is calculated by the processing unit 13 in step S13. It is preferable to use the same type of similarity as that of the cosine similarity calculated in step S13. If you do, Data D DB Data D Q The cosine similarity is calculated as the similarity to It is preferable to release it.

[0209] For example, data D DB [1] Data D Q The cosine similarity for is calculated using the following formula: It is possible.

[0210] (Number 6) JPEG2026027343000003.jpg23167

[0211] Data D DB [2] Data D DB

[0100] Data D Q Cosine similarity to can be calculated in the same way. DB [1] Data D D B

[0100] Data D Q This allows us to calculate the similarity to The database image data GD calculated in step S13 DB [1] or database image data Data GD DB

[0100] , the query image data GD Q It is possible to correct the similarity to Cut.

[0212] The ratio between the number of values ​​in the image feature data and the number of components in the tag vector is adjusted. For example, by adjusting the query image feature data, the search results can be changed. GFD Q and the database image feature data GFD DB Increasing the number of values ​​that or query tag vector TAGV Q The number of components that Vector TAGV DB If the number of components in is reduced, the similarity after correction will be For example, the database image data GD DB The feature of the query image Data GD Q If the feature is similar to the database tag TAG DB is the query tag TA G Q Even if it is slightly different from the database image data GD DB Query image data Data GD Q On the other hand, the similarity after correction for the query image feature data GFD Q The number of values ​​that the database image feature data GFD has DB Decrease the number of values ​​that or query tag vector TAGV Q The number of components and database tag vectors Kutlu TAGV DB As the number of components increases, the similarity after correction becomes more For example, the database tag TAG DB is the query tag TAG Q Similar to For example, database image data GD DB The feature quantity of the query image data GD Q The features and Even if there are slight differences, the database image data GD DB Query image data GD Q The similarity after correction for

[0213] In order to increase or decrease the number of components in the tag vector, In addition, for example, the value of the image feature data can be increased or decreased. By using only some of the values ​​to calculate the similarity, we can calculate the similarity that emphasizes tags. For example, it is possible to use a value that represents the feature amount of a part of an image that does not give a strong impression when viewed. ,By not using it to calculate the similarity, data that has a significantly different appearance from the query image To calculate similarity that emphasizes tags while suppressing the similarity of base images from becoming too high Therefore, the image retrieval system 10 can perform a highly accurate search.

[0214] In addition, the value of the image feature data or the component of the tag vector is multiplied by a predetermined coefficient. For example, the search results can be changed by changing the query image feature data. GFD Q and the database image feature data GFD DB has a value greater than 1 By multiplying by a large real number, the similarity after correction is a result that emphasizes image features. In addition, the query tag vector TAGV Q and database tags Vector TAGV DB By multiplying the component of by a real number between 0 and 1, the corrected The similarity can be a result that emphasizes image features. For example, Data GFD Q and the database image feature data GFD DB has a value of 0 By multiplying it by a real number equal to or greater than 1, the corrected similarity is made to be a result that emphasizes tags. In addition, the query tag vector TAGV Q and database tags Vector TAGV DB The corrected class is obtained by multiplying the components of The similarity can be a tag-oriented result.

[0215] [Step S17] Next, the ranking including information on the corrected similarity ranking calculated in step S16 is The data is generated by the processing unit 13 and output as a search result to the outside of the image search system 10 .

[0216] The processing unit 13 transmits the ranking data to the storage unit 15 or the database via the transmission path 12. The processing unit 13 can also supply the ranking data to the transmission line 12. The output unit 19 can then provide the image data to the image search system. The ranking data can be provided external to the system 10.

[0217] The ranking data is the ranking of the similarity of each database image to the query image. The ranking data can include the value of the file in the database image. It is preferable that the path is included. This allows the user of the image search system 10 to The desired image can be easily accessed from the data. It may also be possible to check the tags associated with the database images. For example, publication data representing the publication in which the database image is published is stored in a database17. If the ranking data is stored, the user of the image search system 10 can You can easily access publications that contain database images. 1 is an example of an image search method using the image search system 10.

[0218] In the image search method using the image search system 10, first, images are searched for in the database without being linked to tags. Image data GD DB Query image data GD Q Then, calculate the similarity to By linking tags, the similarity is corrected. For example, the feature value is similar to the query image. However, it prevents database images with different concepts, technical content, and points of interest from being searched. It is possible.

[0219] For example, query image data GD Q The database images with the 1st to 5th highest similarity to Data GD DB Database tag associated with TAG DB Based on this, processing unit 13 is the query tag TAG Q In this case, the database with the sixth or lower similarity is Image data GD DB In this case, image data with different concepts, technical content, and points of interest from the query image are included. Therefore, it is possible to prevent noise images from being mixed into the search results. This prevents the image you want to search from being output. The system 10 is capable of performing searches with a high degree of accuracy.

[0220] In addition, in the image search method using the image search system 10, the database tag TAG DB of Based on query tag TAG Q The acquisition method is to acquire the database tag TAG DB Query without tagging Q This is a simpler method than the method of obtaining The image search system 10 can perform searches in a short time. TAG DB Based on the query tag TAG Q For example, the image search system 1 0 users all query tags TAG Q Specify and query tag TAG to the user Q Compared with the case where no candidates are presented, the query image data GD QThe outline of the image corresponding to It is possible to comprehensively acquire tags that represent ideas, features, technical content, points of interest, etc. The image search system 10 can perform searches easily and with high accuracy.

[0221] <1-3. Image search method-2> In the image search method shown in FIG. 9 etc., a user of the image search system 10 inputs a query tag TAG Q Although the image search system 1 does not include the input of the image search result, the present invention is not limited to this. 0 users of the query tag TAG Q Image retrieval system 1 when manually entering part of 14 is a flowchart showing an example of an image search method using 0. Even when the image search system 10 is operated, the image search is performed by the image search method shown in FIG. As with the operation of the system 10, it is advisable to perform the process shown in FIG. 2 in advance. .

[0222] [Step S21] First, a user of the image retrieval system 10 inputs query image data GD Q In addition, the query tag TA G Q is input to the input unit 11. The query tag TA G Q and the query tag TAG Q The content of the above can be set by the user. Also, the query tag TAG will be automatically acquired in a later step. Q Query tags, including TAG Q The number of the items may be set by the user.

[0223] FIG. 15 shows the query image data GD Q , and query tag TAG Q The input to the input unit 11 is In the case shown in FIG. 15, the user of the image retrieval system 10 enters a query Image data GD Q In addition, query image data GD Q "Circuit diagram" and "Semiconductor" Query Tags Q is being entered.

[0224] Here, the query tag TAG input to the input unit 11 Q By changing Image data GD DB Query image data GD Q The calculation result of the similarity for For example, the query tag "capacitor" can be used. Q When inputting the above into the input unit 11, It is possible to reduce the similarity of database image data representing circuit diagrams in which no element is drawn. can.

[0225] [Step S22] Next, the query image data GD Q is input to the neural network of the processing unit 13. For example, a query image is input to the neural network 30 having the configuration shown in FIG. 3A or 3B. Data GD Q As a result, the processing unit 13 inputs the query image data G D Q Query image feature data GFD representing the feature Q can be obtained.

[0226] [Step S23] Next, the processing unit 13 extracts the database image feature data GFD DB and database tag base Kutlu TAGV DB and data D including DB Also, the processing unit 13 acquires the query image. Image feature data GFD Qand the query tag vector TAGV Q and data D including Q Get do.

[0227] Here, one database image data GD DB Database tag associated with TAG DB The number of the query tags TAG Q If the number of Data D DB The tags included in the database image data GD DB From the tags associated with For example, select one database image data GD DB 5 database tags T AG DB The query tag TAG input to the input unit 11 is Q In this case, there are five database tags TAG DB Of these, For example, the tag with the highest TF-IDF and the tag with the second highest TF-IDF are taken as data D DB The tag may be a tag that

[0228] [Step S24] Next, data GD DB Data GD Q The processing unit 13 calculates the similarity to the The similarity can be calculated in the same manner as shown in FIG.

[0229] [Step S25] Next, Data D DB Data D Q Based on the similarity calculation result for the query tag TA G Q Add and modify.

[0230] 16A and 16B show the query tag TAGQ FIG. 1 is a diagram illustrating an example of a method for adding , as shown in FIG. 16A, based on the similarity calculated in step S24, data D DB Line up In Figure 16A, 100 data D DB Here is an example of rearranging the , the most data D Q Data D that has high similarity to DB In Figure 16A, the data is sorted in descending order. In the case shown, data D DB The similarity of [2] is 0.999, the highest, and data D DB The similarity of

[0041] is 0.971, which is the second highest, and data D DB The similarity of

[53] is 0.964 The third highest in Data D DB The similarity score of

[22] is 0.951, which is the fourth highest. DB

[88] has the fifth highest similarity score of 0.937.

[0231] Next, data D with high similarity DB The database image data GD DB tied to Database tag TAG DB In the case shown in FIG. 16A, the similarity is 1 or less. The fifth highest data D DB The database image data GD DB It is tied to Database tag TAG DB Specifically, the database image data G D DB [2] Tags associated with "aaa", "bbb", "ccc", "ddd" , and "eee" and database image data GD DB Tags associated with

[41] "aaa", "ccc", "fff", "ggg", and "hhh" and the database image Image data GD DB

[53] Tags associated with "aaa", "bbb", and "fff" , "iii", and "kkk" and the database image data GD DB Linked to

[22] The tags "aaa", "ccc", "ggg", "ppp", and "qqq" are used, and Database image data GD DB

[88] Tags associated with "aaa" and "kkk" , "rrr", "sss", and "ttt". Similarly, the extracted tags may overlap.

[0232] Then, as shown in FIG. 16B, the number of occurrences of each extracted tag is calculated. .

[0233] Next, as shown in FIG. 16B, a predetermined number of tags are further extracted from the extracted tags. The extracted tag is used as a new query tag TAG Q In the case shown in FIG. 16B, In step S21, two tags ("circuit diagram" and "semiconductor") have already been entered in the query tag TAG Q By adding three tags, the query tag TAG Q of The number of items is stored in one database image data GD DB Database tag T associated with AG DB Let's say there are 5, which is the same number as above.

[0234] New query tag TAG Q The tag is extracted in the same manner as shown in FIG. 11B. For example, it is possible to extract tags in order of the most frequently occurring tags. There are multiple tags that appear the same number of times, but it is not possible to extract all of the multiple tags. For example, if the data D has a higher similarity, DB The database image data GD DB In the case shown in Figure 16B, the tag "aaa" can be extracted. , "bbb", "ccc" as new query tags Q It can be extracted as:

[0235] In summary, in the case shown in FIG. 16B, the image search system 10 is used in step S21. In addition to the tags "circuit diagram" and "semiconductor" input by the user to the input unit 11, the tags "aaa" and "bb The five tags with "b" and "ccc" added are now the new query tag TAG Q can be do.

[0236] In addition, a part or all of what the user of the image search system 10 inputs into the input unit 11 is used as a query term. GuTAG Q For example, you can delete the tags "circuit diagram" and "semiconductor" from the tag TAG Q mosquito 16B, and five tags are extracted from the tags shown in FIG. 16B to create a new tag TAG Q as In this case, for example, the tags "aaa", "bbb", "ccc", "fff", "g gg" as a new tag Q It can be said that:

[0237] [Step S26] Next, the query tag TAG Q In response to additions and modifications, data D DB Add the tag that has Modify. For example, one data D DB The database tag vector TAGV DB The number of query tags TAG Q Make it equal to the number of .

[0238] [Step S27] Next, data GD DB Data GD Q The processing unit 13 calculates the similarity for the The similarity can be calculated in the same manner as in step S24. Data GD DB Data GD Q The similarity to the

[0239] [Step S28] Next, the ranking including information on the corrected similarity ranking calculated in step S27 is The data is generated by the processing unit 13 and output as a search result to the outside of the image search system 10 . This allows the user of the image retrieval system 10 to, for example, search for a query image for each database image. Check the similarity ranking for the image, the similarity value, the searched database images, tags, etc. It is possible.

[0240] [Step S29, Step S30] Next, the user of the image search system 10 checks whether the ranking data is the expected result. If the expected result is obtained, the search ends. If the expected result is not obtained, the screen A user of the image retrieval system 10 may input a query tag TAG Q After adding or modifying, step S Return to 23. The above is an example of an image search method using the image search system 10.

[0241] This embodiment mode can be combined with other embodiment modes as appropriate. In the case where multiple configuration examples are shown in one embodiment, the configuration examples may be combined as appropriate. It is possible to do this.

[0242] (Embodiment 2) In the first embodiment, the image retrieval system 10 includes a database of image data GD DB The entire area of and the query image data GD Q By comparing the entire area of ​​the database image data with Data GD DB Query image data GD Q The similarity was calculated as follows: This is not a limitation. For example, the database image data GD DB and a part of the query image Data GD Q By comparing the entire area of ​​with the database image data GD DB of Query image data GD Q Alternatively, the similarity to the database image data may be calculated. Ta GD DB and the entire region of the query image data GD Q By comparing some areas of Database image data GD DB Query image data GD Q Calculate the similarity to That's fine.

[0243] <2-1. Image search method-3> FIG. 17 shows the database image data GD DB and the query image data GD Q of By comparing the entire region with the database image data GD DB Query image data Ta GD Q Image retrieval method using image retrieval system 10 when calculating similarity to First, the image search system 10 performs step S11 shown in FIG. 9 or step S14 shown in FIG. Step 21 shown in is performed.

[0244] [Step S31] Next, the processing unit 13 calculates the query image data GD Qand database image data GD DB Compared to In comparison, the query image data GD Q Database image data containing areas with high matching scores GD DB Here, the extracted database image data GD DB Extract the image data Data GD Ex Query image data GD Q and database image data GD DB Relative to The comparison can be performed, for example, by region-based matching.

[0245] An example of the operation of step S31 will be described in detail with reference to FIGS. In step S31, as shown in FIG. 18A, the query image data GD Q n pieces (n is 1 or more) Integer) database image data GD DB where n is the number of data Database image data GD stored in base 17 DB The number may be the same as the number of It may be smaller than that. Also, n is the number of database entries stored in the database 17. Image data GD DB In this case, the number of items stored in the database 17 may be larger than the number of items stored in the database 17. Database image data GD DB In addition, image data stored in the storage unit 15 or an image input to the processing unit 13 via the input unit 11 from outside the image search system 10. image data and query image data GD Q where n is the database image data. GD DB Even if the number of images is less than the number of images stored in the storage unit 15, the image data or image search Image data input from outside the system 10 to the processing unit 13 via the input unit 11 and a query Image data GD Q may be compared with.

[0246] When n is small, the operation of step S31 can be performed in a short time. If query image data GD Q Database image data containing areas with high matching scores GD DB can be extracted with high accuracy.

[0247] FIG. 18B shows the query image data GD Q and database image data GD DB and area-based 1 is a diagram illustrating a procedure for comparing by image matching. Data GD Q The number of pixels of the corresponding image is 2 × 2, and the database image data GD DB Compatible with The number of pixels of the image to be processed is 4 x 4. In other words, the query image data GD Q is a 2x2 pixel value Database image data GD DB has 4x4 pixel values.

[0248] In FIG. 18B, the query image data GD Q The 2×2 pixel values ​​of vq11, pixel value vq12, pixel value vq21, pixel value vq22. For example, Image data GD Q In this example, the pixel value corresponding to the pixel in the first row and first column is vq11, and the pixel value corresponding to the pixel in the first row and second column is vq21. The pixel value corresponding to the pixel in the 2nd row and 1st column is pixel value vq12, and the pixel value corresponding to the pixel in the 2nd row and 1st column is pixel value vq3. The pixel value corresponding to the pixel in the second row and second column is vq21 and vq22, respectively. Database image data GD DB The 4×4 pixel values ​​of The pixel value is vdb44. For example, the database image data GDDB In 1 row, 1 column The pixel value corresponding to the first pixel is vdb11, and the pixel value corresponding to the pixel in the 1st row and 4th column is vdb2. The pixel value corresponding to the pixel in the 4th row, 1st column is vdb41, and the pixel value corresponding to the pixel in the 4th row, 4th column is vdb42. The pixel value corresponding to the pixel is set to vdb44.

[0249] First, the pixel values ​​vq11, vq12, vq21, and vq22, and the pixel Compare the pixel value vdb11, pixel value vdb12, pixel value vdb21, and pixel value vdb22. This allows the query image data GD Q and database image data GD DB Of It consists of pixel values ​​vdb11, vdb12, vdb21, and vdb22. In FIG. 18B, the degree of coincidence between the area and the database Image data GD DB Among the pixel values ​​of the query image data GD Q The pixel value to be compared with is shown surrounded by a dotted line as a comparison data area 21.

[0250] Next, the comparison data area 21 is DB 1 for pixel values ​​that The pixel values ​​are then moved by the number of columns, and the pixel values ​​are compared in the same way to calculate the degree of match. , pixel value vq12, pixel value vq21, pixel value vq22, pixel value vdb12, pixel value vdb13, pixel value vdb22, and pixel value vdb23 are compared. Eri image data GD Q and database image data GD DB Among them, pixel value vdb12, A region consisting of pixel values ​​vdb13, vdb22, and vdb23, The degree of agreement can be calculated.

[0251] After that, the comparison data area 21 is also DB For pixel values ​​that Then, the pixel values ​​are compared in the same way and the degree of match is calculated. 11, pixel value vq12, pixel value vq21, pixel value vq22, pixel value vdb13, pixel value vdb14, pixel value vdb15, pixel value vdb16, pixel value vdb17, pixel value vdb18, pixel value vdb19, pixel value vdb20, pixel value vdb21, pixel value vdb22, pixel value vdb23, pixel value vdb24, pixel value vdb25, pixel value vdb26, pixel value vdb The pixel value vdb14, pixel value vdb23, and pixel value vdb24 are compared. , query image data GD Q and database image data GD DB Of which pixel value vdb13 , a region consisting of pixel values ​​vdb14, pixel values ​​vdb23, and pixel values ​​vdb24; The degree of match can be calculated.

[0252] Next, the comparison data area 21 is DB 1 for pixel values ​​that Move the row and the database image data GD DB The pixel values ​​in the second and third rows of and the query image data GD Q The pixel values ​​constituting the pixel are compared for each column in the same manner as above. This allows the database image data GD DB It consists of the pixel values ​​of the second and third rows of The area and the query image data GD Q The degree of agreement can be calculated for each column in the same way as above. can.

[0253] Then, the comparison data area 21 is stored in the database image data GD DB For pixel values ​​that Move one line and save the database image data GD DB The pixel values ​​in the third row and the pixel values ​​in the fourth row value and the query image data GD Q are compared with the pixel values ​​constituting the matrix in each column in the same manner as above. This allows the database image data GD DB the pixel values ​​of the third and fourth rows of Query image data GD Q The degree of agreement can be calculated for each column in the same manner as above.

[0254] After the above operations are performed, for example, the highest degree of match is selected as the database image data GD DB Noku Eri image data GD Q The above is the degree of match for n database image data GD DB Then, n pieces of database image data GD DB From among , query image data GD Q Database image data GD with high degree of match DB The extracted image Image data GD Ex For example, the database image data GD DB Extract the specified number of image data GD Ex Or, for example, Eri image data GD Q Database image data GD that matches with the specified value or more DB The Output image data GD Ex It may be extracted as

[0255] Also, the database image data GD DB This means that the query image Data GD Q The database image data GD was compared with DB All of the extracted image data Ta GD Ex This can be rephrased as:

[0256] FIG. 19 shows the database image data GD DB FIG. 19 is a diagram illustrating the extraction of , database image data GD DB [1] to database image data GD DB [3] or Then, one image data is extracted as image data GD Ex An example of extracting the following is shown.

[0257] The query image data GD shown in FIG. Q The corresponding image has, for example, a transistor symbol. Also, the database image data GD shown in FIG. DB [2] The corresponding image contains a transistor symbol, but the database image data GD D B [1] and database image data GD DB [3] does not include the symbol of a transistor. In this case, the database image data GD DB [2] Query image data GD Q The degree of match with the database image data G D DB [1] and database image data GD DB [3] Query image data GD Q Against Therefore, the degree of match is higher than that of the database image data GD DB [2] Extract the image Data GD Ex It can be extracted as:

[0258] In addition, the query image data GD Q and database image data GD DB Comparison of and agreement Calculations are SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), NCC (Normal zed Cross Correlation), ZNCC(Zero-mean No. rmalized Cross Correlation), POC (Phase-On This can be done using methods such as ly Correlation.

[0259] In addition, in FIG. 18B, the comparison data area 21 is the database image data GD DB The pixel Although the values ​​are shifted by one column or one row, this is not a limitation of the present invention. The comparison data area 21 is stored in the database image data GD DB For the pixel values ​​of For example, the pixel value vq11, the pixel value vq21, the pixel value vq31, the pixel value vq41, the pixel value vq51, the pixel value vq61, the pixel value vq71, the pixel value vq81, the pixel value vq91, the pixel value vq10, the pixel value vq11, the pixel value vq12, the pixel value vq13, the pixel value vq14, the pixel value vq15, the pixel value v The pixel values ​​vq12, vq21, and vq22, and the pixel values ​​vdb11 and vdb 12, pixel value vdb21, and pixel value vdb22, and immediately after comparing, pixel value vq11 , pixel value vq12, pixel value vq21, pixel value vq22, pixel value vdb13, pixel value vdb14, pixel value vdb23, and pixel value vdb24 may be compared. , pixel value vq11, pixel value vq12, pixel value vq21, pixel value vq22, and pixel value v db12, pixel value vdb13, pixel value vdb22, and pixel value vdb23 are compared. In addition, the pixel values ​​vq11, vq12, vq21, and value vq22, pixel value vdb13, pixel value vdb14, pixel value vdb23, and pixel value v Immediately after comparing db24 with pixel value vq11, pixel value vq12, pixel value vq21, and and pixel value vq22, pixel value vdb31, pixel value vdb32, pixel value vdb41, and pixel value vdb42. It may be compared with the prime value vdb42.

[0260] By increasing the movement range of the comparison data area 21, the query image data GD Q has Pixel values ​​and database image data GD DB Reduce the number of comparison operations between the pixel values ​​of This allows the database image data GD DB Query image data GD Q The degree of match can be calculated in a short time.

[0261] In FIG. 18A, one query image data GD Q n database image data GD D B 20A and 20B are shown as an example for comparison, but one aspect of the present invention is not limited to this. As shown in FIG. 1, the query image data GD Q Based on the number of pixel values Multiple query image data GD Q 20A shows the input to the processing unit 13. Input query image data GD Q Based on the above, query image data with different numbers of pixel values ​​are generated. Data GD Q [1], Query image data GD Q [2], and query image data GD Q [3] As shown in FIG. 20A, the query image data GD Q [1] The number of pixels of the corresponding image and the query image data GD Q [2] The number of pixels in the image corresponding to the query Image data GD Q [3] and the number of pixels of the corresponding image are different. Image data GD Q [1] to query image data GD Q The image corresponding to [3] is processed by the processing section. Query image data GD entered in 13 QIt is an enlarged or reduced version of the image corresponding to It can be said that.

[0262] Multiple query image data GD Q When generating the query image data GD Q Noso For each, the database image data GD DB [1] to database image data G D DB [n]. This compares the database image data GD DB [1 ] or database image data GD DB For each of [n], multiple query image data are Data GD Q Then, for example, the degree of match for each of the above multiple Query image data GD Q The highest match score among the matches to the database image data is Data GD DB The query image data GD input to the processing unit 13 Q The degree of agreement with It is possible.

[0263] For example, in the case shown in FIG. 20A, the query image data GD Q [1] is a database image data Data GD DB [1] to database image data GD DB Compare with each of [n] and Eri image data GD Q [2] is the database image data GD DB [1] or database Image data GD DB [n] and compare it with the query image data GD Q [3] Data Base image data GD DB [1] to database image data GD DB Each of [n] This compares the database image data GD DB [1] or database image Data GD DB For each of [n], the query image data GD Q Match for [1] degree, query image data GD Q [2] and query image data GD Q [3] The degree of match can be calculated.

[0264] For example, the query image data GD Q [1] Matching score for query image data GD Q [2] and query image data GD Q The highest match for [3] The degree of match is calculated using the database image data GD DB The query image data input to the processing unit 13 Data GD Q For example, the database image data GD D B [1] Query image data GD Q [1] Matching score for query image data GD Q [2 ] and the degree of match with the query image data GD Q The highest match for [3] Matching the database image data GD DB [1] The query image input to the processing unit 13 Data GD Q It can be the degree of agreement with

[0265] Query image data GD Q and the database image data GD DB corresponds to Even if the image and the image show the same elements, if the sizes of the elements are different, The database image data GDDB Query image data GD Q High agreement with In the case shown in FIG. 20B1, the query image data may be judged not to include the region. Data GD Q and the database image data GD DB The image corresponding to The same element, a transistor symbol, is shown on both sides. However, the query image data GD Q The size of the transistor symbol shown in the image corresponding to the database image Data GD DB The size of the transistor symbol shown in the corresponding image is different. In this case, the database image data GD DB Query image data GD Q One against It may be deemed to be of low severity.

[0266] On the other hand, in the case shown in FIG. 20B2, the query image data GD Q The corresponding image and the database Source image data GD DB The image corresponding to the transistor symbol is the same as the image of the transistor symbol ... The elements are shown, and the size of both elements is equal. Therefore, the database image data GD DB Query image data GD Q If a region with a high degree of match is included, the processing unit 13 can be determined.

[0267] As shown in FIG. 20A, a plurality of query image data GD Q To generate and query image data GD Q Enlarge the size of the elements shown in the corresponding images, or Therefore, the query image data GD input to the processing unit 13 can be reduced. Q Compatible with The image to be displayed and the database image data GD DB The image corresponding to and the same element are different Even if the image data is shown by size, it is possible to ensure a high degree of agreement between the two images. For example, the processing unit 13 may receive query image data GD shown in FIG. Q If , the query image data GD Q The number of pixel values ​​in the query Image data GD Q By generating the database image data GD DB Query image of Data GD Q As a result, the degree of match with the database Image data GD DB The query image data GD input to the processing unit 13 Q Degree of agreement with can be calculated with high accuracy.

[0268] [Step S32] Next, the extracted image data GD Ex From the query image data GD Q Areas with high agreement with The partial image data GD part The processing unit 13 extracts the following. For example, FIG. 18B By the method shown in DB Query image data GD for each region Q When the degree of match is calculated for each of the partial image data GD p art Therefore, the partial image data GD part The number of pixel values ​​that Eri image data GD Q can be equal to the number of pixel values ​​that .

[0269] 21A and 21B are diagrams showing an example of the operation of step S32. and the extracted image data GD shown in FIG. 21B Ex [1] to extracted image data GD Ex [4] Query image data GD Q The areas with high agreement are shown with hatching. As shown in Figures 21A and 21B, the hatched areas are extracted and Image data GD part [1] to partial image data GD part [4] can be In FIG. 21A and FIG. 21B, the extracted image data GD Ex [1] to extracted image data G D Ex The image data extracted from [4] are divided into partial image data GD part [1]No Partial image data GD part [4]

[0270] FIG. 21A shows one query image data GD Q In contrast, database Image data GD DB In this example, partial image data GD part The number of pixels of the images corresponding to the above can all be made equal.

[0271] FIG. 21B shows a plurality of query image data GDs with different numbers of pixel values ​​as shown in FIG. 20A. Q to On the other hand, database image data GD DB This shows an example where a comparison is made with Partial image data GD part The number of pixels in the image corresponding to the query with the highest match is, for example, Image data GD Q Therefore, the number of pixels of the partial image can be made equal to the number of pixels of the image corresponding to the partial image. Data GDpart If there are multiple partial image data GD part of the image corresponding to The number of pixels is the partial image data GD part In Figure 21B, Image data GD part [1] to partial image data GD part [4] The image corresponding to the image This shows examples of different prime numbers.

[0272] In addition, the partial image data GD part In this case, the partial image data does not need to be extracted. Data GD part Extract image data GD appropriately Ex By reading it as follows, the following explanation applies. Alternatively, the extracted image data GD Ex The entire image data GD part For example, the query image data GD Q and extracted image data Data GD Ex Increasing or decreasing the number of pixel values ​​in one or both of the image data. By this, the partial image data GD part The image search system 10 can be used without extracting the The image retrieval method can be performed.

[0273] [Step S33] Next, the query image data GD Q is input to the neural network of the processing unit 13. As a result, the processing unit 13 generates the query image feature amount data GFD Q Also, obtain the partial image Data GD part is input to the neural network of the processing unit 13. The processing unit 13 generates the database image feature data GFD DB Get the query image data. GDQ , and partial image data GD part For example, the configuration shown in FIG. 3A or FIG. 3B The image can be input to the neural network 30. When operating in the manner shown in 17, step S02 shown in FIG. 2 does not need to be performed. , database image data GD DB A database image feature data that represents the feature of the entire region Data GFD DB does not have to be obtained.

[0274] As explained in the first embodiment, the data is used as training data for the neural network 30. Database image data GD DB Here, the image data used for training data can be The number of pixel values ​​contained in the image data to be input to the neural network 30 is Therefore, the number of pixel values ​​is preferably equal to the number of pixel values. When doing this, the database image data GD used for learning data DB etc., pixels as needed It is preferable to adjust the number of values ​​by increasing or decreasing it. Query image data GD to 0 Q , or partial image data GD part When entering According to the query image data GD Q , or partial image data GD part The pixel value of It is preferable to increase or decrease the number of pixels. Preferably, this is done by, for example, zero padding.

[0275] FIG. 22A shows the database image data GD DB This section explains how to adjust the number of pixel values ​​that In the case shown in FIG. 22A, the database image data GD DB [1] Database image data GD DB The number of pixel values ​​in [4] is different. , database image data GD DB [1] to database image data GD DB [4] When used as training data for the neural network 30, these It is preferable to make the number of pixel values ​​contained in the image data uniform.

[0276] FIG. 22B shows the partial image data GD part FIG. 10 is a diagram illustrating adjustment of the number of pixel values ​​in Partial image data GD part The number of pixel values ​​that the neural network 3 has is It is preferable to set the number of pixel values ​​equal to the number of pixel values ​​contained in the image data used for learning 0. Query image data GD to neural network 30 Q When entering a query, Image data GD Q The number of pixel values ​​of the image It is preferable to set it equal to the number of pixel values ​​contained in the data.

[0277] After performing step S33, the image retrieval system 10 performs step S13 shown in FIG. 9 or 14, step S23 is performed. Specifically, step S11 is performed before step S31. If so, step S13 is performed after step S33, and step S13 is performed before step S31. If step S21 is performed, step S23 is performed after step S33. Image data GD DB and the query image data GD Q Compare the entire area of By doing so, the database image data GD DB Query image data GD Q Similarities 10 is an example of an image retrieval method using the image retrieval system 10 when calculating the degree.

[0278] In the method shown in FIG. 17, the query image data GD Q and database image data GD DB and By comparing using region-based matching, query image data GD Q Areas with high agreement with Database image data GD including the area DB The extracted image data GD Ex Extracted as Then, the extracted image data GD Ex The area with the highest degree of match is then extracted as partial image data GD part Extract the query image data GD Q and partial image data GD part , processing The neural network of the unit 13 is inputted. Data GD DB By extracting the query image data GD Q Match the corresponding image Database image data GD representing database images that do not contain highly relevant images DB But, This can prevent the information from being input to the neural network of the logic unit 13. So, query image data GD Q A database that contains some images similar to the image corresponding to Images can be searched for with high accuracy in a short time. Q Compared to Database image data GD to compare DB If the number of images is small, Ta GD DBThe above search can be performed with high accuracy in a short time without extracting the above.

[0279] <2-2. Image search method-4> FIG. 23 shows the database image data GD DB and the entire region of the query image data GD Q One By comparing the area of ​​the database image data GD DB Query image data Ta GD Q Image retrieval method using image retrieval system 10 when calculating similarity to First, the image search system 10 performs step S11 shown in FIG. 9 or step S14 shown in FIG. Step 21 shown in is performed.

[0280] [Step S41] Next, the processing unit 13 calculates the query image data GD Q and database image data GD DB Compared to In comparison, the query image data GD Q Database image data GD with high matching degree to part of DB The extracted image data GD Ex Query image data GD Q and database Image data GD DB The comparison with the above is performed by, for example, area-based matching, similar to step S31. This can be done by:

[0281] An example of the operation of step S41 will be described in detail with reference to FIGS. 24 and 25. In step S41, as shown in FIG. 24A, the query image data GD Q n databases Image data GD DB Compare with each of the above.

[0282] FIG. 24B shows the query image data GD Qand database image data GD DB and area-based 1 is a diagram illustrating a procedure for comparing by image matching. Data GD Q The pixel size of the corresponding image is 4x4, and the database image data GD DB Compatible with The number of pixels of the image to be processed is 2 × 2. In other words, the query image data GD Q is a 4x4 pixel value Database image data GD DB has 2x2 pixel values.

[0283] In FIG. 24B, the query image data GD Q The 4×4 pixel values ​​of For example, the query image data GD Q In this case, one line The pixel value corresponding to the pixel in the 1st row and 4th column is pixel value vq11, and the pixel value corresponding to the pixel in the 1st row and 4th column is pixel value vq21. The pixel value vq14 corresponds to the pixel in the 4th row and the 1st column, and the pixel value vq41 corresponds to the pixel in the 4th row and the 4th column. The pixel value corresponding to the database image data GD DB There is The pixel values ​​of the 2 × 2 pixels are vdb11, vdb12, and vdb2, respectively. 1, pixel value vdb22. For example, database image data GD DB In one line The pixel value corresponding to the pixel in the first column is vdb11, and the pixel value corresponding to the pixel in the first row and second column is vdb12. The pixel value corresponding to the pixel in the second row and first column is vdb12, the pixel value corresponding to the pixel in the second row and second column is vdb21, and the pixel value corresponding to the pixel in the second row and second column is vdb22. The pixel value corresponding to the pixel of the eye is set as pixel value vdb22.

[0284] First, pixel values ​​vdb11, vdb12, vdb21, and vdb22 and the pixel value vq11, the pixel value vq12, the pixel value vq21, and the pixel value vq22. This allows the database image data GD DB and the query image data GD Q Of Consists of pixel values ​​vq11, vq12, vq21, and vq22 In FIG. 24B, the degree of match between the query image data GD Q Among the pixel values ​​of the database image data GD DB The pixel value to be compared with The data area 21 is shown surrounded by a dotted line.

[0285] Next, the comparison data area 21 is Q Move one column for the pixel values ​​of The pixel values ​​are compared in the same way to calculate the degree of coincidence. The pixel values ​​vdb12, vdb21, and vdb22, and the pixel values ​​vq12 and v q13, pixel value vq22, and pixel value vq23 are compared. Image data GD DB and the query image data GD Q Among them, pixel value vq12 and pixel value vq1 3. Calculate the degree of coincidence between the pixel value vq22 and the area consisting of the pixel value vq23. This can be done.

[0286] After that, the comparison data area 21 is also Q For each pixel value of The pixel values ​​are then moved and compared in the same way to calculate the degree of match. The pixel values ​​vdb12, vdb21, and vdb22, the pixel values ​​vq13, and The pixel value vq14, pixel value vq23, and pixel value vq24 are compared. Base image data GD DB and the query image data GD Q Among them, pixel value vq13, pixel value v q14, pixel value vq23, and the area consisting of pixel value vq24. It is possible.

[0287] Next, the comparison data area 21 is Q Move one row for the pixel value of Query image data GD Q The pixel values ​​in the second and third rows of Image data GD DB The pixel values ​​constituting the pixel are compared for each column in the same manner as above. , query image data GD Q The area consisting of the pixel values ​​in the second and third rows of Base image data GD DB The degree of agreement can be calculated for each column in the same manner as above.

[0288] Then, the comparison data area 21 is Q The pixel values ​​of Query image data GD Q The pixel values ​​in the third and fourth rows of the Image data GD DB The pixel values ​​constituting the pixel matrix are compared for each column in the same manner as above. Query image data GD Q The area consisting of the pixel values ​​in the third and fourth rows of the data Database image data GD DB The degree of agreement can be calculated for each column in the same way as above. .

[0289] After the above operations are performed, for example, the highest degree of match is selected as the database image data GD DB Noku Eri image data GD QThe above is the degree of match for n database image data GD DB Then, in the same manner as in step S31, the n database images are Image data GD DB From among them, query image data GD Q Highly consistent database images Data GD DB The extracted image data GD Ex As in step S31, Similarly, the database image data GD DB It is not necessary to extract the above.

[0290] FIG. 25 shows the database image data GD DB FIG. 25 is a diagram illustrating the extraction of , database image data GD DB [1] to database image data GD DB [3] or Then, one image data is extracted as image data GD Ex An example of extracting the following is shown.

[0291] The query image data GD shown in FIG. Q The corresponding images include, for example, transistor symbols, The symbols of the capacitor elements are included. Data GD DB The image corresponding to [2] contains the symbol for a transistor, but the database Source image data GD DB [1] and database image data GD DB [ The image corresponding to [3] includes both a transistor symbol and a capacitor symbol. In this case, the database image data GD DB [2] Query image data Data GD Q The degree of match is calculated using the database image data GD DB[1] and database Image data GD DB [3] Query image data GD Q The degree of agreement is higher than that for Database image data GD DB [2] is extracted image data GD Ex Extract as It is possible.

[0292] In addition, the query image data GD Q and database image data GD DB Comparison of and agreement The calculation can be performed using the same method as that used in step S31. In addition, in FIG. 24B, the comparison data area is set to the query image data GD Q 1 for pixel values ​​that The comparison data area 21 is moved by one column or one row at a time, as in step S31. Query image data GD Q The pixel values ​​of may be shifted by more than two columns, or by two rows. Furthermore, as in the case shown in FIG. 20A, the input to the processing unit 13 may be moved by more than one minute. Query image data GD Q Based on this, multiple query image data GD with different numbers of pixel values ​​are Q may be generated.

[0293] [Step S42] Next, the query image data GD Q From the extracted image data GD Ex Areas with high agreement with The partial image data GD part-Q The processing unit 13 extracts the following. For example, 4B, the query image data GD Q Database image data GD D B When the degree of match is calculated for each of the partial image data GD part-Q Therefore, the partial image data GD part-Q The pixel value of The number of extracted image data GD Ex can be equal to the number of pixel values ​​that .

[0294] 26 is a diagram showing an example of the operation of step S42. In the case shown in FIG. 26, the query Image data GD Q The upper left part of the image corresponding to the extracted image data GD Ex [1] This is the area with the highest degree of match. Q The upper left area of The data corresponding to the partial image data GD part-Q On the other hand, the query image data Data GD Q The lower right part of the image corresponding to the extracted image data GD Ex [2] Correspondence Therefore, the query image data GD Q It corresponds to the lower right area of The data to be processed is the partial image data GD part-Q [2]. That is, one query image Data GD Q Multiple partial image data GD part-Q is extracted.

[0295] In step S42, the extracted image data GD Ex The same number of image data as the query image data Data GD Q Partial image data GD part-Q Alternatively, the extracted image may be Data GD Ex The smaller number of image data is used as the query image data GD Q Partial image data from Ta GD part-Q For example, multiple extracted image data GD Ex Against Then, the query image data GD with high matching degree Q If the territories are the same, For the area, query image data GD Q Partial image data GD extracted from part-Q In other words, the number of the same partial image data GD part-Q of, Query image data GD Q There is no need to extract multiple items from the same table.

[0296] In addition, the partial image data GD part-Q In this case, the extraction of the partial image is not necessary. Image data GD part-Q Query image data GD accordingly Q By reading it as Alternatively, the query image data GD Q The entire image data GD part-Q For example, the query image data GD Q Reach and extracted image data GD Ex Increasing or decreasing the number of pixel values ​​contained in one or both of the image data By reducing the partial image data GD part-Q Image search system without extracting An image retrieval method can be performed using the system 10.

[0297] [Step S43] Next, the partial image data GD part-Q and extracted image data GD Ex The processing unit 13 The input is then fed into a neural network.

[0298] The operation in step S43 is to appropriately use the query image data GD Q Partial image data GD pa rt-Q Partial image data GD partExtract image data GD Ex Read as For example, the explanation of step S33 can be referred to by referring to the query image data. Ta GD Q Extract image data GD Ex Partial image data GD part The partial image Data GD part-Q It may also be read as:

[0299] The above is the database image data GD DB and the entire region of the query image data GD Q Part of By comparing the area with the database image data GD DB Query image data for Data GD Q One of the image retrieval methods using the image retrieval system 10 when calculating the similarity between Here is an example.

[0300] In the method shown in FIG. 23, the query image data GD Q and database image data GD DB and By comparing using region-based matching, query image data GD Q The degree of agreement for a part of High-quality database image data GD DB The extracted image data GD Ex It is extracted as. Then, the query image data GD Q The area with the highest degree of match is then extracted as partial image data GD pa rt-Q The partial image data GD part-Q and extracted image data GD Ex of, The data is input to the neural network of the processing unit 13. Image data GD DB By extracting the query image data GD Q and the corresponding image Database image data GD representing database images that do not contain highly matching images DB but , it is possible to prevent the information from being input to the neural network of the processing unit 13. Therefore, the query image data GD Q A database image that is similar to the part of the image corresponding to , it is possible to perform a search with high accuracy in a short time. Q Compare to Database image data GD DB If the number of images is small, the database image data GD DB The above search can be performed with high accuracy in a short time without extracting the above.

[0301] This embodiment mode can be combined with other embodiment modes as appropriate. In the case where multiple configuration examples are shown in one embodiment, the configuration examples may be combined as appropriate. It is possible to do this. [Example]

[0302] In this embodiment, an image is input to an image search system, and database images similar to the input image are searched. The results of the search will be explained below.

[0303] In this example, one image was input as a query image to the image search system. A database of 100 images similar to the image in question was searched under conditions 1 and 2. In both Condition 1 and Condition 2, the query image was a schematic diagram showing a semiconductor manufacturing device. In addition, the database images include schematic diagrams of semiconductor manufacturing equipment, as well as circuit diagrams and circuit layouts. Furthermore, the database images are based on the images published in patent documents. The drawings were made.

[0304] Under condition 1, steps S11 to S13 shown in FIG. 9 are performed, and the database image Then, step S17 is performed to calculate the similarity to the query image. Ranking data representing the 1st to 100th highest database images was generated. .

[0305] In condition 2, first, the database is Database tags from the specification published in the same patent document as the patent document in which the image was published Next, steps S11 to S13 shown in FIG. 9 are performed to obtain the database. After calculating the similarity of the image to the query image, steps S14 to S16 are performed. Then, step S17 is performed, and the similarity is corrected. Ranking data representing the first to 100th database images was generated.

[0306] In condition 1, among the database images with the 1st to 100th similarity to the query image, The 14 images represent semiconductor manufacturing equipment, similar to the query image. The remaining 86 images are circuit diagrams. The images were circuit layout diagrams, block diagrams, etc. On the other hand, in condition 2, all 100 images were The image showed semiconductor manufacturing equipment.

[0307] From the above, in condition 2, more database images with similar concepts to the query image are selected than in condition 1. It was confirmed that it was possible to easily search for information. [Explanation of symbols]

[0308] 10: Image search system, 11: Input unit, 12: Transmission path, 13: Processing unit, 15: Storage unit, 17: Database, 19: Output section, 21: Comparison data area, 30: Neural network 30a: neural network, 31: layer, 32: neuron, 32a: neuron Ron, 32b: Neuron, 32c: Neuron, 40: Neural Network

Claims

1. A database, a processing unit, and an input unit, the database has a function of storing document data and a plurality of database image data; the processing unit has a function of acquiring database image feature amount data representing feature amounts of the database image data, and then acquiring words that are candidates for database tags from the document data by morphological analysis; the processing unit has a function of acquiring a distributed representation vector by inputting the candidate words into a first neural network; the processing unit has a function of generating the same number of clusters as the number of database tags by performing clustering on the distributed representation vectors; the processing unit has a function of acquiring a database tag vector representing a representative point in the cluster and assigning a weight to the database tag vector; the weight is a value obtained by dividing the number of database word vectors included in the cluster by the total number of words that are candidates for the database tag associated with the database image data, the processing unit has a function of acquiring, when query image data is input to the input unit, query image feature amount data representing feature amounts of the query image data by a second neural network; the processing unit has a function of calculating a first similarity, which is a similarity of the database image data to the query image data, for each of the plurality of database image data; the processing unit has a function of acquiring a query tag associated with the query image data by using a part of the database tag based on the first similarity; Image search system.

2. In claim 1, An image retrieval system, wherein the first similarity and the second similarity are cosine similarities.

Images