Image retrieval system and image retrieval method

Abstract

An image retrieval system with high retrieval accuracy is provided. The present invention is an image retrieval system comprising a database and a processing unit. The database has a function of storing a plurality of database image data, each of which is associated with a database label. The processing unit has a function of obtaining, for each database image data, database image feature quantity data representing a feature quantity of the database image data. In addition, the processing unit has a function of obtaining query image feature quantity data representing a feature quantity of the query image data. Furthermore, the processing unit has a function of calculating, for each database image data, a first similarity of the database image data to the query image data. In addition, the processing unit has a function of obtaining a query label of a label associated with the query image data using a portion of the database label.

Description

Technical Field

[0001] One embodiment of the present invention relates to an image retrieval system and an image retrieval method.

[0002] Note that one embodiment of the present invention is not limited to the above-mentioned technical field. Examples of the technical field of one embodiment of the present invention include semiconductor devices, display devices, light-emitting devices, power storage devices, memory devices, electronic devices, lighting devices, and methods for driving or manufacturing the same. Background Art

[0003] By conducting prior art searches for inventions prior to filing, you can investigate whether relevant intellectual property exists. The prior art documents, such as domestic and international patent documents and papers, obtained through prior art searches can be used to confirm the novelty and inventiveness of an invention and determine whether to file a patent application. Furthermore, by conducting invalidation research on prior art documents, you can determine whether your patents have been invalidated or whether you can invalidate the patents of others.

[0004] For example, the prior art search can be performed by searching for prior art documents that contain drawings similar to drawings that embody prior art. Specifically, for example, a user can input drawings into an image search system to retrieve prior art documents that contain drawings similar to the input drawings.

[0005] Images similar to the input image can be retrieved using, for example, a neural network. For example, Patent Document 1 discloses a method for determining the similarity between images using a neural network.

[0006] [Prior technical literature]

[0007] [Patent Document]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2017-207947 Summary of the Invention

[0009] Technical problem to be solved by the invention

[0010] When calculating the similarity between an input image and the search target image using only image data, images with different concepts from the input image may be retrieved. This can lead to noise images being mixed into the search results, and the desired image not being output. Consequently, the accuracy of similar image retrieval may decrease.

[0011] One object of one embodiment of the present invention is to provide an image retrieval system with high retrieval accuracy. Another object of one embodiment of the present invention is to provide an image retrieval system capable of performing retrieval in a short time. Another object of one embodiment of the present invention is to provide an image retrieval system capable of performing retrieval in a simple manner. Another object of one embodiment of the present invention is to provide a novel image retrieval system.

[0012] One object of one embodiment of the present invention is to provide an image retrieval method with high retrieval accuracy. Another object of one embodiment of the present invention is to provide an image retrieval method capable of performing retrieval in a short time. Another object of one embodiment of the present invention is to provide an image retrieval method capable of performing retrieval in a simple manner. Another object of one embodiment of the present invention is to provide a novel image retrieval method.

[0013] Note that the inclusion of the above-mentioned objectives does not preclude the existence of other objectives. One embodiment of the present invention does not necessarily achieve all of the above-mentioned objectives. Objectives other than the above-mentioned objectives may be extracted from the description, drawings, and claims.

[0014] Means of solving technical problems

[0015] One embodiment of the present invention is an image retrieval system including: a database; a processing unit; and an input unit, wherein the database has a function of storing document data and a plurality of database image data; the processing unit has a function of obtaining database image feature data representing a feature of each of the plurality of database image data; the processing unit has a function of generating a plurality of database labels using the document data and associating the database labels with the database image data; the processing unit has a function of obtaining a database label vector representing the database label for each of the plurality of database labels; the processing unit has a function of obtaining query image feature data representing a feature of the query image data when query image data is input to the input unit; the processing unit has a function of calculating a first similarity of the database image data to the query image data for each of the plurality of database image data; the processing unit has a function of obtaining a query label associated with the query image data using a portion of the database label based on the first similarity; the processing unit has a function of obtaining a query label vector representing the query label; the processing unit has a function of obtaining first data including the database image feature data and the database label vector; the processing unit has a function of obtaining second data including the query image feature data and the query label vector; and the processing unit has a function of calculating a second similarity of the first data to the second data.

[0016] In the above manner, database tags may also include words.

[0017] In the above-mentioned embodiment, the processing unit may have a function of generating a database tag by performing a morphological analysis on the document data.

[0018] In the above embodiment, the processing unit may include a first neural network and a second neural network, the database image feature data and the query image feature data are obtained using the first neural network, and the database label vector and the query label vector are obtained using the second neural network.

[0019] In the above manner, the first neural network may also include a convolutional layer and a pooling layer, and the database image feature data and the query image feature data are output from the pooling layer.

[0020] In the above method, the database label vector and the query label vector may also be distribution representation vectors.

[0021] In the above embodiment, the first similarity and the second similarity may also be cosine similarities.

[0022] One embodiment of the present invention is an image retrieval method, which uses an image retrieval system including file data, a database storing multiple database images, and an input unit, wherein database image feature data representing feature quantities of the database image data is obtained for each of the multiple database image data, multiple database tags are generated using the file data to associate the database tags with the database image data, a database tag vector representing the database tag for each of the multiple database tags is obtained, query image data is input into the input unit, query image feature data representing feature quantities of the query image data is obtained, a first similarity of the database image data to the query image data is calculated for each of the multiple database image data, a query tag associated with the query image data is obtained using a portion of the database tag based on the first similarity, a query tag vector representing the query tag is obtained, first data including the database image feature data and the database tag vector and second data including the query image feature data and the query tag vector are obtained, and a second similarity of the first data to the second data is calculated.

[0023] In the above manner, database tags may also include words.

[0024] In the above method, database tags can also be generated by performing morphological analysis on the file data.

[0025] In the above-described method, the database image feature data and the query image feature data may be obtained using the first neural network, and the database label vector and the query label vector may be obtained using the second neural network.

[0026] In the above method, the first neural network includes a convolution layer and a pooling layer, and the database image feature data and the query image feature data can also be output from the pooling layer.

[0027] In the above method, the database label vector and the query label vector may also be distribution representation vectors.

[0028] In the above embodiment, the first similarity and the second similarity may also be cosine similarities.

[0029] Effects of the Invention

[0030] According to one embodiment of the present invention, an image retrieval system with high retrieval accuracy can be provided. According to one embodiment of the present invention, an image retrieval system capable of performing retrieval in a short time can be provided. According to one embodiment of the present invention, an image retrieval system capable of performing retrieval in a simple manner can be provided. According to one embodiment of the present invention, a novel image retrieval system can be provided.

[0031] According to one embodiment of the present invention, a method for image retrieval with high retrieval accuracy can be provided. According to one embodiment of the present invention, a method for image retrieval capable of performing retrieval in a short time can be provided. According to one embodiment of the present invention, a method for image retrieval capable of performing retrieval in a simple manner can be provided. According to one embodiment of the present invention, a novel method for image retrieval can be provided.

[0032] Note that the description of these effects does not preclude the existence of other effects. One embodiment of the present invention does not necessarily have all of the above effects. Effects other than the above effects can be extracted from the description of the specification, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a block diagram showing a structural example of an image retrieval system.

[0034] Figure 2 This is a flowchart showing an example of a method for generating search data.

[0035] Figure 3A and Figure 3B is a diagram showing a structural example of a neural network.

[0036] Figure 4 A diagram showing an example of convolution processing and pooling processing.

[0037] Figure 5 is a diagram showing a structural example of a neural network.

[0038] Figure 6A and Figure 6B This is a diagram showing an example of a method for generating search data.

[0039] Figure 7A is a diagram showing an example of a method for generating search data. Figure 7B is a diagram showing a structural example of a neural network.

[0040] Figure 8A and Figure 8B This is a diagram showing an example of a method for generating search data.

[0041] Figure 9 This is a flowchart showing an example of an image retrieval method.

[0042] Figure 10 This is a diagram showing an example of an image retrieval method.

[0043] Figure 11A and Figure 11B This is a diagram showing an example of an image retrieval method.

[0044] Figure 12A and Figure 12B This is a diagram showing an example of an image retrieval method.

[0045] Figure 13 This is a diagram showing an example of an image retrieval method.

[0046] Figure 14 This is a flowchart showing an example of an image retrieval method.

[0047] Figure 15 This is a diagram showing an example of an image retrieval method.

[0048] Figure 16A and Figure 16B This is a diagram showing an example of an image retrieval method.

[0049] Figure 17 This is a flowchart showing an example of an image retrieval method.

[0050] Figure 18A and Figure 18B This is a diagram showing an example of an image retrieval method.

[0051] Figure 19 This is a diagram showing an example of an image retrieval method.

[0052] Figure 20A 、 Figure 20B1 and Figure 20B2 This is a diagram showing an example of an image retrieval method.

[0053] Figure 21A and Figure 21B This is a diagram showing an example of an image retrieval method.

[0054] Figure 22A and Figure 22B This is a diagram showing an example of an image retrieval method.

[0055] Figure 23 This is a flowchart showing an example of an image retrieval method.

[0056] Figure 24A and Figure 24B This is a diagram showing an example of an image retrieval method.

[0057] Figure 25 This is a diagram showing an example of an image retrieval method.

[0058] Figure 26 This is a diagram showing an example of an image retrieval method. DETAILED DESCRIPTION

[0059] The embodiments are described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following description. A person skilled in the art will readily appreciate that the embodiments and details can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited to the embodiments described below.

[0060] (Implementation 1)

[0061] In this embodiment, an image retrieval system and an image retrieval method according to one embodiment of the present invention will be described with reference to the drawings.

[0062] An image retrieval system according to one embodiment of the present invention includes an input unit, a database, and a processing unit. The processing unit includes a first neural network and a second neural network. The first and second neural networks are provided with layers including neurons.

[0063] In this specification and other documents, a neural network refers to any model that simulates a biological neural circuit network and determines the strength of connections between neurons through learning, thereby having problem-solving capabilities.

[0064] In this specification and other descriptions of a neural network, determining the connection strength (also called weight coefficient) between neurons based on existing information may be referred to as "learning."

[0065] In this specification and other documents, constructing a neural network using the connection strength obtained through learning and deriving a new conclusion from this structure may be referred to as "inference."

[0066] The database stores image data. When a user of the image retrieval system according to one embodiment of the present invention inputs image data into the input unit, the image retrieval system according to one embodiment of the present invention retrieves and outputs image data similar to the input image data from the database.

[0067] In this specification, etc., image data stored in a database is referred to as database image data. Furthermore, image data input to an input unit is referred to as query image data. Furthermore, database image data and query image data may be collectively referred to as image data.

[0068] Hereinafter, an image retrieval method using the image retrieval system according to one embodiment of the present invention will be described.

[0069] Image feature data can be acquired by inputting image data into the first neural network included in the processing unit.

[0070] In this specification, data representing features of image data is referred to as image feature data. For example, data representing features of database image data is referred to as database image feature data, and data representing features of query image data is referred to as query image feature data.

[0071] The first neural network may be, for example, a convolutional neural network including a convolutional layer and a pooling layer. When a convolutional neural network is used as the first neural network, the image data input to the first neural network and the data output from the pooling layer may be image feature data.

[0072] Database image data is associated with tags. For example, file data associated with the database image data is stored in the database and morphological analysis is performed on the file data to associate tags. Tags can be keywords representing concepts, technical content, or key points of interest associated with the database image data. For example, a tag can represent a single word. Database image data can be associated with multiple tags.

[0073] In this specification, etc., a label associated with database image data is referred to as a database label. Also, a label associated with query image data is referred to as a query label.

[0074] By inputting the label into the second neural network included in the processing unit, the label can be represented as a vector. For example, the label can be represented as a 300-dimensional distribution representation vector.

[0075] In this specification, a vector representing a label is referred to as a label vector. For example, a vector representing a database label is referred to as a database label vector, and a vector representing a query label is referred to as a query label vector. Furthermore, a label vector refers to a label vector corresponding to a label.

[0076] In this specification, the word "vector" refers to a collection of multiple values. Furthermore, the number of values ​​that make up a vector is called its dimension. For example, a vector represented by (5, 1, 4, 3, 2) is considered a 5-dimensional vector. Note that the values ​​that make up a vector are sometimes referred to as components.

[0077] In an image retrieval method using an image retrieval system according to one embodiment of the present invention, in addition to the database image data, database image feature data representing the features of the database image is pre-stored in the database. Furthermore, database labels associated with the database image data and database label vectors representing the database labels are pre-stored in the database. Note that the database labels themselves do not necessarily need to be stored in the database.

[0078] In an image retrieval method using an image retrieval system according to one embodiment of the present invention, when a user of the image retrieval system inputs query image data into an input unit, the query image data is input into a first neural network, and query image feature data is generated. Subsequently, the similarity of the database image data to the query image data is calculated using the database image feature data and the query image feature data. For example, cosine similarity is calculated. This calculation of similarity to the query image data can be performed for each of all database image data, for example.

[0079] Then, based on the similarity calculation results, the database tags are used to obtain a query tag. For example, the database tag that appears most frequently among the database tags associated with the database image data with high similarity can be used as the query tag. Note that the number of query tags can be equal to the number of database tags associated with a single database image data item.

[0080] In this specification and the like, one piece of image data refers to image data of one image displayed during one frame period, for example.

[0081] Next, first data comprising database image feature data and a database label vector is obtained. Furthermore, second data comprising query image feature data and a query label vector is obtained. Then, by calculating the similarity between the first and second data, the similarity of the database image data to the query image data is corrected. For example, this correction is performed by calculating the cosine similarity between the first and second data.

[0082] Here, a first data set may, for example, include database image feature data and a database label vector corresponding to a database label, wherein the database label is associated with the database image data corresponding to the database image feature data. In other words, the number of first data sets is equal to the number of database image feature data sets. Furthermore, a second data set may include query image feature data and a number of query label vectors equal to the number of database label vectors included in the first data set.

[0083] Next, ranking data including information regarding the ranking of the corrected similarities is generated and output as a search result to the outside of the image retrieval system of one embodiment of the present invention. By correcting the similarity of the database image data to the query image data in this manner, for example, database images with similar features to the query image but different concepts can be prevented from being retrieved. This prevents images from being mixed with noise in the search results and prevents the output of the desired image. Consequently, the image retrieval system of one embodiment of the present invention can perform searches with high accuracy.

[0084] In an image retrieval method using an image retrieval system according to one embodiment of the present invention, query tags are obtained using database tags. This acquisition method is simpler than, for example, a method of acquiring query tags based on query image feature data. Therefore, the image retrieval system according to one embodiment of the present invention can perform retrieval in a shorter time. Furthermore, compared to, for example, a method in which a user of the image retrieval system according to one embodiment of the present invention specifies all query tags, the method of acquiring query tags using database tags can comprehensively acquire tags representing the concepts, technical content, and points of interest of images corresponding to the query image data. Therefore, the image retrieval system according to one embodiment of the present invention can perform retrieval simply and with high accuracy.

[0085] <1-1. Image Retrieval System>

[0086] Figure 1 This is a block diagram illustrating an example configuration of the image retrieval system 10. The block diagrams in this specification illustrate components categorized by function within separate blocks. However, in practice, components are difficult to clearly separate by function, and a single component may have multiple functions. Furthermore, a single function may involve multiple components. For example, multiple processes performed in the processing unit 13 may be performed on different servers.

[0087] The image retrieval system 10 includes at least a processing unit 13 . Figure 1 The image retrieval system 10 shown further includes an input unit 11 , a transmission channel 12 , a storage unit 15 , a database 17 , and an output unit 19 .

[0088] [Input unit 11]

[0089] Image data and the like are supplied from outside the image retrieval system 10 to the input unit 11. The image data and the like supplied to the input unit 11 are supplied to the processing unit 13, the storage unit 15, or the database 17 via the transmission channel 12. As described above, the image data input to the input unit 11 is referred to as query image data.

[0090] [Transmission Channel 12]

[0091] The transmission channel 12 has a function of transmitting image data, etc. Information can be sent and received between the input unit 11 , the processing unit 13 , the storage unit 15 , the database 17 , and the output unit 19 via the transmission channel 12 .

[0092] [Processing unit 13]

[0093] The processing unit 13 has the function of performing calculations, inferences, and the like using image data supplied from the input unit 11, the storage unit 15, the database 17, and the like. The processing unit 13 includes a neural network and can perform calculations, inferences, and the like using this neural network. Furthermore, the processing unit 13 can perform calculations, inferences, and the like without using a neural network. The processing unit 13 can supply calculation results, inference results, and the like to the storage unit 15, the database 17, the output unit 19, and the like.

[0094] A transistor containing a metal oxide in the channel formation region is preferably used in the processing unit 13. Since the off-state current of this transistor is extremely low, it can be used as a switch to retain the charge (data) flowing into the capacitor used as a storage element, thereby ensuring a long data retention period. By using this transistor in at least one of the registers and cache memory included in the processing unit 13, the processing unit 13 can be operated only when necessary, while previously processed information is stored in the storage element, allowing the processing unit 13 to be shut down. In other cases, normally off computing can be achieved, thereby achieving low power consumption of the image retrieval system.

[0095] In this specification, etc., metal oxide refers to an oxide of a metal in a broad sense. Metal oxides are classified into oxide insulators, oxide conductors (including transparent oxide conductors) and oxide semiconductors (Oxide Semiconductor, which may also be referred to as OS). For example, when a metal oxide is used for a semiconductor layer of a transistor, the metal oxide is sometimes referred to as an oxide semiconductor. In other words, when a metal oxide has at least one of an amplifying effect, a rectifying effect and a switching effect, the metal oxide may be referred to as a metal oxide semiconductor (metal oxide semiconductor), or may be abbreviated as OS.

[0096] Note that in this specification and the like, a transistor using an oxide semiconductor or a metal oxide in a channel formation region is referred to as an oxide semiconductor transistor or an OS transistor.

[0097] The metal oxide possessed by the channel formation region preferably contains indium (In). When the metal oxide possessed by the channel formation region contains indium, the carrier mobility (electron mobility) of the OS transistor is improved. In addition, the metal oxide possessed by the channel formation region is preferably an oxide semiconductor containing element M. Element M is preferably aluminum (Al), gallium (Ga) or tin (Sn) or the like. Other elements that can be used as element M include boron (B), silicon (Si), titanium (Ti), iron (Fe), nickel (Ni), germanium (Ge), yttrium (Y), zirconium (Zr), molybdenum (Mo), lanthanum (La), cerium (Ce), neodymium (Nd), hafnium (Hf), tantalum (Ta), tungsten (W) or the like. Note that as element M, multiple of the above elements can sometimes be combined. Element M is, for example, an element with a high bond energy with oxygen. Element M is, for example, an element with a higher bond energy with oxygen than indium. In addition, the metal oxide possessed by the channel formation region preferably contains zinc (Zn). Metal oxides containing zinc are sometimes easy to crystallize.

[0098] The metal oxide contained in the channel formation region is not limited to a metal oxide containing indium. The semiconductor layer may be a metal oxide containing zinc, gallium, or tin without containing indium, such as zinc tin oxide or gallium tin oxide.

[0099] The processing unit 13 includes, for example, a calculation circuit or a central processing unit (CPU).

[0100] The processing unit 13 may also include a microprocessor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The microprocessor may also be implemented as a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array). The processing unit 13 can perform various data processing and program control by interpreting and executing instructions from various programs through the processor. Programs executable by the processor are stored in at least one of a memory area included in the processor and the storage unit 15.

[0101] The processing unit 13 may include a main memory. The main memory includes at least one of a volatile memory such as a RAM (Random Access Memory) and a non-volatile memory such as a ROM (Read Only Memory).

[0102] RAM, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory), is used. Virtual memory space is allocated to this RAM as a workspace for the processing unit 13 and is used by the processing unit 13. The operating system, application programs, program modules, program data, and lookup tables stored in the storage unit 15 are loaded into the RAM during execution. The processing unit 13 directly accesses and operates these data, programs, and program modules loaded into the RAM.

[0103] ROM can store non-rewritable BIOS (Basic Input / Output System) and firmware. Examples of ROM include mask ROM, OTPROM (One Time Programmable Read Only Memory), and EPROM (Erasable Programmable Read Only Memory). Examples of EPROM include UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory), which can erase stored data by ultraviolet light, EEPROM (Electrically Erasable Programmable Read Only Memory), and flash memory.

[0104] [Storage unit 15]

[0105] The storage unit 15 has a function of storing programs executed by the processing unit 13. The storage unit 15 may also have a function of storing calculation results and inference results generated by the processing unit 13, image data input to the input unit 11, and the like.

[0106] The storage unit 15 includes at least one of a volatile memory and a non-volatile memory. The storage unit 15 may include, for example, a volatile memory such as DRAM or SRAM. The storage unit 15 may also include, for example, a non-volatile memory such as ReRAM (Resistive Random Access Memory, also known as resistance random access memory), PRAM (Phase change Random Access Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Random Access Memory, also known as magnetoresistive random access memory), or flash memory. In addition, the storage unit 15 may also include a recording medium drive such as a hard disk drive (HDD) or a solid state drive (SSD).

[0107] [Database 17]

[0108] The database 17 has the function of storing image data of search targets. As described above, image data stored in the database is referred to as database image data. Furthermore, the database 17 has the function of storing the calculation results and inference results generated by the processing unit 13. The database 17 may also have the function of storing image data input to the input unit 11. Note that the storage unit 15 and the database 17 do not need to be separate. For example, the image retrieval system 10 may include a storage unit that performs the functions of both the storage unit 15 and the database 17.

[0109] [Output unit 19]

[0110] The output unit 19 has a function of supplying information to the outside of the image retrieval system 10. For example, the calculation results or inference results in the processing unit 13 can be supplied to the outside.

[0111] <1-2. Image Retrieval Method-1>

[0112] First, the processing performed in advance for performing a search using the image search system 10 will be described. Figure 2 This is a flowchart showing an example of the method of this processing.

[0113] [Step S01]

[0114] First, the database image data GD DB The database image data GD is input to the processing unit 13 through the transmission channel 12. DBThis can be data representing drawings included in information representing intellectual property. Examples of intellectual property information include patent documents (publication of patents, patent gazettes, etc.), utility model gazettes, design gazettes, and publications such as theses. This information is not limited to publications published domestically; publications published worldwide can also be used as intellectual property information.

[0115] Intellectual property information is not limited to publications. For example, various files such as image files owned by users of image retrieval systems or groups can be used as database image data GD. DB Furthermore, examples of intellectual property information include drawings that illustrate inventions, designs, or patterns.

[0116] Database image data GD DB For example, the data may include data showing drawings described in patent documents of a specific applicant or data showing drawings described in patent documents in a specific technical field.

[0117] The image retrieval system 10 has a function of retrieving database image data GD similar to the query image data. DB This function allows the use of the image search system 10, for example, to search for patent documents, papers, or industrial products similar to the invention before the application. This allows for prior art research related to the invention before the application. By understanding relevant prior art and reconsidering it, the invention can be strengthened and become a powerful patent that other companies cannot avoid.

[0118] Furthermore, by using the image retrieval system 10, for example, patent documents, papers, or industrial products similar to the industrial products before they are sold can be retrieved. DB If there is data corresponding to an image described in the patent document of the company, it is possible to confirm whether the technology of the industrial product before sale is sufficient for the patent application in the company. DB By having data corresponding to images described in other companies' patent documents, you can verify whether pre-sale industrial products infringe on other companies' intellectual property rights. By reviewing relevant prior art, you can discover novel inventions and create powerful patents that contribute to your company's business. Note that this search is not limited to pre-sale industrial products, but also includes post-sale industrial products.

[0119] For example, patent documents, papers, or industrial products similar to a specific patent can be retrieved using the image search system 10. In particular, by searching based on the patent's application date, it is possible to easily and accurately determine whether the patent has grounds for invalidation.

[0120] [Step S02]

[0121] Next, the database image data GD DB The data is input to the neural network included in the processing unit 13 .

[0122] Figure 3A 1 is a diagram showing a configuration example of a neural network 30 included in the processing unit 13. The neural network 30 includes layers 31[1] to 31[m] (m is an integer greater than or equal to 1).

[0123] Layers 31[1] to 31[m] include neurons, and the neurons in each layer are connected to each other. For example, the neurons in layer 31[1] are connected to the neurons in layer 31[2]. In addition, the neurons in layer 31[2] are connected to the neurons in layer 31[1] and the neurons in layer 31[3]. In other words, layers 31[1] to 31[m] constitute a hierarchical neural network.

[0124] Database image data GD DB Input is given to layer 31[1], and layer 31[1] outputs data corresponding to the input image data. This data is input to layer 31[2], and layer 31[2] outputs data corresponding to the input data. Data output from layer 31[m-1] is input to layer 31[m], and layer 31[m] outputs data corresponding to the input data. As described above, layer 31[1] can be an input layer, layers 31[2] to 31[m-1] can be intermediate layers, and layer 31[m] can be an output layer.

[0125] The neural network 30 is pre-learned, for example, so that the data output from layers 31[1] to 31[m] represents the feature quantity of the image data input to the neural network 30. Learning can be performed using unsupervised learning, supervised learning, or the like. Unsupervised learning is particularly preferred because it does not require supervised data (also known as correct labeling). Furthermore, regardless of whether learning is performed using unsupervised or supervised learning, a backpropagation algorithm or the like can be used as the learning algorithm.

[0126] Here, it is preferable to use the database image data GD as the learning data. DB Thus, for example, the data output from the layers 31[1] to 31[m] can accurately represent the feature quantity of the image data input to the neural network 30. For example, by combining all the database image data GD stored in the database 17 DB For learning data, the neural network 30 can perform learning. For example, by DB A portion of the data is used for learning data, and the neural network 30 can learn. For example, in addition to the database image data GD DBIn addition, the neural network 30 can perform learning by using image data stored in the storage unit 15 and image data input to the processing unit 13 from outside the image retrieval system 10 through the input unit 11 as learning data.

[0127] Note that the database image data GD may not be used as learning data. DB For example, the neural network 30 can perform learning by using only image data input to the processing unit 13 from outside the image retrieval system 10 through the input unit 11 as learning data.

[0128] The neural network 30 may be a convolutional neural network (CNN). Figure 3B 1 is a diagram showing a configuration example of the neural network 30 when a CNN is used as the neural network 30. Here, the neural network 30 using the CNN is a neural network 30a.

[0129] The neural network 30a includes a convolutional layer CL, a pooling layer PL and a fully connected layer FCL. Figure 3B In the example, the neural network 30a includes m layers (m is an integer greater than 1) of convolutional layers CL and m layers of pooling layers PL, and includes one fully connected layer FCL. Note that the neural network 30a may also include two or more fully connected layers FCL.

[0130] The convolutional layer CL has the function of convolving the data input to the convolutional layer CL. For example, the convolutional layer CL[1] has the function of convolving the image data input to the processing unit 13. In addition, the convolutional layer CL[2] has the function of convolving the data output from the pooling layer PL[1]. In addition, the convolutional layer CL[m] has the function of convolving the data output from the pooling layer PL[m-1].

[0131] Convolution is performed by repeatedly performing product-sum operations on the data input to the convolution layer CL and the weighted filter. By performing convolution in the convolution layer CL, features of the image corresponding to the image data input to the neural network 30a are extracted.

[0132] After the convolutional data is transformed using an activation function, it is output to the pooling layer PL. ReLU (Rectified Linear Units) and other activation functions can be used. ReLU is a function that outputs "0" for negative input values ​​and outputs the input value as is for values ​​above "0." Other activation functions include the sigmoid function and the tanh function.

[0133] The pooling layer PL pools the data input from the convolutional layer CL. Pooling involves dividing the data into multiple regions, extracting specific data from each region, and arranging the data in a matrix. Pooling reduces the amount of data while preserving the features extracted by the convolutional layer CL. It also improves robustness to slight variations in the input data. Possible pooling methods include max pooling, average pooling, and Lp pooling.

[0134] The fully connected layer (FCL) uses the data output from the pooling layer (PL[m]) to determine the image. The fully connected layer (FCL) has a structure in which all nodes in one layer are connected to all nodes in the next layer. The data output from the convolutional layer (CL) or the pooling layer (PL) is a two-dimensional feature map, which is expanded one-dimensionally when input to the fully connected layer (FCL). Furthermore, the fully connected layer (FCL) outputs a vector derived from the inference of the fully connected layer (FCL).

[0135] Note that the structure of the neural network 30a is not limited to Figure 3B For example, a pooling layer PL may be provided for each plurality of convolutional layers CL. In other words, the number of pooling layers PL included in the neural network 30a may be smaller than the number of convolutional layers CL. In addition, when it is desired to preserve the position information of the extracted features as much as possible, the pooling layer PL may not be provided.

[0136] The neural network 30a can optimize the filter values ​​of the weight filters, the weight coefficients of the fully connected layer FCL, and the like through learning.

[0137] Next, refer to Figure 4 An example of the convolution processing performed in the convolution layer CL and the pooling processing performed in the pooling layer PL is described. Figure 4 In the example, the data input to the convolutional layer CL has 3 rows and 3 columns of input data values ​​(input data value i11, input data value i12, input data value i13, input data value i21, input data value i22, input data value i23, input data value i31, input data value i32, input data value i33). In addition, the weight filter has 2 rows and 2 columns of filter values ​​(filter value f11, filter value f12, filter value f21, filter value f22).

[0138] Here, for example, the data input to the convolutional layer CL[1] may be image data. In this case, the input data value may be a pixel value in the image data.

[0139] In this specification and other documents, a pixel value represents the grayscale value of the brightness of light emitted by a pixel. For example, if the pixel value is an 8-bit value, the pixel can emit light with 256 grayscale brightness levels. Image data can have a set of pixel values, for example, it can have the same pixel value as the pixel. For example, if the number of pixels in an image is 2×2, the image data representing the image can also be said to have 2×2 pixel values.

[0140] For example, the input data value input to the convolution layer CL[2] may be the output value of the pooling layer PC[1], and the input data value input to the convolution layer CL[m] may be the output value of the pooling layer PC[m-1].

[0141] Convolution is performed by performing a product-sum operation on the input data value and the filter value. The filter value may be data indicating a predetermined characteristic (also referred to as feature data). By comparing the input data value and the filter value, features can be extracted from the image data input to the neural network 30a.

[0142] Figure 4 It is shown that the convolution layer CL obtains the convolution value C11 included in the data output from the convolution layer CL by filtering the input data value i11, the input data value i12, the input data value i21, and the input data value i22. In addition, it is shown that the convolution layer CL obtains the convolution value C12 included in the data output from the convolution layer CL by filtering the input data value i12, the input data value i13, the input data value i22, and the input data value i23. In addition, it is shown that the convolution layer CL obtains the convolution value C21 included in the data output from the convolution layer CL by filtering the input data value i21, the input data value i22, the input data value i31, and the input data value i32. Furthermore, it is shown that the convolution layer CL obtains the convolution value C22 included in the data output from the convolution layer CL by filtering the input data value i22, the input data value i23, the input data value i32, and the input data value i33. Thus, Figure 4 The convolution process shown may have a step size of 1.

[0143] The convolution value C11, the convolution value C12, the convolution value C21, and the convolution value C22 can be obtained by performing a product-sum operation as shown in the following formula.

[0144] (Formula 1)

[0145] C11=i11·f11+i12·f12+i21·f21+i22·f22 (1)

[0146] (Formula 2)

[0147] C12=i12·f11+i13·f12+i22·f21+i23·f22 (2)

[0148] (Formula 3)

[0149] C21=i21·f11+i22·f12+i31·f21+i32·f22 (3)

[0150] (Formula 4)

[0151] C22=i22·f11+i23·f12+i32·f21+i33·f22 (4)

[0152] The convolution values ​​C11, C12, C21, and C22 obtained by the convolution layer CL are arranged in a matrix according to the address and then output to the pooling layer PL. Specifically, the convolution value C11 is arranged in the first row and first column, the convolution value C12 is arranged in the first row and second column, the convolution value C21 is arranged in the second row and first column, and the convolution value C22 is arranged in the second row and second column.

[0153] Figure 4 The figure shows how convolution values ​​C11, C12, C21, and C22 are input to the pooling layer PL, and a single value is set as the pooling value P based on these four convolution values. For example, the maximum value among the convolution values ​​C11, C12, C21, and C22 can be the pooling value P. Alternatively, the average value of the convolution values ​​C11, C12, C21, and C22 can be the pooling value P. The pooling value P becomes the output value of the pooling layer PL.

[0154] Figure 4 The example of processing the data input to the convolution layer CL by one weight filter is shown, but it is also possible to process it by two or more weight filters. In this case, multiple features included in the image data input to the neural network 30a can be extracted. When processing the data input to the convolution layer CL by two or more weight filters, each filter is processed. Figure 4 In addition, as mentioned above, Figure 4 The step length is set to 1, and the step length can be set to 2 or more.

[0155] Figure 5 3 is a diagram showing a structural example of a convolutional layer CL and a pooling layer PL included in the neural network 30a. Figure 5 Shows the convolution layer CL and pooling layer PL Figure 4 Examples of work shown.

[0156] Figure 5The neuron 32 is shown. Specifically, the neuron 32a, the neuron 32b, and the neuron 32c are shown as the neuron 32. Figure 5 In , the value output from the neuron 32 is recorded inside the neuron 32. The value is output in the direction of the arrow. In addition, when the value is multiplied by the weight coefficient, the weight coefficient is recorded near the arrow. Figure 5 In the filter value f11, filter value f12, filter value f21 and filter value f22 are weight coefficients.

[0157] Neuron 32a is Figure 5 The layer L of the previous layer of the convolutional layer CL shown includes neurons 32. Layer L is for example Figure 5 The convolutional layer CL shown is the convolutional layer CL[1], which can be the input layer. Figure 5 The convolutional layer CL shown is the convolutional layer CL[2], which can be the pooling layer PL[1]. Figure 5 The convolutional layer CL shown is the convolutional layer CL[m] and may be the pooling layer PL[m-1].

[0158] exist Figure 5 In FIG, neurons 32a[1] to 32a[9] are shown as neurons 32a. Figure 5 In the case shown, neuron 32a[1] outputs input data value i11, neuron 32a[2] outputs input data value i12, neuron 32a[3] outputs input data value i13, neuron 32a[4] outputs input data value i21, neuron 32a[5] outputs input data value i22, neuron 32a[6] outputs input data value i23, neuron 32a[7] outputs input data value i31, neuron 32a[8] outputs input data value i32, and neuron 32a[9] outputs input data value i33.

[0159] Neuron 32b is Figure 5 The convolutional layer CL shown includes neurons 32. Figure 5 , neurons 32b[1] to 32b[4] are shown as neurons 32b.

[0160] exist Figure 5 In the case shown, the values ​​of input data value i11 multiplied by filter value f11, input data value i12 multiplied by filter value f12, input data value i21 multiplied by filter value f21, and input data value i22 multiplied by filter value f22 are input to neuron 32b[1]. The convolution value C11, which is the sum of these values, is output from neuron 32b[1].

[0161] The value of multiplying the filter value f11 by the input data value i12, the value of multiplying the filter value f12 by the input data value i13, the value of multiplying the filter value f21 by the input data value i22, and the value of multiplying the filter value f22 by the input data value i23 are input to neuron 32b[2]. The convolution value C12, which is the sum of these values, is output from neuron 32b[2].

[0162] The value of multiplying the input data value i21 by the filter value f11, the value of multiplying the input data value i22 by the filter value f12, the value of multiplying the input data value i31 by the filter value f21, and the value of multiplying the input data value i32 by the filter value f22 are input to neuron 32b[3]. The convolution value C21, which is the sum of these values, is output from neuron 32b[3].

[0163] Furthermore, the value of multiplying the input data value i22 by the filter value f11, the value of multiplying the input data value i23 by the filter value f12, the value of multiplying the input data value i32 by the filter value f21, and the value of multiplying the input data value i33 by the filter value f22 are input to neuron 32b[4]. The convolution value C22, which is the sum of these values, is output from neuron 32b[4].

[0164] like Figure 5 As shown, each of neurons 32b[1] to 32b[4] is connected to a portion of neurons 32a[1] to 32a[9]. Therefore, the convolutional layer CL can be said to be a partially connected layer.

[0165] Neuron 32c is Figure 5 The pooling layer PL shown includes neurons 32. Figure 5 In the case shown, convolution values ​​C11, C12, C21, and C22 are input to neuron 32c. Furthermore, pooling value P is output from neuron 32c. Note that the convolution value output from neuron 32b is not multiplied by the weight coefficient. As described above, the weight coefficient is a parameter optimized through learning in the neural network. Therefore, a configuration is possible in which the parameters used in the calculation of pooling layer PL do not have parameters optimized through learning.

[0166] As described above, by converting the database image data GD DB Input to the neural network 30, the processing unit 13 can obtain the database image data GD DB The feature quantity database image feature quantity data GFD DB For example, Figure 3A As shown, the data output from layer 31[m] can be the database image feature data GFD DB Or, as Figure 3B As shown, the data output from the pooling layer PL[m] can be the database image feature data GFD DBNote that the database image feature data GFD DB It can also have more than 2 layers of output data. DB Output data with more layers, database image feature data GFD DB Can more accurately represent the database image data GD DB The database image feature data GFD obtained by the processing unit 13 DB It can be stored in the database 17.

[0167] [Step S03]

[0168] Next, the processing unit 13 obtains the database image data GD DB Related database tag TAG DB . Preferably, the image data GD corresponding to the database is represented DB The concept, technical content, focus, etc. of the image become the database tag TAG DB Get the database tag TAG DB . Figure 6A It shows that the database tag TAG is obtained DB An example of the method is shown in Figure 1. Note that Figure 6A The diagrams of each data shown are merely examples and are not limited thereto. In addition, the diagrams of each data, vectors, etc. shown in other drawings are merely examples and are not limited to the contents of the diagrams.

[0169] In this specification, etc., when the same reference numeral is used for a plurality of elements and it is necessary to distinguish them, a reference numeral such as “[1]” or “[2]” may be added to the reference numeral for identification.

[0170] exist Figure 6A In the method shown, as an example, the database image data GD DB [1] To database image data GD DB Each of

[100] is associated with a label. In addition, corresponding to the database image data GD DB File data TD DB It is stored in the database 17 in advance. DB Associated with the figure number.

[0171] File Data TD DB For example, it can be recorded in the database image data GD DB The data corresponding to the documents recorded in the patent documents, utility model publications, design publications and papers of the drawings shown. For example, in the database image data GD DBWhen the publication of the drawings is a patent document or utility model gazette, the data corresponding to the specification may be the document data TD DB Alternatively, the data corresponding to the claims, utility model claims or abstracts may be document data TD DB In addition, in the database image data GD DB When the publication recorded is a design gazette, the data corresponding to the application form may be the document data TD DB .

[0172] For example, in the file data TD DB For data corresponding to manuals or papers, the database tag TAG DB The image data GD can be described by DB The segments of the figures shown are obtained by morphological analysis. Figure 6A In the figure, the image data corresponding to the database GD is shown. DB The image number of [1] is " Figure 1 ", in the database image data GD DB [1] Associated file data TD DB The paragraph [0xx0] of the document represented by [1] contains " Figure 1 Therefore, for example, it is considered that the database image data GD is recorded in paragraph [0xx0] DB [1] The description of the accompanying figure shows that the database tag TAG can be obtained by performing a morphological analysis on the article recorded in paragraph [0xx0] DB [1]. In addition, Figure 6A In FIG. 1 , the image data GD is shown. DB The image of

[100] is numbered " Figure 15 ", in the database image data GD DB

[100] Associated file data TD DB The paragraph [0xx7] of the document represented by

[100] contains " Figure 15 Therefore, for example, it is considered that the database image data GD is recorded in paragraph [0xx7] DB

[100] The description of the accompanying figure shows that the database tag TAG can be obtained by performing a morphological analysis on the article recorded in paragraph [0xx7]. DB

[100] .

[0173] Note that not all file data TD DB For example, in the database image data GD DB [1] Corresponding to the “ Figure 1 ", database image data GD DB[2] Corresponding to the same publication “ Figure 2 ", you can make the file data TD DB [1] The file and file data TD represented DB [2] The files represented are identical.

[0174] In morphological analysis, a text written in natural language is divided into morphemes (the smallest unit of meaning in language) and the word class of the morphemes can be identified. Thus, for example, only the nouns recorded in paragraph [0xx0] can be extracted to obtain the database tag TAG DB [1]. Figure 6A In the case shown, words such as "circuit diagram", "aaa", "bbb", "ccc", and "ddd" are database tags. DB [1]. In addition, words such as “box diagram”, “ggg”, “aaa”, “ccc”, and “hhh” are database tags. DB

[100] .

[0175] As mentioned above, the database tag TAG DB For example, by comparing the database image data GD DB Associated file data TD DB By using this method, the database tag TAG is obtained. DB , we can fully obtain the image data GD corresponding to the database DB Label the image with concepts, technical content, focus, etc.

[0176] In this specification, a tag means, for example, a word. For example, in Figure 6A In the case shown, the database tag TAG DB The number of [1] can be more than 5. In addition, the database tag TAG DB The number of

[100] can be more than 5.

[0177] Note that, for example, all words extracted by morphological analysis do not need to be tagged with the database tag TAG. DB For example, a specified number of words are extracted from the extracted words, and the extracted words can be a database tag TAG DB For example, only a specified number of words with a high TF-IDF (Term Frequency-Inverse Document Frequency) are extracted from the extracted words, and the extracted words can be database tags TAG DB By comparing with the database image data GD DB Related database tag TAG DBThe number of multiple database image data GD DB Each of them is equal, which can simplify the image retrieval method using the image retrieval system 10.

[0178] TF-IDF is calculated based on two indicators: word frequency (TF) and inverse document frequency (IDF). Therefore, the TF of words that appear frequently in the entire document increases, but the IDF decreases. Therefore, when extracting database tags TAG DB The TF-IDF of words that appear frequently in the entire document is lower than that of words that appear frequently in paragraphs, etc. and less frequently in other paragraphs, etc. For example, words that appear frequently in the entire document may not be words that strongly represent the characteristics of the image such as concepts, technical content, and focus points. Therefore, when using TF-IDF to obtain the database tag TAG DB For example, when using only TF to obtain the database tag TAG DB Compared with the case of DB Therefore, the image retrieval system 10 can perform retrieval with high accuracy. Note that instead of calculating TF-IDF, for example, only TF can be used to obtain the database tag TAG DB In this case, the calculations performed by the processing unit 13 can be simplified.

[0179] Note that morphological analysis can also be performed on documents with more than two paragraphs. For example, in addition to viewing the document as containing the database image data GD DB In addition to the paragraphs describing the figures, the morphological analysis can also be performed on the paragraphs before and after the paragraph. Figure 6A The file data TD shown DB [1] When performing morphological analysis, in addition to paragraph [0xx0], the morphological analysis can also be performed on the next paragraph [0xx1]. In this case, for example, the word "eee" recorded in paragraph [0xx1] can be a database tag TAG DB [1]. In addition, for example, Figure 6A The file data TD shown DB

[100] When performing morphological analysis, in addition to paragraph [0xx7], the morphological analysis can also be performed on paragraph [0xx6] of the previous paragraph. In this case, for example, the word "fff" recorded in paragraph [0xx6] can be a database tag TAG DB

[100] .

[0180] Alternatively, the image data GD may be recorded in the database. DB Alternatively, the morphological analysis can be performed on all paragraphs of the image data GD DBMorphological analysis is performed on paragraphs that have associated figure numbers and no other figure numbers.

[0181] Alternatively, it is also possible to perform morphological analysis on only a portion of the sentences in the document recorded in a specified paragraph. Figure 6A In the case shown in paragraph [0xx0], it is also possible to only include " Figure 1 At this time, the word "ddd" is not a database tag TAG DB [1].

[0182] Note that not only in the file data TD DB The word itself in the document being represented, and the synonym of the word can also be the database tag TAG DB For example, the synonym dictionary data is stored in advance in the storage unit 15 or the database 17, and the words extracted by morphological analysis and the words registered as synonyms of the words in the synonym dictionary can be database tags TAG DB Here, as synonyms, a commonly available synonym dictionary may be used, or synonyms extracted by word distribution representation may be used. Furthermore, synonyms extracted by distribution representation may be extracted using a database containing other documents in the field to which the search target document belongs.

[0183] Not only in file data TD DB The word itself in the document represented by the document. The synonym of the word is the database tag TAG DB , so the database tag TAG DB Can be used to strongly represent the database image data GD DB The concepts, technical content, and characteristics of the focus.

[0184] Note that you can get the database tag TAG DB Instead of using morphological analysis, for example, the image feature data GFD can be used to DB Get database tag TAG DB .

[0185] Figure 6B It shows the database image data GD DB An example of a method is shown in the figure associated with the figure number. Figure 6B In the publication data PD, the image data GD is included. DB [1] and image data GD DB [2] and file data TD DB In addition, the publication represented by the publication data PD contains the text " Figure 1 xxx" and text " Figure 2yyy”. Note that the data representing the text “ Figure 1 xxx” and the data representing the text “ Figure 2 yyy” are not included in the file data TD DB . In addition, Figure 6B The “x1”, “x2”, “x1 < x2”, dotted lines, arrows, etc. shown are added for ease of explanation and are not actually recorded on the publication represented by the publication data PD

[0186] In Figure 6B the method shown, for example, when the text “Figure N” is recorded within a specified distance range from the attached drawing, the drawing number set closest to the text “Figure N” can be “N”. Here, for example, the distance between the coordinates of the center of the text (center coordinates) and the center coordinates of the attached drawing can be the distance from the text to the attached drawing. Note that “N” is not limited to an integer and can include letters, for example. For example, N can also be “1(A)”.

[0187] In Figure 6B the case shown, the distance x1 between the center coordinates of the text “ Figure 1 xxx” and the center coordinates of the attached drawing corresponding to the database image data GD DB [1] is shorter than the distance x2 between the center coordinates of the text “ Figure 1 xxx” and the center coordinates of the attached drawing corresponding to the database image data GD DB [2]. Therefore, the attached drawing set at the distance closest to the text “ Figure 1 xxx” can be the database image data GD DB [1]. Therefore, the drawing number associated with the database image data GD DB [1] can be “1”.

[0188] In addition, in Figure 6B it is shown that in the paragraph [0zz3] of the file represented by the file data TD DB it is recorded that “ Figure 1 is”, and in the paragraph [0zz4] it is recorded that “ Figure 2 is”. As described above, in Figure 6B the case shown, the database label TAG DB [1] associated with the database image data GD DB [1] can be obtained, for example, by performing morphological analysis on the file recorded in the paragraph [0zz3]. In Figure 6B the words “block diagram”, “iii”, “kkk”, “hhh”, “ppp”, etc. recorded in the paragraph [0zz3] are the database label TAG DB [1].

[0189] Note that, for example, the center coordinates of all the drawings may be arranged into a first 1-dimensional arrangement, and for example, the center coordinates of all the text "Figure N" may be arranged into a second 1-dimensional arrangement. Furthermore, by comparing the coordinates included in the first 1-dimensional arrangement with the coordinates included in the second 1-dimensional arrangement, each drawing may be associated with the text "Figure N" recorded at the nearest coordinate. That is, the drawing number of the drawing whose coordinates are located closest to the coordinates representing the position of the text "Figure N" may be "N". The comparison of the coordinates included in the first 1-dimensional arrangement and the coordinates included in the second 1-dimensional arrangement may be performed, for example, by calculating the sum of the square of the difference in the x-coordinate and the square of the difference in the y-coordinate. When the comparison is performed using this method, for example, the element with the smallest value of the above sum may be the element located closest to the coordinate.

[0190] Through the above method, the processing unit 13 can obtain the database tag TAG DB The database tag TAG obtained by the processing unit 13 DB It can be stored in the database 17.

[0191] [Step S04]

[0192] Next, the database tag TAG is represented by a vector DB . Will represent the database tag TAG DB The vector is called the database tag vector TAGV DB . Figure 7A It shows Figure 6A The database tag TAG shown DB A diagram showing what it looks like when represented as a vector.

[0193] Database tag vector TAGV DB You can use the database tag TAG DB For example, it is input to the neural network included in the processing unit 13 to obtain. At this time, the database label vector TAGV DB For example, a distribution representation vector can be used. A distribution representation vector is a vector that represents a continuous value by quantifying each feature element (dimension) of a word. Vectors of words with similar meanings are close to each other.

[0194] The neural network used to obtain the distribution representation vector may have a structure different from that of the neural network used to obtain the above-mentioned image feature quantity data. Figure 7B 1 is a diagram showing a configuration example of a neural network 40 that is a neural network for obtaining a distribution representation vector.

[0195] In this specification and other documents, for example, a neural network used to obtain image feature data may be referred to as a first neural network, and a neural network used to obtain a distribution representation vector may be referred to as a second neural network. Note that the ordinal numbers are merely examples; for example, the neural network used to obtain a distribution representation vector may be referred to as a first neural network, and the neural network used to obtain image feature data may be referred to as a second neural network. Furthermore, for example, the neural network used to obtain image feature data may be referred to as a third neural network, and the neural network used to obtain a distribution representation vector may be referred to as a third neural network.

[0196] like Figure 7B As shown, the neural network 40 includes an input layer IL, an intermediate layer ML, and an output layer OL. Here, the neural network 40 may have a structure including an intermediate layer ML. The neural network 40 may use, for example, the open-source algorithm Word2Vec to obtain a distribution representation vector representing the word input to the input layer IL. Figure 7B The neural network 40 of the structure shown in the figure obtains the database tag TAG representing the input to the input layer IL DB Database tag vector TAGV DB An example of the method.

[0197] The input to the input layer IL is represented by a one-hot vector representing the database tag TAG DB Here, the 1 component of the one-hot vector represents a word. The component corresponding to the word input to the input layer IL can be 1, and the other components can be 0. In other words, the one-hot vector can be said to be a vector with the 1 component being 1 and the other components being 0. The number of neurons included in the input layer IL can be equal to the number of components constituting the one-hot vector.

[0198] The intermediate ML layer generates a distribution representation vector based on the one-hot vector input to the input layer IL. For example, the intermediate ML layer can generate a distribution representation vector by multiplying the one-hot vector by a predetermined weight. Since these weights can be represented as matrices, the neural network 40 generates a distribution representation vector by performing a sum-of-product operation between the one-hot vector and the weight matrix.

[0199] Note that the number of neurons included in the intermediate layer ML can be equal to the dimension of the distribution representation vector. For example, when the dimension of the distribution representation vector is 300, the intermediate layer ML can have a structure including 300 neurons.

[0200] The weight matrix can be obtained by learning, such as supervised learning. Specifically, a vector representing a word as a one-hot vector is input to the input layer IL, and a vector representing the surrounding words of the word input to the input layer IL as a one-hot vector is input to the output layer OL. Here, a plurality of surrounding words are input to the output layer OL for each word input to the input layer IL. Furthermore, the value of the weight matrix of the neural network 40 is adjusted in such a way that the output layer OL can output the probability of the surrounding words of the word input to the input layer IL for each word. Note that, for example, one neuron included in the output layer OL corresponds to one word. The above is an example of a learning method for the neural network 40.

[0201] As described above, each of the input layer IL and the output layer OL can correspond one neuron to one word. Therefore, the number of neurons included in the input layer IL can be equal to the number of neurons included in the output layer OL.

[0202] The number of neurons included in the intermediate layer ML can be smaller than the number of neurons included in the input layer IL. For example, the number of words that can be processed by the neural network 40, that is, the number of neurons included in the input layer IL, can be 10,000, and the dimension of the distributed representation vector, that is, the number of neurons included in the intermediate layer ML, can be 300. Therefore, since the dimension can be kept low even if the number of words that can be represented increases in the distributed representation, the computational complexity is not likely to increase even if the number of words that can be represented increases. Therefore, the image retrieval system 10 can perform retrieval in a short time.

[0203] Through the above method, the processing unit 13 can obtain the database tag vector TAGV DB The database tag vector TAGV obtained by the processing unit 13 DB It can be stored in the database 17.

[0204] Through the above steps S01 to S04, the processing unit 13 obtains the database image feature data GFD DB , database tag TAG DB and database tag vector TAGV DB , stored in the database 17. Thus, the image retrieval system 10 can retrieve database images similar to the query image. Note that the database tag TAG DB It may not be stored in the database 17.

[0205] exist Figure 2 In the method shown, after the processing unit 13 obtains the database image feature data GFD in step S01 and step S02, DB Afterwards, the processing unit 13 obtains the database tag TAG through steps S03 and S04.DB and database tag vector TAGV DB , but one embodiment of the present invention is not limited thereto. For example, the database tag TAG may be obtained in the processing unit 13 DB and database tag vector TAGV DB Then obtain the database image feature data GFD DB .

[0206] exist Figure 2 In the method shown, by adding the database tag TAG DB The vector input to the neural network 40 and output from the neural network 40 is itself the database tag vector TAGV DB , but one embodiment of the present invention is not limited to this. DB A variation of the method for obtaining .

[0207] First, the processing unit 13 obtains the database tag TAG DB The candidate word. Database tag TAG DB For example, the candidate words Figure 6A and Figure 6B The results can be obtained through morphological analysis.

[0208] Next, the acquired words are represented as vectors. For example, by inputting the acquired words into the neural network 40, the vector representation can be represented as a distribution.

[0209] Then, by clustering the distribution representation vector, a specified number of clusters are generated. For example, the database tag TAG that you want to obtain is generated. DB Clustering can be performed using the K-means method, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, or the like.

[0210] exist Figure 8A As shown in the database image data GD DB [1] The candidates of the associated tags, 20 words are obtained by the processing unit 13, and each of these words is represented by a database word vector WORDV DB In addition, Figure 8A In the figure, we show the word vectors WORDV based on 20 databases. DB An example of generating five clusters (cluster CST1, cluster CST2, cluster CST3, cluster CST4, and cluster CST5). Note that for ease of explanation, Figure 8AThe vector shown is a two-dimensional vector. The horizontal axis represents the component of one side of the two-dimensional vector, and the vertical axis represents the component of the other side of the two-dimensional vector. In fact, the database word vector WORDV DB For example, it can be a 300-dimensional vector.

[0211] Next, the vector representing the representative point of each of the clusters CST1 to CST5 is obtained. The vector representing the representative point can be the database label vector TAGV DB [1]. Figure 8A In the example, the vector representing the representative point of cluster CST1 is the database label vector TAGV1 DB [1], the vector representing the representative point of cluster CST2 is the database label vector TAGV2 DB [1], the vector representing the representative point of cluster CST3 is the database label vector TAGV3 DB [1], the vector representing the representative point of cluster CST4 is the database label vector TAGV4 DB [1], the vector representing the representative point of cluster CST5 is the database label vector TAGV5 DB [1].

[0212] Each component of the vector representing the representative point can be, for example, a database word vector WORDV included in the cluster. DB For example, when a cluster includes 5 database word vectors WORDV (0.1, 0.7), (0.2, 0.5), (0.3, 0.5), (0.4, 0.2), and (0.5, 0.1), DB When , the vector representing the representative point of the cluster can be, for example, (0.3, 0.4). Through the above steps, the processing unit 13 can obtain the database label vector TAGV DB [1].

[0213] In the database tag vector TAGV DB [2] Subsequent vectors can also be obtained using the same method. Figure 8B It shows the database image data GD DB [1] To database image data GD DB Each of

[100] obtains 5 database tag vectors TAGV DB (Database label vector TAGV1 DB , database tag vector TAGV2 DB , database tag vector TAGV3 DB , database tag vector TAGV4 DB and database tag vector TAGV5 DB ) is a table of the components of each vector when . Note that Figure 8B The composition shown is an example for ease of illustration.

[0214] like Figure 8B As shown, the database tag vector TAGV can be DB The weight can be, for example, the database word vector WORDV included in a cluster. DB The number of images divided by the database image data GD DB The total value of the words obtained by the candidate processing unit 13 of the associated label. Figure 8A and Figure 8B In FIG. 1 , the processing unit 13 is shown as a database image data GD DB [1] The candidate of the associated label is an example of 20 words. In addition, the cluster CST1 includes 8 database word vectors WORDV DB , in cluster CST2, there are 4 database word vectors WORDV DB , in cluster CST3, there are 2 database word vectors WORDV DB , in cluster CST4 there are 3 database word vectors WORDV DB , in cluster CST5, there are 3 database word vectors WORDV DB Therefore, if Figure 8B As shown, for example, regarding the database image data GD DB [1], the database tag vector TAGV1 included in cluster CST1 DB The weight of [1] can be 8 / 20, including the database tag vector TAGV2 in cluster CST2 DB The weight of [1] can be 4 / 20, including the database tag vector TAGV3 in cluster CST3 DB The weight of [1] can be 2 / 20, including the database tag vector TAGV4 in cluster CST4 DB The weight of [1] can be 3 / 20, including the database tag vector TAGV5 in cluster CST5 DB The weight of [1] could be 3 / 20.

[0215] By the above method, for example, the image data GD representing the database image can be increased. DB The weight of the label vector of the concept, technical content, focus, etc. As a result, the image retrieval system 10 can perform retrieval with high accuracy.

[0216] Next, an image retrieval method using the image retrieval system 10 will be described. Figure 9 This is a flowchart showing an example of the method of this processing.

[0217] [Step S11]

[0218] First, the user of the image retrieval system 10 inputs query image data GD into the input unit 11. Q . Query image data GD Q The image data GD is supplied from the input unit 11 to the processing unit 13 through the transmission channel 12. Q The data may be stored in the storage unit 15 or the database 17 via the transmission channel 12 , and supplied from the storage unit 15 or the database 17 to the processing unit 13 via the transmission channel 12 .

[0219] Query image data GD Q For example, it may include images illustrating inventions, designs or patterns before the application is filed, industrial products before they are sold, technical information or technical ideas, etc.

[0220] [Step S12]

[0221] Next, query the image data GD Q is input to the neural network included in the processing unit 13. For example, Figure 3A or Figure 3B The neural network 30 of the structure shown in FIG. 1 inputs the query image data GD Q Thus, the processing unit 13 can obtain the query image data GD Q The query image feature data GFD of the feature quantity Q For example, from Figure 3A The data output by the layer 31[m] shown in FIG. 31 can be the query image feature data GFD. Q Or, from Figure 3B The data output by the pooling layer PL[m] shown in FIG. 1 can be the query image feature data GFD Q Note that the query image feature data GFD Q Image feature data GFD DB Similarly, it can include more than 2 layers of output data. Q Includes output data of more layers, query image feature data GFD Q The query image data GD can be represented more accurately Q characteristics.

[0222] [Step S13]

[0223] Next, the processing unit 13 calculates the database image data GD DB For query image data GD Q similarity.

[0224] Figure 10 It shows the database image data GD DBFor query image data GD Q The similarity of the graph is calculated. Figure 10 As an example, one query image data GD is shown. Q and 100 database image data GD DB Input to Figure 3B An example of a neural network 30a is shown. In addition, Figure 10 In FIG, the query image feature data GFD is shown. Q and database image feature data GFD DB Each of the examples includes x rows and y columns (x and y are integers greater than or equal to 1).

[0225] In this specification, etc., the query image feature data GFD Q The included pooling value is recorded as pooling value P Q , database image feature data GFD DB The included pooling value is recorded as pooling value P DB In addition, for example, the database image feature data GFD DB [1] The included pooling value is recorded as pooling value P1 DB , database image feature data GFD DB The pooling value included in

[100] is recorded as pooling value P100 DB .

[0226] exist Figure 10 In the case shown, regarding the database image feature data GFD DB [1] To the database image feature data GFD DB

[100] , calculate the query image feature data GFD Q And, the similarity can be the database image data GD DB [1] To database image data GD DB

[100] for the query image data GD Q Note that, regarding all database image feature data GFD stored in the database 17, DB , we can also calculate the query image feature data GFD Q Or, regarding the database image feature data GFD stored in the database 17 DB As part of the query image feature data GFD can also be calculated Q similarity.

[0227] The above similarity is preferably cosine similarity. Alternatively, it can be Euclidean similarity or Minkowski similarity. For example, the database image feature data GFDDB [1] for query image feature data GFD Q The cosine similarity of can be calculated by the following formula. The larger the cosine similarity value, the more similar the database image data GD is. DB The more similar it is to the query image data GD Q .

[0228] (Formula 5)

[0229]

[0230] Database image feature data GFD DB [2] To the database image feature data GFD DB

[100] for query image feature data GFD Q The cosine similarity of can also be calculated by the same method. As mentioned above, the database image data GD can be calculated DB [1] To database image data GD DB

[100] for the query image data GD Q similarity.

[0231] By calculating cosine similarity as the similarity, the image retrieval system 10 can perform searches with high accuracy. Furthermore, cosine similarity can be calculated using simple calculations. Therefore, if the processing unit 13 includes a GPU, the similarity can be calculated using the GPU. This allows the similarity to be calculated in a short time, enabling the image retrieval system 10 to perform searches in a short time.

[0232] [Step S14]

[0233] Next, according to the database image data GD DB For query image data GD Q The processing unit 13 obtains the result of similarity calculation with the query image data GD Q Query tag TAG of the associated tag Q .

[0234] Figure 11A and Figure 11B It shows the query tag TAG Q The following is an example of a method for obtaining . First, as Figure 11A As shown, according to the similarity calculated in step S13, the database image data GD DB [1] To database image data GD DB

[100] Rearrange. For example, for the query image data GD Q The database image data GD with the highest similarity DB Rearrange in descending order. Figure 11AIn the case shown, the database image data GD DB [2] has the highest similarity, which is 0.999. The database image data GD DB

[31] has the second highest similarity, which is 0.971. DB

[73] has the third highest similarity, which is 0.964. The database image data GD DB

[52] has the fourth highest similarity, which is 0.951. The database image data GD DB

[28] has the fifth highest similarity, which is 0.937.

[0235] Next, the database image data GD with high similarity is extracted. DB Related database tag TAG DB .exist Figure 11A In the case shown, the database image data GD with the first to fifth highest similarities are extracted. DB Related database tag TAG DB Specifically, extract the image data GD from the database DB [2] The associated tags "aaa", "bbb", "ccc", "ddd" and "eee", and the database image data GD DB

[31] The associated tags "aaa", "ccc", "fff", "ggg" and "hhh", and the database image data GD DB

[73] The associated tags "aaa", "bbb", "fff", "iii" and "kkk", and the database image data GD DB

[52] The associated tags "aaa", "ccc", "ggg", "ppp" and "qqq", and the database image data GD DB

[28] The associated tags are “aaa”, “kkk”, “rrr”, “sss” and “ttt”. Figure 11A As shown, the extracted labels can also be repeated.

[0236] Note that in the above, the database tag TAG will be extracted DB Database image data GD DB The number of is set to a predetermined number, but one embodiment of the present invention is not limited thereto. For example, the database image data GD having a similarity greater than a predetermined value may be extracted. DB The associated database tag. In other words, you can also not extract the database tag TAG DB Database image data GD DB The number of is fixed.

[0237] Then, if Figure 11BAs shown in FIG, the number of occurrences of each extracted label is calculated. For example, since the label "aaa" is consistent with the database image data GD DB [2] Database image data GD DB

[31] Database image data GD DB

[73] Database image data GD DB

[52] and database image data GD DB

[28] are all associated, so the number of occurrences is 5. Since the label “ddd” is only associated with the database image data GD DB [2] Database image data GD DB

[31] Database image data GD DB

[73] Database image data GD DB

[52] and database image data GD DB

[28] Database image data GD DB [2] is associated, so the number of occurrences is 1.

[0238] Then, if Figure 11B As shown, for example, a predetermined number of tags are extracted from the tags with the highest number of occurrences, and the extracted tags are set as query tags TAG Q .exist Figure 11B In the case shown, the tags that appear the most times are used as query tags. Q Five tags are extracted. Specifically, the tag "aaa" that appears the most, 5 times, and the tag "ccc" that appears the second most, 3 times, are extracted.

[0239] When there are multiple labels with the same number of appearances, if all of the multiple labels cannot be extracted, for example, the database image data GD with a higher similarity can be extracted. DB Associated labels. For example, database image data GD is represented by a numerical value DB And, for the database image data GD that are associated with each other with the same number of labels, DB By comparing the sum of the values ​​of the order of similarity, labels with smaller total values ​​can be extracted in order.

[0240] exist Figure 11B In the case shown, query tag TAG QThe number of labels is 5, the number of occurrences of label "aaa" is 5, and the number of occurrences of label "ccc" is 3. Therefore, it is necessary to extract 3 labels from the labels with an occurrence of 2 or less. However, there are 4 labels with an occurrence of 2: "bbb", "fff", "ggg", and "kkk". Therefore, it is necessary to select 3 labels from these 4 labels. Here, the database image data GD associated with label "bbb" is DB The order of similarity of [2] is 1, and the database image data GD DB The order of similarity of

[73] is 3. Therefore, the total of the order of similarity according to the tag "bbb" is 4. Similarly, the total of the order of similarity according to the tag "fff" is 5, the total of the order of similarity according to the tag "ggg" is 6, and the total of the order of similarity according to the tag "kkk" is 8. As mentioned above, the tag with the smallest total value of the order of similarity according to the number of occurrences of 2 is the tag "bbb", and the tags "fff", "ggg", and "kkk" are arranged in ascending order. Therefore, the tags "bbb", "fff", and "ggg" can be the query tag TAG Q .

[0241] In short, Figure 11B In the case shown, the tag "aaa" with 5 occurrences, the tag "ccc" with 3 occurrences, and the tags "bbb", "fff", and "ggg" with the smallest total similarity among the tags with 2 occurrences can be the query tag TAG Q .

[0242] Note that not only the database tag TAG DB The word itself can also be included in the database tag TAG DB The synonyms of the words in the query tag TAG are included Q For example, the synonym dictionary data is stored in advance in the storage unit 15 or the database 17, and the synonym dictionary data can be included in the database tag TAG DB The word in and the words registered as synonyms of the word in the synonym dictionary are included in the query tag TAG Q middle.

[0243] exist Figure 11B In the case shown, the processing unit 13 extracts the database tag TAG DB Automatically select query tag TAG Q , but one embodiment of the present invention is not limited thereto. For example, the extracted database tag TAG is displayed to the user of the image retrieval system 10 DB The user of the image retrieval system 10 can also select a query tag TAG from the displayed tags. QAlternatively, for example, a database image with a high similarity may be displayed to the user of the image retrieval system 10, and the user of the image retrieval system 10 may select the displayed database image. And the database image data GD representing the selected database image may be displayed to the user of the image retrieval system 10. DB Related database tag TAG DB Part or all of the query tag TAG Q .

[0244] exist Figure 11A and Figure 11B In the method shown, the tag TAG is DB Select the query tag TAG Q , but one embodiment of the present invention is not limited thereto. For example, it is also possible to use a database tag TAG DB Generate a new tag, and the tag is the query tag TAG Q The following describes how the processing unit 13 uses the tag TAG to represent the database. DB Database tag vector TAGV DB Get the query tag TAG Q The query tag vector TAGV Q An example of the method.

[0245] exist Figure 11A The method described in can be used to obtain the query tag vector TAGV using the following method Q That is, the database tag TAG DB The extraction can be used with Figure 11A The same method as shown was performed.

[0246] Extract the database tag TAG DB Afterwards, the extracted database tag TAG DB Database tag vector TAGV DB Perform clustering to generate a specified number of clusters. For example, generate the query tag TAG you want to obtain Q Clustering can be performed using the K-means method, DBSCAN method, etc.

[0247] exist Figure 12A In the figure, it is shown that the processing unit 13 obtains Figure 11A The 25 database tags TAG shown DB In addition, Figure 12A In the figure, it is shown that the Figure 11A The database tag TAG shown DB Database tag vector TAGV DB, an example of generating 5 clusters (cluster CST1, cluster CST2, cluster CST3, cluster CST4 and cluster CST5). Note that for ease of explanation, Figure 12A The vector shown is a 2-dimensional vector, the horizontal axis direction represents the component of one side of the 2-dimensional vector, and the vertical axis direction represents the component of the other side of the 2-dimensional vector. In fact, it can be a 300-dimensional vector, for example. Figure 12A The numbers in the brackets show the extracted database tags TAG DB For example, “aaa(5)” indicates that the number of times the tag “aaa” appears is 5.

[0248] Next, the vector representing the representative point of each of the clusters CST1 to CST5 is obtained. The vector representing the representative point can be the query tag vector TAGV Q .exist Figure 12A In the example, the vector representing the representative point of cluster CST1 is the query label vector TAGV1 Q , the vector representing the representative point of cluster CST2 is the query label vector TAGV2 DB , the vector representing the representative point of cluster CST3 is the query label vector TAGV3 Q , the vector representing the representative point of cluster CST4 is the query label vector TAGV4 Q , the vector representing the representative point of cluster CST5 is the query label vector TAGV5 Q .

[0249] Each component of the vector representing the representative point can be, for example, a database tag vector TAGV included in the cluster. DB Through the above steps, the processing unit 13 can obtain the query tag vector TAGV Q .

[0250] Figure 12B It shows the query tag vector TAGV1 Q To query tag vector TAGV5 Q Table of ingredients. Note, Figure 12B The composition shown is an example for ease of illustration.

[0251] like Figure 12B As shown, the query tag vector TAGV Q The weight can be, for example, the database tag vector TAGV included in one cluster. DB Divide the number of Figure 12A The database tag TAG extracted by the method shown in the following example DB For example, Figure 12A and Figure 12B Shows 25 database tags TAG extractedDB In addition, cluster CST1 includes 11 database tag vectors TAGV DB , in cluster CST2, there are 4 database tag vectors TAGV DB , in cluster CST3 there are 5 database tag vectors TAGV DB , in cluster CST4, there are 2 database tag vectors TAGV DB , in cluster CST5 there are three database tag vectors TAGV DB Therefore, if Figure 12B As shown, the query tag vector TAGV1 included in the cluster CST1 Q The weight can be 11 / 25, including the query tag vector TAGV2 in cluster CST2 Q The weight of the query tag vector TAGV3 can be 4 / 25, including the query tag vector TAGV3 in cluster CST3 Q The weight can be 5 / 25, including the query tag vector TAGV4 in cluster CST4 Q The weight of the query tag vector TAGV5 can be 2 / 25, including the query tag vector TAGV5 in the cluster CST5 Q The weight can be 3 / 25.

[0252] By using the above method, for example, the query image data GD can be increased. Q The weight of the label vector of the concept, technical content, focus, etc. As a result, the image retrieval system 10 can perform retrieval with high accuracy.

[0253] The query tag TAG shown in step S13 and step S14 Q The method of obtaining the tag TAG is not based on the database. DB Get query tag TAG Q Therefore, the image retrieval system 10 can perform retrieval in a short time. In addition, the query tag TAG of the method shown in step S13 and step S14 is used. Q For example, the acquisition of all query tags TAG specified by the user of the image retrieval system 10 Q , and the query tag TAG is not displayed to the user Q Compared with the candidate case, the representation corresponding to the query image data GD can be fully obtained Q The image retrieval system 10 can thereby perform retrieval simply and with high accuracy.

[0254] [Step S15]

[0255] Next, the processing unit 13 obtains the database image feature data GFD DB, database tag vector TAGV DB Data D DB In addition, the processing unit 13 obtains the query image feature data GFD Q , query tag vector TAGV Q Data D Q .

[0256] Figure 13 It shows data D DB and data D Q The diagram shows an example of the structure of the database image feature data GFD. DB And query image feature data GFD Q Can have Figure 10 The structure is the same as that shown in the following example. DB Can have components including VC DB [1] To ingredient VC DB [h] (h is an integer greater than 2). Query tag vector TAGV Q Can have components including VC Q [1] To ingredient VC Q Here, for example, when one image data is associated with five labels represented by a 300-dimensional vector, h is 1500.

[0257] In this specification and other documents, for example, the database tag vector TAGV DB [1] The components included are recorded as component VC1 DB , database tag vector TAGV DB

[100] The components included are recorded as component VC100 DB .

[0258] As mentioned above, components can sometimes be referred to as values. In this case, both image feature data and label vectors can be considered sets of multiple values. Therefore, data and vectors can sometimes be interchanged.

[0259] [Step S16]

[0260] Next, the processing unit 13 calculates the data D DB For data D Q The similarity of . Figure 13 In the case shown, calculate the data D DB [1] to data D DB

[100] for each of the data D Q And, the similarity can be the database image data GD DB [1] To database image data GD DB

[100] for the query image data GDQ Therefore, the database image data GD calculated by the processing unit 13 in step S13 can be corrected. DB For query image data GD Q similarity.

[0261] Here, in Figure 8B and Figure 12B When adding weights to label vectors, weighting can be performed, for example, by multiplying the components included in the label vector by the weights.

[0262] Data D DB For data D Q The similarity of is preferably the same type as the similarity calculated by the processing unit 13 in step S13. For example, when the cosine similarity is calculated in step S13, it is preferable to use the data D DB For data D Q The cosine similarity is calculated based on the similarity.

[0263] For example, data D DB [1] for data D Q The cosine similarity can be calculated by the following formula.

[0264] (Formula 6)

[0265]

[0266] The data D can also be calculated in the same way DB [2] To data D DB

[100] for data D Q As mentioned above, the cosine similarity of data D can be calculated DB [1] to data D DB

[100] for data D Q Thus, the database image data GD calculated in step S13 can be corrected. DB [1] To database image data GD DB

[100] for the query image data GD Q similarity.

[0267] Note that by adjusting the ratio of the number of values ​​included in the image feature data to the number of components included in the label vector, the search results can be changed. Q Included values ​​and database image feature data GFD DB The number of values ​​included or reduced query tag vector TAGV Q The number of components included and the database label vector TAGV DBWhen the number of components included is large, the corrected similarity becomes the result of focusing on the image feature quantity. For example, in the database image data GD DB The feature quantity and query image data GD Q When the feature quantities of are similar, even if the database tag TAG DB and query tag TAG Q Somewhat different, for the database image data GD DB Query image data GD Q On the other hand, when reducing the query image feature data GFD Q The number of values ​​included and the database image feature data GFD DB The number of values ​​included or the query tag vector TAGV Q The number of components included and the database label vector TAGV DB When the number of components included is large, the corrected similarity becomes the result of emphasizing the label. For example, in the database label TAG DB and query tag TAG Q Similar, even if the database image data GD DB The feature quantity and query image data GD Q The feature quantities of the database image data GD are slightly different. DB For query image data GD Q The corrected similarity also increases.

[0268] To increase or decrease the number of components included in the label vector, the number of labels associated with the image data can be increased or decreased. Furthermore, by, for example, using only a portion of the values ​​included in the image feature data for similarity calculation, it is possible to calculate similarity that prioritizes labels. For example, by not using feature values ​​representing portions that do not create a strong impression when viewing an image for similarity calculation, it is possible to calculate similarity that prioritizes labels while suppressing the increase in similarity for database images that differ significantly from the query image's appearance. Therefore, the image retrieval system 10 can perform retrieval with high accuracy.

[0269] In addition, by multiplying the value included in the image feature data or the component included in the label vector by a predetermined coefficient, the search result can be changed. For example, by querying the image feature data GFD Q Included values ​​and database image feature data GFD DB By multiplying the included values ​​by a real number greater than 1, the corrected similarity-weighted image feature quantity can be obtained. In addition, by querying the tag vector TAGV Q Included components and database tag vector TAGV DBThe included components are multiplied by a real number greater than 0 and less than 1 to obtain the corrected similarity-based image feature value. For example, by querying the image feature value data GFD Q Included values ​​and database image feature data GFD DB By multiplying the included values ​​by a real number greater than 0 and less than 1, the corrected similarity-weighted label result can be obtained. In addition, by querying the tag vector TAGV Q Included components and database tag vector TAGV DB The included components are multiplied by a real number greater than 1 to obtain the result of the corrected similarity-weighted labeling.

[0270] [Step S17]

[0271] Next, the processing unit 13 generates ranking data including information on the order of the corrected similarities calculated in step S16 , and outputs the ranking data to the outside of the image retrieval system 10 as a search result.

[0272] The processing unit 13 can supply the ranking data to the storage unit 15 or the database 17 via the transmission channel 12. In addition, the processing unit 13 can supply the ranking data to the output unit 19 via the transmission channel 12. Thus, the output unit 19 can supply the ranking data to the outside of the image retrieval system 10.

[0273] The ranking data may include the order of similarity of each database image to the query image, the value of the similarity, and the like. Note that the ranking data preferably includes the file path to the database image. This allows the user of the image retrieval system 10 to easily access the target image from the ranking data. Furthermore, the user can also confirm the tags associated with the query image and the output database image. Furthermore, for example, when publication data representing publications containing database images is stored in the database 17, the user of the image retrieval system 10 can easily access publications containing database images associated with the ranking data. The above is an example of an image retrieval method using the image retrieval system 10.

[0274] In the image retrieval method using the image retrieval system 10, first, the database image data GD is calculated without being associated with a label. DB For query image data GD Q Then, the similarity is corrected by associating it with the label. This can, for example, prevent the retrieval of database images that have similar features to the query image but different retrieval concepts, technical contents, and focus points.

[0275] For example, according to the query image data GD Q The database image data GD with the highest similarity from 1 to 5 DB Related database tag TAGDB , processing unit 13 obtains the query tag TAG Q In this case, it is possible to suppress the transfer of the database image data GD with the similarity ranking below 6. DB Image data with different concepts, technical contents, and focus points from the query image is mixed in. Therefore, it is possible to prevent images from being mixed into the search results and causing them to become noise, and prevent the desired image from being output. As described above, the image retrieval system 10 can perform retrieval with high accuracy.

[0276] In the image retrieval method using the image retrieval system 10, according to the database tag TAG DB Get query tag TAG Q This method is not based on the database tag TAG DB Get query tag TAG Q Therefore, the image retrieval system 10 can perform retrieval in a short time. DB Get query tag TAG Q For example, the user of the image retrieval system 10 specifies all query tags TAG Q , and the query tag TAG is not displayed to the user Q Compared with the candidate case, the representation corresponding to the query image data GD can be fully obtained Q The image retrieval system 10 can thereby perform retrieval simply and with high accuracy.

[0277] <1-3. Image Retrieval Method-2>

[0278] exist Figure 9 In the image retrieval method shown in FIG. 1 , the user of the image retrieval system 10 does not input a query tag TAG Q , but one embodiment of the present invention is not limited thereto. Figure 14 The user of the image retrieval system 10 manually inputs a query tag TAG Q A flowchart of an example of an image retrieval method using the image retrieval system 10 when using a portion of Figure 14 The method shown enables the image retrieval system 10 to work in the case of using Figure 9 The image retrieval method shown in the figure makes the image retrieval system 10 work in the same manner as before. Figure 2 Just follow the steps shown.

[0279] [Step S21]

[0280] First, the user of the image retrieval system 10 searches for the image data GD Q In addition, the tag TAG will be searchedQ Input to the input unit 11. The query tag TAG input by the user of the image retrieval system 10 Q The number of query tags TAG Q The content can be set by the user. In addition, the user can also set the query tag TAG automatically obtained in the following steps. Q Query tag TAG Q The number of .

[0281] Figure 15 The query image data GD Q and query tag TAG Q FIG for inputting the input unit 11. Figure 15 In the case shown, the user of the image retrieval system 10 not only searches for the image data GD Q In addition, the query image data GD is also output. Q 2 search tags for "circuit diagram" and "semiconductor" Q .

[0282] Here, by changing the query tag TAG input to the input unit 11 Q , you can change the database image data GD DB For query image data GD Q For example, when the query tag TAG of "capacitor" is Q When input to the input unit 11, the similarity to the database image data representing the circuit diagram without describing the capacitor can be reduced.

[0283] [Step S22]

[0284] Next, query the image data GD Q is input to the neural network included in the processing unit 13. For example, Figure 3A or Figure 3B The neural network 30 of the structure shown in FIG. 1 inputs the query image data GD Q Thus, the processing unit 13 can obtain the query image data GD Q The query image feature data GFD of the feature quantity Q .

[0285] [Step S23]

[0286] Next, the processing unit 13 obtains the database image feature data GFD DB , database tag vector TAGV DB Data D DB In addition, the processing unit 13 obtains the query image feature data GFD Q , query tag vector TAGVQ Data D Q .

[0287] Here, in the image data GD DB Related database tag TAG DB The number of query tags TAG input to the input unit 11 is greater than Q When the number of GD DB The associated tags are selected to be included in the data D DB For example, a database image data GD DB With 5 database tags TAG DB Furthermore, the query tag TAG input to the input unit 11 Q The number of is 2. At this time, data D DB The included tags can be 5 database tags TAG DB For example, the tag with the highest TF-IDF and the tag with the second highest TF-IDF.

[0288] [Step S24]

[0289] Next, the processing unit 13 calculates the data GD DB For data GD Q The similarity can be measured using Figure 13 The calculation is done in the same way as shown.

[0290] [Step S25]

[0291] Then, according to data D DB For data D Q Add and modify the query tag TAG in the calculation result of similarity Q .

[0292] Figure 16A and Figure 16B It shows the query tag TAG Q The following is an example of the append method. First, as Figure 16A As shown, according to the similarity calculated in step S24, the data D DB Rearrange. Figure 16A In the figure, 100 data D are shown. DB Rearrange the example. For example, for data D Q The data with the highest similarity D DB Rearrange in descending order. Figure 16A In the case shown, data D DB [2] has the highest similarity, which is 0.999. DB The similarity of

[41] is the second highest, which is 0.971.DB

[53] has the third highest similarity, which is 0.964. DB

[22] has the fourth highest similarity, which is 0.951. DB

[88] has the fifth highest similarity, which is 0.937.

[0293] Next, extract the data D with high similarity DB Included database image data GD DB Related database tag TAG DB .exist Figure 16A In the case shown, the data D with the first to fifth highest similarity are extracted. DB Included database image data GD DB Related database tag TAG DB Specifically, extract the image data GD from the database DB [2] The associated tags "aaa", "bbb", "ccc", "ddd" and "eee", and the database image data GD DB

[41] The associated tags "aaa", "ccc", "fff", "ggg" and "hhh", and the database image data GD DB

[53] The associated labels "aaa", "bbb", "fff", "iii" and "kkk", and the database image data GD DB

[22] The associated tags “aaa”, “ccc”, “ggg”, “ppp” and “qqq”, and the database image data GD DB

[88] The associated tags are "aaa", "kkk", "rrr", "sss" and "ttt". Figure 11A As in the case shown, the extracted labels may also be repeated.

[0294] Then, if Figure 16B As shown, the number of occurrences of each extracted tag is calculated.

[0295] Then, if Figure 16B As shown, a predetermined number of tags are extracted from the above-extracted tags, and the extracted tags are set as new query tags TAG Q .exist Figure 16B In the case shown, step S21 is used as the query tag TAG Q Two tags ("circuit diagram" and "semiconductor") have been obtained. And, by adding three tags, the search tag TAG Q The number of images in the database is 5, which is the same as the number of images in the database GD DB Related database tag TAG DB The number of is equal.

[0296] New query tag TAG Q The label can be extracted using Figure 11B For example, the tags with the highest number of occurrences can be extracted in sequence. In addition, when there are multiple tags with the same number of occurrences, if all of the multiple tags cannot be extracted, for example, the tags with the highest similarity can be extracted. DB Included database image data GD DB The associated tags. Figure 16B In the case shown, it can be used as a new query tag TAG Q The tags "aaa", "bbb", and "ccc" can be extracted.

[0297] In short, Figure 16B In the case shown in FIG, in addition to the tags "circuit diagram" and "semiconductor" inputted by the user of the image retrieval system 10 to the input unit 11 in step S21, the five tags "aaa", "bbb", and "ccc" can be new query tags TAG Q .

[0298] Note that you can also query the TAG Q Delete part or all of the input to the input unit 11 by the user of the image retrieval system 10. For example, you can also delete the tag TAG Q Delete the tags "circuit diagram", "semiconductor", from Figure 16B Extract 5 tags from the displayed tags and set them as new tags TAG Q At this time, for example, the tags "aaa", "bbb", "ccc", "fff", and "ggg" can be new tags TAG Q .

[0299] [Step S26]

[0300] Next, corresponding to the query tag TAG Q The addition and correction of data D DB The included tags are added or modified. For example, a data D DB Included database tag vector TAGV DB The number of query tags TAG Q The number of is equal.

[0301] [Step S27]

[0302] Next, the processing unit 13 calculates the data GD again. DB For data GD Q The similarity can be calculated using the same method as that shown in step S24.DB For data GD Q similarity.

[0303] [Step S28]

[0304] Next, the processing unit 13 generates ranking data including information on the order of the corrected similarities calculated in step S27, and outputs the data as a search result to the outside of the image search system 10. This allows the user of the image search system 10 to confirm, for example, the order of similarity of each database image to the query image, similarity values, retrieved database images, labels, and the like.

[0305] [Step S29, Step S30]

[0306] Next, the user of the image retrieval system 10 confirms whether the result of the sorted data is the expected result. If the expected result is obtained, the search ends. If the expected result is not obtained, the user of the image retrieval system 10 returns the query tag TAG Q After addition and correction, the process returns to step S23 . The above is an example of an image retrieval method using the image retrieval system 10 .

[0307] This embodiment mode can be appropriately combined with other embodiment modes. In addition, in this specification, when a plurality of structural examples are shown in one embodiment mode, the structural examples can be appropriately combined.

[0308] (Implementation Method 2)

[0309] In the first embodiment, the image retrieval system 10 performs a search on the database image data GD DB The entire region and query image data GD Q Compare the entire area and calculate the query image data GD Q For database image data GD DB However, one embodiment of the present invention is not limited thereto. For example, by performing the similarity analysis on the database image data GD DB A part of the region and query image data GD Q By comparing the whole area, the database image data GD can also be calculated. DB For query image data GD Q Alternatively, by comparing the database image data GD DB The entire region and query image data GD Q By comparing the area of ​​a part of the image, the database image data GD can also be calculated. DB For query image data GD Q similarity.

[0310] <2-1. Image Retrieval Method-3>

[0311] Figure 17 It is through the database image data GD DB A part of the region and query image data GD Q Compare the entire area and calculate the database image data GD DB For query image data GD Q An example of an image retrieval method using the image retrieval system 10 when the similarity is Figure 9 Step S11 or Figure 14 Step 21 is shown.

[0312] [Step S31]

[0313] Next, the processing unit 13 processes the query image data GD Q Image data with database GD DB Compare and extract the image data GD that is consistent with the query image Q Database image data GD of the area with high consistency DB Here, the extracted database image data GD DB To extract image data GD Ex . Query image data GD Q Image data with database GD DB The comparison can be performed, for example, by region matching.

[0314] Reference Figures 18A to 20B2 An example of the operation of step S31 is described in detail. In step S31, Figure 18A As shown, for the query image data GD Q and n (n is an integer greater than 1) database image data GD DB Here, n may be equal to or less than the database image data GD stored in the database 17. DB In addition, n can also be more than the database image data GD stored in the database 17. DB At this time, in addition to the database image data GD stored in the database 17 DB In addition, the image data stored in the storage unit 15 or the image data input to the processing unit 13 from outside the image retrieval system 10 through the input unit 11 and the query image data GD Q Note that even if n is the database image data GD DB The number of image data stored in the storage unit 15 or the image data input to the processing unit 13 from the outside of the image retrieval system 10 through the input unit 11 and the query image data GD Q Make a comparison.

[0315] When n is small, the operation of step S31 can be performed in a short time. On the other hand, when n is large, the operation including the query image data GD can be performed with high accuracy. Q Database image data GD of the area with high consistency DB Extraction.

[0316] Figure 18B It is to illustrate the query image data GD through region matching Q Image data with database GD DB Here, the query image data GD Q The number of pixels of the image is 2×2, corresponding to the database image data GD DB The number of pixels of the image is 4×4. That is, the query image data GD Q Including 2×2 pixel values, database image data GD DB Contains 4×4 pixel values.

[0317] exist Figure 18B In the query image data GD Q The included 2×2 pixel values ​​are pixel value vq11, pixel value vq12, pixel value vq21, and pixel value vq22. For example, in the query image data GD Q , the pixel value corresponding to the pixel in the first row and first column is pixel value vq11, the pixel value corresponding to the pixel in the first row and second column is pixel value vq12, the pixel value corresponding to the pixel in the second row and first column is pixel value vq21, and the pixel value corresponding to the pixel in the second row and second column is pixel value vq22. DB The included 4×4 pixel values ​​are pixel values ​​vdb11 to pixel values ​​vdb44. For example, in the database image data GD DB , the pixel value corresponding to the pixel in the 1st row and 1st column is pixel value vdb11, the pixel value corresponding to the pixel in the 1st row and 4th column is pixel value vdb14, the pixel value corresponding to the pixel in the 4th row and 1st column is pixel value vdb41, and the pixel value corresponding to the pixel in the 4th row and 4th column is pixel value vdb44.

[0318] First, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb11, vdb12, vdb21, and vdb22. Q Image data with database GD DB The consistency of the area composed of pixel value vdb11, pixel value vdb12, pixel value vdb21 and pixel value vdb22 in . Figure 18B The database image data GD is shown surrounded by a dotted line. DB The pixel values ​​included are consistent with the query image data GD Q The pixel values ​​to be compared are provided as a comparison data area 21 .

[0319] Next, for the database image data GD DB The included pixel values ​​are moved by the comparison data area 21 in a column, and the pixel values ​​are compared in the same manner to calculate the consistency. Specifically, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb12, vdb13, vdb22, and vdb23. In this way, the query image data GD can be calculated. Q Image data with database GD DB The consistency of the area composed of pixel value vdb12, pixel value vdb13, pixel value vdb22 and pixel value vdb23.

[0320] Then, for the database image data GD DB The included pixel values ​​are moved by the comparison data area 21 in a column, and the pixel values ​​are compared in the same manner to calculate the consistency. Specifically, the pixel values ​​vq11, vq12, vq21, and vq22 are compared with the pixel values ​​vdb13, vdb14, vdb23, and vdb24. In this way, the query image data GD can be calculated. Q Image data with database GD DB The consistency of the area composed of pixel value vdb13, pixel value vdb14, pixel value vdb23 and pixel value vdb24.

[0321] Next, for the database image data GD DB The pixel values ​​included in the comparison data area 21 are moved by one row, and the database image data GD is compared for each column in the same manner as described above. DB The pixel values ​​of the second row and the pixel values ​​of the third row constitute the query image data GD Q Thus, the database image data GD can be calculated for each column in the same manner as above. DB The area consisting of the pixel values ​​of the second and third rows and the query image data GD Q degree of consistency.

[0322] Then, for the database image data GD DB The pixel values ​​included in the comparison data area 21 are moved by one row, and the database image data GD is compared for each column in the same manner as described above. DBThe pixel values ​​of the 3rd row and the pixel values ​​of the 4th row constitute the query image data GD Q Thus, the database image data GD can be calculated for each column in the same manner as above. DB The pixel values ​​of the 3rd and 4th rows included are consistent with the query image data GD Q degree of consistency.

[0323] After the above work, for example, the highest consistency is the database image data GD DB For query image data GD Q The consistency of n database image data GD DB Then, from n database image data GD DB Extract and query image data GD Q Database image data GD with high consistency DB As the extracted image data GD Ex For example, we can also use the database image data GD with high consistency DB Sequentially extract a specified number of data as extracted image data GD Ex Alternatively, for example, the query image data GD may be extracted. Q The database image data GD whose consistency is above the specified value DB As the extracted image data GD Ex .

[0324] In addition, the database image data GD may not be DB In other words, the query image data GD Q Database image data GD for comparison DB All of the extracted image data GD Ex .

[0325] Figure 19 It is the description database image data GD DB The extracted picture. Figure 19 It shows the image data GD from the database DB [1] To database image data GD DB [3] Extract one image data as image data GD Ex example.

[0326] Corresponding to Figure 19 The query image data GD shown Q The image includes, for example, the symbol of a transistor. In addition, the image corresponding to Figure 19 The database image data GD shown DB The image of [2] includes the symbol of a transistor, and corresponds to the database image data GD DB[1] and the corresponding database image data GD DB The image of [3] does not include the transistor symbol. At this time, the database image data GD DB [2] for the query image data GD Q The consistency of the database image data GD DB [1] and database image data GD DB [3] for the query image data GD Q Therefore, the database image data GD can be extracted DB [2] As the extracted image data GD Ex .

[0327] Note that the query image data GD Q Image data with database GD DB The comparison and calculation of the consistency can be performed by SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), NCC (Normalized Cross Correlation), ZNCC (Zero-mean Normalized Cross Correlation), POC (Phase-Only Correlation), etc.

[0328] In addition, Figure 18B In the database image data GD DB The included pixel values ​​are moved in the comparison data area 21 by one column or one row, but one embodiment of the present invention is not limited thereto. DBThe included pixel values ​​are moved by an amount equivalent to two or more columns or two or more rows in the comparison data area 21. For example, after comparing pixel values ​​vq11, vq12, vq21, and vq22 with pixel values ​​vdb11, vdb12, vdb21, and vdb22, pixel values ​​vq11, vq12, vq21, and vq22 may be compared with pixel values ​​vdb13, vdb14, vdb23, and vdb24. In this case, pixel values ​​vq11, vq12, vq21, and vq22 are not compared with pixel values ​​vdb12, vdb13, vdb22, and vdb23. In addition, after comparing pixel value vq11, pixel value vq12, pixel value vq21 and pixel value vq22 with pixel value vdb13, pixel value vdb14, pixel value vdb23 and pixel value vdb24, pixel value vq11, pixel value vq12, pixel value vq21 and pixel value vq22 can be compared with pixel value vdb31, pixel value vdb32, pixel value vdb41 and pixel value vdb42.

[0329] By increasing the moving width of the comparison data area 21, the query image data GD can be reduced. Q The pixel values ​​included are consistent with the database image data GD DB The number of pixel value comparison operations included is thus the number of times the database image data GD is compared. DB For query image data GD Q Calculation of consistency.

[0330] exist Figure 18A In FIG, a query image data GD is shown. Q With n database image data GD DB This is an example of comparison, but one embodiment of the present invention is not limited to this. Figure 20A As shown, according to the query image data GD input to the processing unit 13 Q , multiple query image data GD with different pixel values ​​can be generated Q . Figure 20A The query image data GD input to the processing unit 13 is shown. Q Generate query image data GD with different numbers of pixel values Q [1] Query image data GD Q [2] and query image data GD Q [3] For example. Figure 20A As shown, corresponding to the query image data GD Q[1] The number of pixels of the image corresponding to the query image data GD Q [2] The number of pixels of the image corresponding to the query image data GD Q [3] has different numbers of pixels from each other. That is, it can be said that the image data GD Q [1] To query image data GD Q The image of [3] is enlarged or reduced according to the query image data GD input to the processing unit 13. Q image.

[0331] In generating multiple query image data GD Q When the multiple query image data GD Q Each of the database image data GD DB [1] To database image data GD DB [n] are compared with each other. Thus, the database image data GD can be calculated DB [1] To database image data GD DB [n] each of the multiple query image data GD Q And, for example, for the above-mentioned multiple query image data GD Q The highest consistency among the consistency can be the database image data GD DB For the query image data GD input to the processing unit 13 Q degree of consistency.

[0332] For example, in Figure 20A In the case shown, the query image data GD Q [1] and database image data GD DB [1] To database image data GD DB Each of [n] is compared with the query image data GD Q [2] and database image data GD DB [1] To database image data GD DB Each of [n] is compared with the query image data GD Q [3] and database image data GD DB [1] To database image data GD DB [n] are compared with each other. Thus, the database image data GD can be calculated DB [1] To database image data GD DB For each of [n], the query image data GD Q [1] consistency, for the query image data GD Q [2] consistency and the query image data GD Q[3] consistency.

[0333] And, for example, for the query image data GD Q [1] consistency, for the query image data GD Q [2] consistency and the query image data GD Q The highest consistency among the consistency of [3] can be the database image data GD DB For the query image data GD input to the processing unit 13 Q For example, the database image data GD DB [1] for the query image data GD Q [1] consistency, for the query image data GD Q [2] consistency and the query image data GD Q The highest consistency among the consistency of [3] can be the database image data GD DB [1] For the query image data GD input to the processing unit 13 Q degree of consistency.

[0334] Even if the query image data GD Q The image corresponds to the database image data GD DB The images represent the same element, and when the sizes of the elements of both sides are different, it is sometimes judged that the image data GD in the database DB The query image data GD is not included Q The area with high consistency. Figure 20B1 In the case shown, the query image data GD Q The image and the corresponding database image data GD DB The same element of the transistor symbol is shown on both sides of the image. However, in the image corresponding to the query image data GD Q The size of the transistor symbol represented in the image and the size of the transistor symbol corresponding to the database image data GD DB The size of the transistor symbol represented in the image is different. At this time, it is possible to judge that the database image data GD DB For query image data GD Q The consistency is low.

[0335] On the other hand, Figure 20B2 In the case shown, in the case corresponding to the query image data GD Q The image and the corresponding database image data GD DB The same element of the transistor symbol is shown on both sides of the image, and the size of the elements on both sides is the same. Therefore, the processing unit 13 can determine that the image data in the database GD DB Including the query image data GDQ The area with high consistency.

[0336] like Figure 20A As shown, by generating a plurality of query image data GD with different numbers of pixel values Q , you can zoom in or out on the image data GD corresponding to the query Q Therefore, even if the query image data GD input to the processing unit 13 Q The image corresponds to the database image data GD DB When the same element is represented in different sizes in the image, it can be judged that the consistency of the two image data is high. Figure 20B1 The query image data GD shown Q When the query image data GD Q The number of pixel values ​​included is different Figure 20B2 The query image data GD shown Q , which can make the database image data GD DB For query image data GD Q As described above, the database image data GD can be calculated with high precision. DB For the query image data GD input to the processing unit 13 Q degree of consistency.

[0337] [Step S32]

[0338] Next, the processing unit 13 extracts the image data GD Ex Extract the query image data GD Q Partial image data GD of the area with high consistency part For example, by Figure 18B The method shown is used to calculate the database image data GD DB The query image data GD for each region Q When the consistency is greater than , the area with the highest consistency is extracted as the partial image data GD part Therefore, part of the image data GD part The number of pixel values ​​included can be the same as the query image data GD Q The number of included pixel values ​​is equal.

[0339] Figure 21A and Figure 21B FIG. 1 is a diagram showing an example of the operation of step S32. Figure 21A and Figure 21B The extracted image data GD shown Ex [1] To extract image data GD ExIn [4], for the query image data GD Q The areas with high consistency are shaded. Figure 21A and Figure 21B As shown, the shaded area can be extracted and used as the partial image data GD part [1] To partial image data GD part [4]. Figure 21A and Figure 21B , extract the image data GD from Ex [1] To extract image data GD Ex [4] The extracted image data are partial image data GD part [1] To partial image data GD part [4].

[0340] Figure 21A It shows that Figure 18A As shown in FIG1 query image data GD Q Image data with database GD DB In this case, the image data GD part The number of pixels of the images are equal.

[0341] Figure 21B Shown as Figure 20A The plurality of query image data GD having different numbers of pixel values ​​are shown. Q Image data with database GD DB An example of comparison. At this time, the corresponding partial image data GD part For example, the number of pixels of the image can be the same as the query image data GD corresponding to the highest degree of consistency. Q Therefore, when there are multiple partial image data GD part Sometimes it corresponds to part of the image data GD part The number of pixels of the image and the image data GD of each part part Different. Figure 21B In FIG. 1 , the image data GD corresponding to the portion of the image data GD is shown. part [1] To partial image data GD part [4] is an example of images with different numbers of pixels.

[0342] Note that it is also possible not to perform GD on some image data. part At this time, by extracting part of the image data GD part It is appropriately replaced by the extracted image data GD Ex , the following description can be applied. Or, it can be said that the image data GD is extracted Ex The whole is the partial image data GD partFor example, by increasing or decreasing the query image data GD Q and extract image data GD Ex The number of pixel values ​​included in the image data on one side or above is not required even if the partial image data GD is not performed. part The extraction may also be performed by performing an image retrieval method using the image retrieval system 10.

[0343] [Step S33]

[0344] Next, by converting the query image data GD Q The input is sent to the neural network included in the processing unit 13, and the processing unit 13 obtains the query image feature data GFD Q In addition, by converting part of the image data GD part The neural network included in the processing unit 13 is input, and the processing unit 13 obtains the database image feature value data GFD DB . Query image data GD Q And part of the image data GD part For example, you can enter Figure 3A or Figure 3B The structure of the neural network 30 is shown. Note that when using Figure 17 The method shown in the figure can be used to make the image retrieval system 10 work without Figure 2 That is, it is also possible not to obtain the database image data GD DB The database of the feature quantity of the entire region, the image feature quantity data GFD DB .

[0345] As described in the first embodiment, the database image data GD can be used as the learning data of the neural network 30. DB Here, the number of pixel values ​​included in the image data used for learning data is preferably equal to the number of pixel values ​​included in the image data input to the neural network 30. Therefore, when the neural network 30 is learning, it is preferred that the database image data GD used for learning data is equal to the number of pixel values ​​included in the image data input to the neural network 30. DB The number of pixel values ​​can be adjusted by increasing or decreasing the number of pixel values ​​as needed. Q Or part of the image data GD part When the query image data GD is increased or decreased as needed, Q Or part of the image data GD part The number of pixel values ​​included. Here, the increase in pixel values ​​is preferably performed by padding, for example, preferably by zero padding.

[0346] Figure 22A It is the description database image data GD DB The number of pixel values ​​included in the adjustment graph. Figure 22A In the case shown, the database image data GD DB [1] To database image data GD DB [4] The number of pixel values ​​included is different. At this time, in the database image data GD DB [1] To database image data GD DB [4] When used for learning data of neural network 30, such as Figure 22A As shown, it is preferable to make the number of pixel values ​​included in these image data consistent.

[0347] Figure 22B It is the description of the image data GD part The image data GD includes the number of pixel values ​​adjusted. part The number of pixel values ​​included is preferably equal to the number of pixel values ​​included in the image data used for learning the neural network 30. Similarly, when the query image data GD is input to the neural network 30, Q When the query image data GD Q The number of pixel values ​​included is preferably equal to the number of pixel values ​​included in the image data used for learning the neural network 30 .

[0348] After step S33, the image retrieval system 10 performs Figure 9 Step S13 shown or Figure 14 Specifically, if step S11 is performed before step S31, step S13 is performed after step S33; if step S21 is performed before step S31, step S23 is performed after step S33. DB A part of the region and the query image data GD Q The database image data GD is calculated by comparing the entire area DB Query image data GD Q This is an example of an image retrieval method using the image retrieval system 10 when the similarity is .

[0349] exist Figure 17 In the method shown, the query image data GD is matched by region matching or the like. Q Image data with database GD DB Compare and extract the query image data GD Q Database image data GD of the area with high consistency DB As the extracted image data GD Ex Then, the image data GD is extracted from Ex The area with high consistency is extracted as the partial image data GD part , query the image data GDQ And part of the image data GD part The image data GD is input to the neural network included in the processing unit 13. DB , can suppress representations that do not include data GD corresponding to the query image Q A database of images with high image consistency GD DB The image data GD corresponding to the query image data GD can be retrieved with high accuracy and in a short time. Q The image in the database is similar to the image in the query image data GD. Q Database image data GD for comparison DB In the case of a small number of cases, even if the database image data GD is not extracted DB , the above search can also be performed with high accuracy and in a short time.

[0350] <2-2. Image Retrieval Method-4>

[0351] Figure 23 It is through the database image data GD DB The whole region and the query image data GD Q The database image data GD is calculated by comparing the part of the area DB For query image data GD Q An example of an image retrieval method using the image retrieval system 10 when the similarity is Figure 9 Step S11 or Figure 14 Step 21 is shown.

[0352] [Step S41]

[0353] Next, the processing unit 13 processes the query image data GD Q Image data with database GD DB Compare and extract the query image data GD Q A part of the database image data GD with high consistency DB As the extracted image data GD Ex . Query image data GD Q Image data with database GD DB The comparison can be performed, for example, by area matching, similarly to step S31.

[0354] Reference Figure 24A 、 Figure 24B and Figure 25 An example of the operation of step S41 is described in detail. In step S41, Figure 24AAs shown, for the query image data GD Q With n database image data GD DB Compare each one.

[0355] Figure 24B It is to illustrate the query image data GD through region matching Q Image data with database GD DB Here, the query image data GD Q The number of pixels of the image is 4×4, corresponding to the database image data GD DB The number of pixels of the image is 2×2. That is, the query image data GD Q Including 4×4 pixel values, database image data GD DB Contains 2×2 pixel values.

[0356] exist Figure 24B In the query image data GD Q The included 4×4 pixel values ​​are pixel value vq11 to pixel value vq44. For example, in the query image data GD Q , the pixel value corresponding to the pixel in the 1st row and 1st column is pixel value vq11, the pixel value corresponding to the pixel in the 1st row and 4th column is pixel value vq14, the pixel value corresponding to the pixel in the 4th row and 1st column is pixel value vq41, and the pixel value corresponding to the pixel in the 4th row and 4th column is pixel value vq44. In addition, the database image data GD DB The included 2×2 pixel values ​​are pixel value vdb11, pixel value vdb12, pixel value vdb21, and pixel value vdb22. For example, in the database image data GD DB , the pixel value corresponding to the pixel in the 1st row and 1st column is pixel value vdb11, the pixel value corresponding to the pixel in the 1st row and 2nd column is pixel value vdb12, the pixel value corresponding to the pixel in the 2nd row and 1st column is pixel value vdb21, and the pixel value corresponding to the pixel in the 2nd row and 2nd column is pixel value vdb22.

[0357] First, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq11, vq12, vq21, and vq22. In this way, the database image data GD can be calculated. DB and query image data GD Q The consistency of the area composed of pixel value vq11, pixel value vq12, pixel value vq21 and pixel value vq22 in . Figure 24B The query image data GD is shown surrounded by a dotted line. Q The pixel values ​​included are consistent with the database image data GDDB The pixel values ​​to be compared are provided as a comparison data area 21 .

[0358] Next, for the query image data GD Q The included pixel values ​​are moved by the comparison data area 21 in a column, and the pixel values ​​are compared in the same manner to calculate the consistency. Specifically, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq12, vq13, vq22, and vq23. In this way, the database image data GD can be calculated. DB and query image data GD Q The consistency of the area composed of pixel value vq12, pixel value vq13, pixel value vq22 and pixel value vq23.

[0359] Then, for the query image data GD Q The included pixel values ​​are moved in a column-equivalent manner in the comparison data area 21, and the pixel values ​​are compared in the same manner to calculate the degree of consistency. Specifically, the pixel values ​​vdb11, vdb12, vdb21, and vdb22 are compared with the pixel values ​​vq13, vq14, vq23, and vq24. In this way, the database image data GD can be calculated. DB and query image data GD Q The consistency of the area composed of pixel value vq13, pixel value vq14, pixel value vq23 and pixel value vq24.

[0360] Next, for the query image data GD Q The pixel values ​​included in the comparison data area 21 are moved by one row, and the query image data GD is compared for each column in the same manner as described above. Q The pixel values ​​of the second row and the pixel values ​​of the third row constitute the database image data GD DB Thus, the query image data GD can be calculated for each column in the same manner as above. Q The area consisting of the pixel values ​​of the second and third rows and the database image data GD DB degree of consistency.

[0361] Then, for the query image data GD Q The pixel values ​​included in the comparison data area 21 are moved by one row, and the query image data GD is compared for each column in the same manner as described above. Q The pixel values ​​of the 3rd row and the pixel values ​​of the 4th row constitute the database image data GD DB Thus, the query image data GD can be calculated for each column in the same manner as above. QThe area consisting of the pixel values ​​of the 3rd and 4th rows and the database image data GD DB degree of consistency.

[0362] After the above work, for example, the highest consistency is the database image data GD DB For query image data GD Q The consistency of n database image data GD DB Then, similarly to step S31, the image data GD from the n databases is DB Extract and query image data GD Q Database image data GD with high consistency DB As the extracted image data GD Ex Note that, similarly to step S31, it is also possible not to extract the database image data GD DB .

[0363] Figure 25 It is the description database image data GD DB The extracted picture. Figure 25 It shows the image data GD from the database DB [1] To database image data GD DB [3] Extract one image data as extracted image data GD Ex example.

[0364] Corresponding to Figure 25 The query image data GD shown Q The image includes, for example, the symbol of a transistor and the symbol of a capacitor. Figure 25 The database image data GD shown DB The image of [2] includes the symbol of a transistor, and corresponds to the database image data GD DB [1] and the corresponding database image data GD DB The image of [3] does not include the symbols of transistors and capacitors. DB [2] for the query image data GD Q The consistency of the database image data GD DB [1] and database image data GD DB [3] for the query image data GD Q Therefore, the database image data GD can be extracted DB [2] As the extracted image data GD Ex .

[0365] Note that the query image data GD Q Image data with database GD DBThe comparison and calculation of the consistency can be performed using the same method as that used in step S31. Figure 24B For query image data GD Q The pixel values ​​included in the comparison data area are moved by one column or one row. Similar to step S31, the query image data GD Q The pixel values ​​included are moved by more than 2 columns or more than 2 rows in the comparison data area 21. Figure 20A Similarly to the case shown in FIG. 1 , the query image data GD input to the processing unit 13 may be used. Q , generate multiple query image data GD with different numbers of pixel values Q .

[0366] [Step S42]

[0367] Next, the processing unit 13 obtains the query image data GD Q Extraction of image data GD Ex Partial image data GD of the area with high consistency part-Q For example, in Figure 24B In the method shown, the query image data GD is calculated respectively. Q For each region of the database image data GD DB When the consistency is greater than , the area with the highest consistency is extracted as the partial image data GD part-Q Therefore, part of the image data GD part-Q The number of pixel values ​​included can be the same as the number of extracted image data GD Ex The number of included pixel values ​​is equal.

[0368] Figure 26 FIG. 4 is a diagram showing an example of the operation of step S42. Figure 26 In the case shown, the query image data GD Q The upper left portion of the image is for the extracted image data GD Ex [1] is the area with the highest consistency. Therefore, the query image data GD Q The data corresponding to the upper left area is the partial image data GD part-Q [1]. On the other hand, corresponding to the query image data GD Q The lower right part of the image is for the extracted image data GD Ex [2] is the area with the highest consistency. Therefore, the query image data GD Q The data corresponding to the lower right area is the partial image data GD part-Q [2]. That is, from 1 query image data GD Q Extract multiple partial image data GD part-Q .

[0369] In step S42, the query image data GD may also be Q Extract and extract image data GD Ex An equal number of image data is used as partial image data GD part-Q Alternatively, you can query the image data GD Q Extraction ratio extracted image data GD Ex A small number of image data is used as partial image data GD part-Q For example, in the case of multiple extracted image data GD Ex The query image data GD with high consistency Q When the area is the same, in the same area, from the query image data GD Q Extracted partial image data GD part-Q The number of can be 1. That is, it is also possible not to query the image data GD Q Extract multiple identical partial image data GD part-Q .

[0370] Note that it is also possible not to perform GD on some image data. part-Q At this time, by extracting part of the image data GD part-Q Appropriately replaced by query image data GD Q , the following description can be applied. Or, it can be said that the query image data GD Q The whole is the partial image data GD part-Q For example, by increasing or decreasing the query image data GD Q And extract image data GD Ex The number of pixel values ​​included in one or both of the image data is not affected by the partial image data GD part-Q The extraction may also be performed by performing an image retrieval method using the image retrieval system 10.

[0371] [Step S43]

[0372] Next, part of the image data GD part-Q And extract image data GD Ex The data is input to the neural network included in the processing unit 13 .

[0373] The operation of step S43 is to appropriately convert the query image data GD Q Alternatively called partial image data GD part-Q And part of the image data GD part Alternatively called extracted image data GD Ex Please refer to the description of step S33. Q Alternatively called extracted image data GDEx , part of the image data GD part Alternatively called partial image data GD part-Q .

[0374] The above is through the database image data GD DB The whole region and the query image data GD Q The database image data GD is calculated by comparing the area of ​​the database image data GD DB Query image data GD Q This is an example of an image retrieval method using the image retrieval system 10 when the similarity is .

[0375] exist Figure 23 In the method shown, the query image data GD is matched by region matching or the like. Q Image data with database GD DB Compare and extract the query image data GD Q A part of the database image data GD with high consistency DB As the extracted image data GD Ex Then, from the query image data GD Q The area with high consistency is extracted as the partial image data GD part-Q , part of the image data GD part-Q And extract image data GD Ex The image data GD is input to the neural network included in the processing unit 13. DB , can suppress representations that do not include data GD corresponding to the query image Q A database of images with high image consistency GD DB The image data GD corresponding to the query image can be retrieved with high accuracy and in a short time. Q The part of the image is similar to the database image. Note that in the query image data GD Q Database image data GD for comparison DB In the case of a small number of cases, even if the database image data GD is not extracted DB , the above search can also be performed with high accuracy and in a short time.

[0376] This embodiment mode can be appropriately combined with other embodiment modes. In addition, in this specification, when a plurality of structural examples are shown in one embodiment mode, the structural examples can be appropriately combined.

[0377] [Example]

[0378] In this embodiment, a description will be given of the result of inputting an image into an image retrieval system and retrieving database images similar to the image.

[0379] In this embodiment, a single image is input as a query image into the image retrieval system. Then, based on conditions 1 and 2, 100 database images similar to the query image are retrieved. In conditions 1 and 2, the query image is a schematic diagram of semiconductor manufacturing equipment. In addition to schematic diagrams of semiconductor manufacturing equipment, database images also include circuit diagrams, circuit layout diagrams, block diagrams, and the like. Furthermore, the database images are drawings described in patent documents.

[0380] Under condition 1, Figure 9 In steps S11 to S13, the similarity of the database image to the query image is calculated. Then, step S17 is performed to generate ranking data showing the database images ranked from the 1st to the 100th in order of the highest similarity.

[0381] Under condition 2, first, by Figure 2 In the steps S01 to S04 shown, a database tag is obtained from the specification in the same patent document as the patent document in which the database image is recorded. Figure 9 After calculating the similarity between the database image and the query image in steps S11 to S13, the similarity is corrected in steps S14 to S16. Then, step S17 is performed to generate ranking data showing the database images ranked from 1st to 100th in order of the highest similarity after correction.

[0382] Under condition 1, among the database images ranked 100th in similarity to the query image, 14 depict the same semiconductor manufacturing equipment as the query image, and the remaining 86 are images of circuit diagrams, circuit layouts, block diagrams, etc. On the other hand, under condition 2, all 100 images depict semiconductor manufacturing equipment.

[0383] From the above, it can be confirmed that more databases with concepts similar to the query image can be retrieved under condition 2 compared to condition 1.

[0384] [Explanation of symbols]

[0385] 10: Image retrieval system, 11: Input unit, 12: Transmission channel, 13: Processing unit, 15: Storage unit, 17: Database, 19: Output unit, 21: Comparison data area, 30: Neural network, 30a: Neural network, 31: Layer, 32: Neuron, 32a: Neuron, 32b: Neuron, 32c: Neuron, 40: Neural network

Claims

1. An image retrieval system comprising: database; Processing Department; as well as Input section, The database has the function of storing file data and multiple database image data. The processing unit has a function of acquiring database image feature data indicating a feature of each of the plurality of database image data. The processing unit has a function of generating a plurality of database tags using the document data and associating the database tags with the database image data. The plurality of database tags include words representing concepts of the database image data, The processing unit has a function of obtaining a database label vector representing each of the plurality of database labels. The processing unit has a function of acquiring query image feature data indicating a feature of the query image data when query image data is input to the input unit. The processing unit has a function of calculating a first similarity for each of the plurality of database image data, wherein the first similarity is a similarity of the database image data to the query image data. The processing unit has a function of obtaining a query label associated with the query image data using a part of the database label based on the first similarity. The processing unit has a function of obtaining a query tag vector representing the query tag. The processing unit has a function of obtaining first data including the database image feature data and the database label vector. The processing unit has a function of obtaining second data including the query image feature data and the query label vector. Furthermore, the processing unit has a function of calculating a second similarity, which is a similarity between the first data and the second data.

2. The image retrieval system according to claim 1, The database tags include words.

3. The image retrieval system according to claim 1, The processing unit has a function of generating the database tag by performing morphological analysis on the file data.

4. The image retrieval system according to claim 1, wherein the processing unit includes a first neural network and a second neural network, The database image feature data and the query image feature data are obtained using the first neural network. Furthermore, the database label vector and the query label vector are obtained using the second neural network.

5. The image retrieval system according to claim 4, The first neural network includes a convolutional layer and a pooling layer. Furthermore, the database image feature data and the query image feature data are output from the pooling layer.

6. The image retrieval system according to claim 4, The database label vector and the query label vector are both distribution representation vectors.

7. The image retrieval system according to claim 1, The first similarity and the second similarity are both cosine similarities.

8. An image retrieval method, comprising: using an image retrieval system including a database and an input unit, wherein the database stores document data and a plurality of database image data; and comprising: acquiring database image feature data representing a feature of each of the plurality of database image data; generating a plurality of database tags using the document data and associating the database tags with the database image data; Obtaining a database label vector representing each of the plurality of database labels; inputting query image data into the input unit; obtaining query image feature quantity data representing a feature quantity of the query image data; calculating a first similarity for each of the plurality of database image data, wherein the first similarity is a similarity of the database image data to the query image data; obtaining a query tag associated with the query image data using a portion of the database tags according to the first similarity; Obtaining a query label vector representing the query label; Obtaining first data including the database image feature data and the database label vector and second data including the query image feature data and the query label vector; and a function of calculating a second similarity, wherein the second similarity is the similarity of the first data to the second data, The plurality of database tags include words representing concepts of the database image data.

9. The image retrieval method according to claim 8, The database tags include words.

10. The image retrieval method according to claim 8, The database tag is generated by performing morphological analysis on the file data.

11. The image retrieval method according to claim 8, wherein the database image feature data and the query image feature data are obtained using a first neural network, Furthermore, the database label vector and the query label vector are obtained using a second neural network.

12. The image retrieval method according to claim 11, The first neural network includes a convolutional layer and a pooling layer. Furthermore, the database image feature data and the query image feature data are output from the pooling layer.

13. The image retrieval method according to claim 11, The database label vector and the query label vector are both distribution representation vectors.

14. The image retrieval method according to claim 8, The first similarity and the second similarity are both cosine similarities.

Images