Retrieval system and retrieval method

Abstract

To provide a retrieval system capable of retrieving images with similar concept to be represented.SOLUTION: The retrieval system comprises an input part, a sentence extraction part, a tag acquisition part, and a tag similarity calculation part. The sentence extraction part has a function of extracting, when image data with an image label and document data including an image label are supplied to the input part, tag acquisition sentence data from the document data on the basis of the image label. The tag acquisition part has a function of acquiring a tag including at least part of words contained in the tag acquisition sentence data. The tag similarity calculation part has a function of calculating similarity between tags. It is possible to retrieve images with similar concepts to be represented though the feature quantity of the image per se increases.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] One aspect of the present invention relates to a search system and a search method.

[0002] It should be noted that one aspect of the present invention is not limited to the above-mentioned technical field. Examples of technical fields of one aspect of the present invention include semiconductor devices, display devices, light-emitting devices, energy storage devices, memory devices, electronic devices, lighting devices, methods for driving them, or methods for manufacturing them. [Background technology]

[0003] By conducting a prior art search on an invention before filing an application, it is possible to investigate whether or not related intellectual property rights exist. The prior art documents obtained through the prior art search, such as domestic and international patent documents and papers, can be used to confirm the novelty and inventiveness of the invention, and to decide whether or not to file a patent application. Furthermore, by conducting an invalidation search of prior art documents, it is possible to investigate whether there is a risk of one's own patent rights being invalidated, or whether it is possible to invalidate the patent rights of others.

[0004] For example, the above-mentioned prior art search can be conducted by searching for prior art documents that contain drawings similar to drawings that embody the technology before filing. Specifically, for example, a user can input drawings into a search system to search for prior art documents that contain drawings similar to the input drawings.

[0005] Searching for images similar to an input image can be done, for example, using a neural network. For instance, Patent Document 1 discloses a method for determining the similarity between images using a neural network. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-207947 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] If the similarity between an input image and a target image is calculated using only image data, it may result in the search yielding images that represent concepts different from the input image. This can lead to the inclusion of noisy images in the search results, potentially preventing the desired image from being displayed. Therefore, the accuracy of similar image searches may be low.

[0008] Therefore, one aspect of the present invention aims to provide a search system that can search for images that represent similar concepts. Alternatively, one aspect of the present invention aims to provide a search system that can perform searches easily. Alternatively, one aspect of the present invention aims to provide a novel search system.

[0009] Alternatively, one aspect of the present invention aims to provide a search method that can search for drawings that represent similar concepts. Alternatively, one aspect of the present invention aims to provide a search method that can perform searches easily. Alternatively, one aspect of the present invention aims to provide a novel search method.

[0010] Furthermore, the description of these problems does not preclude the existence of other problems. One aspect of the present invention does not necessarily have to solve all of these problems. It is possible to extract other problems from the description in the specification, drawings, and claims. [Means for solving the problem]

[0011] One aspect of the present invention is a search system comprising an input unit, a text extraction unit, a tag acquisition unit, and a tag similarity calculation unit, wherein the text extraction unit has the function of extracting database text data for tag acquisition from the database document data based on the database image labels when database image data with database image labels and database document data including the database image labels are supplied to the input unit, the text extraction unit has the function of extracting text data for tag acquisition from the document data based on the image labels when image data with image labels and document data including the image labels are supplied to the input unit, the tag acquisition unit has the function of acquiring database tags that include at least some of the words contained in the tag acquisition database text data, and the tag similarity calculation unit has the function of calculating the similarity of database tags to tags.

[0012] Alternatively, in the above embodiment, the text extraction unit may have the function of extracting at least a portion of paragraphs containing database image labels from among the paragraphs included in the database document data as first database texts, and using the first database texts as database text data for tag acquisition.

[0013] Alternatively, in the above embodiment, the text extraction unit may have a function to extract paragraphs from the database document data that begin with a database image label as a first database document, and the text extraction unit may have a function to extract paragraphs from the document data that begin with an image label as a first document.

[0014] Alternatively, in the above embodiment, the text extraction unit may have a function to extract at least a portion of the paragraphs in the database document data that contain the quoted words in the first database document as a second database document, and include the second database document in the tag acquisition database document data, and the text extraction unit may have a function to extract at least a portion of the paragraphs in the document data that contain the quoted words in the first document as a second document, and include the second document in the tag acquisition document data.

[0015] Alternatively, in the above embodiment, the text extraction unit may have a function to extract paragraphs from the database document data that include a coded word from the first database document at the beginning and are within a predetermined number of paragraphs from the first database document as a second database document, and the text extraction unit may also have a function to extract paragraphs from the document data that include a coded word from the first document at the beginning and are within a predetermined number of paragraphs from the first document as a second document.

[0016] Alternatively, in the above embodiment, the text extraction unit may have the function of extracting paragraphs from the database document data that are within a predetermined number of paragraphs from the paragraphs included in the tag acquisition database document data, and that begin with a parallel conjunction, as a third database document, and including the third database document in the tag acquisition database document data. The text extraction unit may also have the function of extracting paragraphs from the document data that are within a predetermined number of paragraphs from the paragraphs included in the tag acquisition document data, and that begin with a parallel conjunction, as a third document, and including the third document in the tag acquisition document data.

[0017] Alternatively, in the above embodiment, the text extraction unit has a function to extract text data for tag acquisition using a machine learning model based on image data and document data, and the machine learning model may be a model trained using training image data to which training image labels have been assigned and training document data to which training image labels have been included.

[0018] Alternatively, in the above embodiment, the text represented by the training document data may be assigned a tag label indicating whether or not it is a text to be used for tag acquisition.

[0019] Alternatively, in the above embodiment, tag labels may be assigned based on training image labels.

[0020] Alternatively, in the above embodiment, tag labels may be assigned to each paragraph included in the training document data.

[0021] Alternatively, one aspect of the present invention is a method for searching database images to which database tags containing words have been assigned, wherein when image data to which an image label has been assigned and document data to which the image label has been assigned are input, the method extracts tag acquisition text data from the document data based on the image label, acquires tags that include at least some of the words contained in the tag acquisition text data, and calculates the similarity of the database tags to the tags.

[0022] Alternatively, in the above embodiment, at least a portion of the paragraphs containing image labels from among the paragraphs included in the document data may be extracted as a first sentence, and the first sentence may be used as text data for tag acquisition.

[0023] Alternatively, in the above embodiment, paragraphs included in the document data that begin with an image label may be extracted as the first sentence.

[0024] Alternatively, in the above embodiment, at least a portion of the paragraphs included in the document data that contain the quoted words in the first sentence may be extracted as a second sentence, and the second sentence may be included in the document data for tag acquisition.

[0025] Alternatively, in the above embodiment, paragraphs included in the document data that begin with a punctuated word contained in the first text and are within a predetermined number of paragraphs from the first text may be extracted as the second text.

[0026] Alternatively, in the above embodiment, paragraphs included in the document data that are within a predetermined number of paragraphs from the paragraphs included in the tag acquisition text data and that begin with a parallel conjunction may be extracted as a third sentence, and the third sentence may be included in the tag acquisition text data.

[0027] Alternatively, in the above embodiment, a machine learning model may extract text data for tag acquisition based on image labels and document data, and the machine learning model may be a model trained using training image labels and training document data containing the training image labels.

[0028] Alternatively, in the above embodiment, the text represented by the training document data may be assigned a tag label indicating whether or not it is a text to be used for tag acquisition.

[0029] Alternatively, in the above embodiment, tag labels may be assigned based on training image labels.

[0030] Alternatively, in the above embodiment, tag labels may be assigned to each paragraph included in the training document data. [Effects of the Invention]

[0031] According to one aspect of the present invention, a search system can be provided that can search for images that represent similar concepts. Alternatively, according to one aspect of the present invention, a search system can be provided that allows for easy searching. Alternatively, according to one aspect of the present invention, a novel search system can be provided.

[0032] Alternatively, according to one aspect of the present invention, a search method can be provided that allows searching for drawings that represent similar concepts. Alternatively, according to one aspect of the present invention, a search method can be provided that allows searching to be performed easily. Alternatively, according to one aspect of the present invention, a novel search method can be provided.

[0033] Furthermore, the description of these effects does not preclude the existence of other effects. One aspect of the present invention does not necessarily have to possess all of these effects. Other effects can be extracted from the description, drawings, and claims. [Brief explanation of the drawing]

[0034] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of a search system. [Figure 2] Figure 2 is a flowchart showing an example of how to retrieve database tags. [Figure 3] Figure 3 is a schematic diagram showing an example of literature data. [Figure 4] Figures 4(A), 4(B1), and 4(B2) are schematic diagrams illustrating an example of a method for extracting text data for tag acquisition. [Figure 5] Figures 5(A) and 5(B) are schematic diagrams showing examples of tags. [Figure 6] Figure 6 is a flowchart showing an example of a search method. [Figure 7] Figure 7 is a schematic diagram illustrating an example of a classifier learning method. [Figure 8] Figures 8(A) and 8(B) are schematic diagrams illustrating an example of a method for obtaining text data for tag acquisition. [Figure 9] Figures 9(A) and 9(B) are schematic diagrams illustrating an example of how tags are obtained. [Figure 10] Figure 10 shows an example of a search system. [Modes for carrying out the invention]

[0035] Embodiments will be described in detail with reference to the drawings. However, it will be readily apparent to those skilled in the art that the present invention is not limited to the following description, and that its form and details can be modified in various ways without departing from the spirit and scope of the present invention. Accordingly, the present invention shall not be construed as being limited to the descriptions of the embodiments shown below.

[0036] (Embodiment 1) In this embodiment, a search system and a search method according to one aspect of the present invention will be described with reference to the drawings.

[0037] One aspect of the present invention relates to a search system and a search method that, when inputting bibliographic data including image data and document data, outputs database image data that is similar to the image data specified by the user in terms of the concepts, technical content, or points of interest represented. In the search system of one aspect of the present invention, a database is used in which multiple bibliographic data (database bibliographic data) including database image data and database document data are registered. Here, the image data (database image data) is assigned an image label (database image label). Here, the image label (database image label) represents, for example, a figure number. For example, the database image label assigned to the database image data representing the image of Figure 1 included in the database bibliographic data is "Figure 1".

[0038] From the database document data, database tags are obtained based on the database image labels, and these database tags are attached to the database image data and registered in the database. In one embodiment of the present invention, the search system can obtain tags (database tags) without using the feature quantities of the image itself represented by the image data (database image data). The database tags can be, for example, a set of nouns obtained by performing morphological analysis on the text obtained from the database document data based on the database image labels.

[0039] In a search method using a search system according to one embodiment of the present invention, when bibliographic data including image data and document data is input to the search system, tags are obtained in the same manner as in the method for obtaining database tags. Next, the similarity of the database tags to the obtained tags is calculated. Subsequently, information relating to database image data to which database tags with high similarity are assigned is output. Information relating to database bibliographic data containing the database image data can also be output. As a result, a user of the search system according to one embodiment of the present invention can search for database image data with high similarity to the input image data, and database bibliographic data containing the database image data.

[0040] By the method described above, a search system according to one embodiment of the present invention can search a database of image data for images that, although their feature quantities differ significantly, have similar characteristics such as the concepts, content, or points of interest they represent. Furthermore, compared to, for example, a user of the search system specifying all the words to be included in the tags to be assigned to the image data and not presenting the user with candidate words to be included in the tags, the method above allows for the acquisition of tags that comprehensively include words representing the concepts, content, or points of interest represented by the image data. Therefore, a search system according to one embodiment of the present invention can perform searches in a simple manner.

[0041] In this specification, a “word” is a linguistic unit composed of one or more morphemes. Words are grouped by part of speech. Parts of speech include nouns, verbs, adjectives, adverbs, and conjunctions.

[0042] In this specification, the user of a device or equipment on which a system such as a search system is installed is simply referred to as the "system user." For example, the user of an information processing device on which a search system is installed is referred to as the search system user.

[0043] <Search System_1> Figure 1 is a block diagram showing an example configuration of the search system 10. The search system 10 includes an input unit 11, a database 13, a processing unit 20, and an output unit 15. The processing unit 20 includes a text extraction unit 21, a tag acquisition unit 23, and a tag similarity calculation unit 25.

[0044] In Figure 1, the exchange of data and other information between the components of the search system 10 is indicated by arrows. Note that the data exchange shown in Figure 1 is just one example; for example, data may be exchanged between components not connected by arrows. Furthermore, even between components connected by arrows, data may not be exchanged. The same applies to block diagrams other than Figure 1.

[0045] The search system 10 may be installed on an information processing device such as a personal computer (PC). Alternatively, the processing unit 20 and database 13 of the search system 10 may be installed on a server, and it may be used by accessing it from a client PC via a network. Note that the database 13 does not have to be included in the search system 10. For example, if the input unit 11, processing unit 20, and output unit 15 are installed on a client PC, and the database 13 is installed on a server, the search system 10 can be said to be installed on the client PC.

[0046] In Figure 1, for example, the components are classified by function and shown as independent blocks in the block diagram. However, in reality, it is difficult to completely separate the components by function, and one component may be involved in multiple functions. Also, one function may be involved in multiple components; for example, multiple processes performed in the processing unit 20 may be executed by different servers. Furthermore, some of the multiple processes performed in the processing unit 20 may be performed on the client PC, while the remaining processes are performed on the server.

[0047] [Input section 11] Data is supplied to the input unit 11 from outside the search system 10. The data supplied to the input unit 11 is then supplied to the processing unit 20. Alternatively, the data supplied to the input unit 11 is registered in the database 13.

[0048] The input unit 11 can accept, for example, image data and document data, including bibliographic data. The document data includes text that describes the image data. Here, the image data is assigned an image label. The image label represents, for example, the figure number. For example, the image label assigned to the image data representing Figure 1 included in the bibliography is "Figure 1".

[0049] Examples of document data include documents related to patent applications and documents related to utility model registration applications. In this case, drawings can be provided as image data, and specifications can be provided as document data.

[0050] [Database 13] Database 13 has the function of storing data to be searched. Database 13 contains, for example, multiple bibliographic records. In this specification, storing data in a database is referred to as registering data in a database. For example, supplying bibliographic data to a database for storage is referred to as registering bibliographic data in a database, or registering a bibliography in a database.

[0051] In this specification, bibliographic data registered in a database is referred to as database bibliographic data. Furthermore, image data, document data, and image labels included in the database bibliographic data are referred to as database image data, database document data, and database image labels, respectively.

[0052] Examples of database document data include documents related to applications. Examples of applications include patent applications and utility model registration applications, and other intellectual property applications. There are no restrictions on the status of each application, and it does not matter whether it has been published, is pending at the Japan Patent Office, or is registered. Documents that are not yet filed, are not yet examined, are under examination, or are already registered can all be registered in Database 13.

[0053] Furthermore, database 13 may contain at least one of the following: application management number (including a company-specific number) for identifying an application, application family management number for identifying an application family, application number, publication number, registration number, drawings, abstract, filing date, priority date, publication date, status, classification (patent classification or utility model classification, etc.), category, and keywords. Each of these pieces of information may be used to identify the database document data when supplying database document data to the input unit 11. Alternatively, each of these pieces of information may be output to the output unit 15 along with the processing results of the processing unit 20.

[0054] In addition, various types of literature, such as books, magazines, newspapers, and academic papers, can be registered in the database 13. Furthermore, literature describing industrial products can be registered in the database 13. For example, a photograph of an industrial product or a drawing showing an industrial product can be used as image data, and data including text describing the photograph or drawing can be used as document data. In the above case, the database 13 may also register at least one of the following for each document: an identification number, title, publication date, author, and publisher. This information may be used to identify the database document data when supplying the database document data to the input unit 11. Alternatively, this information may be output to the output unit 15 along with the processing results of the processing unit 20.

[0055] The literature supplied to the input unit 11 can be of the same type as the literature that can be used as the database literature data described above. For example, if the database literature data is literature related to a patent application, the literature data supplied to the input unit 11 can also be literature related to a patent application.

[0056] Database 13 can register data obtained by processing the database bibliographic data performed by the processing unit 20. For example, data obtained by processing the database bibliographic data by the text extraction unit 21 and the tag acquisition unit 23 of the processing unit 20 can be registered in database 13.

[0057] [Processing 20] The processing unit 20 has the function of performing calculations and other processing using data supplied from the input unit 11 or the database 13, etc. The processing results, that is, the data generated by the calculations and other processing, can be supplied to the database 13 or the output unit 15, etc.

[0058] The processing unit 20 may have, for example, a central processing unit (CPU). The processing unit 20 may also have a microprocessor such as a DSP (Digital Signal Processor) and a GPU (Graphics Processing Unit). The microprocessor may be implemented using a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) and an FPAA (Field Programmable Analog Array). The processing unit 20 can perform various data processing and program control by interpreting and executing instructions from various programs via the processor. Programs that can be executed by the processor are stored, for example, in the memory area of ​​the processor.

[0059] The processing unit 20 may have main memory. The main memory includes at least one of volatile memory such as RAM (Random Access Memory) and non-volatile memory such as ROM (Read Only Memory).

[0060] For RAM, for example, DRAM and SRAM are used, and a memory space is virtually allocated and used as the workspace for the processing unit 20.

[0061] ROM can store BIOS (Basic Input / Output System) and firmware, etc., which do not require rewriting. Examples of ROM include mask ROM, OTPROM (One Time Programmable Read Only Memory), and EPROM (Erasable Programmable Read Only Memory). Examples of EPROM include UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory), which allows data to be erased by ultraviolet irradiation, EEPROM (Electrically Erasable Programmable Read Only Memory), and flash memory.

[0062] The components of the processing unit 20 will be described below.

[0063] ≪Text extraction part 21≫ The text extraction unit 21 has the function of extracting text from document data based on image labels, for example, when literature data including image data with image labels such as figure numbers and document data is supplied to the processing unit 20. Specifically, it has the function of extracting text that describes the image data from the document data.

[0064] ≪Tag Acquisition Section 23≫ The tag acquisition unit 23 has the function of acquiring tags that include at least some of the words contained in the text extracted by the text extraction unit 21. For example, the tag acquisition unit 23 can perform morphological analysis on the text extracted by the text extraction unit 21 and acquire a set of nouns contained in the morphologically analyzed text as tags. Morphological analysis divides a text written in natural language into morphemes (the smallest units that have meaning as language), and can, for example, determine the part of speech of the morphemes.

[0065] As described above, tags are obtained based on the text extracted by the text extraction unit 21. Therefore, in this specification, the text extracted by the text extraction unit 21 is referred to as tag acquisition text data. Furthermore, when the text extraction unit 21 extracts text from database document data, the extracted text is referred to as tag acquisition database text data. In addition, tags obtained based on tag acquisition database text data are referred to as database tags. Database tags obtained by the tag acquisition unit 23 can be attached to database document data and registered in the database 13.

[0066] <<Tag Similarity Calculation Unit 25>> The tag similarity calculation unit 25 has the function of calculating the similarity between database tags and tags obtained from document data supplied to the input unit 11. The similarity can be calculated using, for example, the Jaccard coefficient, the Dice coefficient, or the Simpson coefficient. Alternatively, by vectorizing the words contained in the database tags and the words contained in the tags obtained from document data supplied to the input unit 11, the similarity can be calculated using cosine similarity, covariance, unbiased covariance, Pearson's product-moment correlation coefficient, or deviation pattern similarity. For word vectorization, for example, open-source algorithms such as Word2vec, BoW (Bag of Words), or BERT (Bidirectional Encoder Representations from Transformers) can be used.

[0067] [Output section 15] The output unit 15 has the function of supplying information to the outside of the search system 10. This information can be search results. The output unit 15 has the function of supplying information relating to database bibliographic data to the outside of the search system 10, for example, based on the similarity score. The output unit 15 has the function of supplying information relating to database bibliographic data to the outside of the search system 10, for example, based on the database tags with a high similarity score. The information supplied by the output unit 15 to the outside of the search system 10 can be displayed, for example, by a display device provided outside the search system 10. This allows the information acquired by the processing unit 20 to be presented to the user of the search system 10.

[0068] For example, the search system 10 can present to the user of the search system 10 database image data to which database tags with a similarity score of a predetermined value or higher have been assigned, and database bibliographic data containing said database image data. Alternatively, the search system 10 can extract a predetermined number of database tags, counting from those with the highest similarity scores, and present to the user of the search system 10 database image data to which said database tags have been assigned, and database bibliographic data containing said database image data. For example, the search system 10 may include a display device that presents information to the user of the search system 10. Furthermore, the selection of information to be supplied to the outside of the search system 10 by the output unit 15 may be performed by the output unit 15 or by the processing unit 20. If the processing unit 20 performs the selection, for example, the tag similarity calculation unit 25 may perform the selection.

[0069] As described above, when the search system 10 receives bibliographic data containing image data and document data from the input unit 11, it extracts sentences describing the image data from the document data. Next, the search system 10 obtains a set of words contained in the extracted sentences as tags. Subsequently, the search system 10 calculates the similarity between these tags and database tags obtained in the same manner as these tags.

[0070] As described above, the search system 10 can search the database image data for images that, while differing significantly in the feature quantities of the images themselves, have similar characteristics such as the concepts, technical content, or points of interest they represent. Furthermore, it can search the database document data for documents containing such images. Therefore, the search system 10 can, for example, search for patent documents, papers, or industrial products related to or similar to an invention prior to filing. This allows for prior art searches related to the invention prior to filing. By understanding and re-examining relevant prior art, the invention can be strengthened and made into a strong patent that is difficult for competitors to circumvent.

[0071] Furthermore, for example, the search system 10 can be used to search for patent documents, papers, or industrial products related to or similar to industrial products before their release. For example, if the database of literature data includes the company's own patent documents, it can be confirmed whether the technology related to the industrial product before its release has been sufficiently patented within the company. Alternatively, if the database of literature data includes information on another company's intellectual property, it can be confirmed whether the industrial product before its release infringes on the other company's intellectual property rights. By understanding the relevant prior art and re-examining the technology related to the industrial product before its release, it is possible to discover new inventions and develop them into strong patentable inventions that contribute to the company's business. Note that the search may be limited to industrial products after their release, not just before.

[0072] Furthermore, for example, the search system 10 can be used to search for patent documents, papers, or industrial products related to or similar to a specific patent. In particular, by searching based on the filing date of the patent, it is possible to easily and accurately investigate whether the patent contains any grounds for invalidation.

[0073] Furthermore, compared to a scenario where, for example, a user of the search system specifies all the words to be included in the tags to be attached to the image data and no candidate words to be included in the tags are presented to the user, the search system 10 can acquire tags that comprehensively include words representing the concepts, content, or points of interest that the image data represents. Therefore, the search system 10 can perform searches in a simple manner.

[0074] <Search method> Hereinafter, an example of a search method using the search system 10 will be described. Specifically, an example of a method for searching for an image similar to the image data input to the search system 10 among the database image data, with features such as the represented concept, technical content, or points of interest, will be described.

[0075] [Obtaining database tags] FIG. 2 is a flowchart showing an example of a method for obtaining database tags using database document data. To obtain database tags, first, as shown in step S01, the database image data GD DB and the database document data DD DB are input. Specifically, the database document data including the database image data GD DB and the database document data DD DB is supplied to the input unit 11. The database document data supplied to the input unit 11 is supplied to the text extraction unit 21.

[0076] FIG. 3 is a schematic diagram showing a configuration example of the database document data LD DB supplied to the input unit 11 in step S01. In FIG. 3, an example of supplying the database document data LD DB [1] to the database document data LD DB [n] (n is an integer of 1 or more) to the input unit 11 is shown.

[0077] In this specification and the like, when the same reference numerals are used for a plurality of elements, particularly when it is necessary to distinguish them, identification symbols such as “[ ]” or “( )” may be appended to the reference numerals for description.

[0078] The database document data LD DB includes the database image data GD DB and the database document data DD DB For example, the database document data LD DBWhen the documents pertaining to a patent application or utility model registration application are used, the drawings are used as database image data (GD). DB This can be done, and the specifications can be stored in the database document data DD. DB This can be done. Database image data GD DB This includes database image labels GL DB As shown in Figure 3, the database image label GL is assigned. DB This can be, for example, a figure number. Also, the database image label GL DB This may include, for example, alphabets, Greek letters, kana, and other characters. Furthermore, the database image label GL DB This may include symbols such as ( ). For example, “Figure 1(a)” may be used as the database image label GL. DB It can be done this way.

[0079] Figure 3 shows the database bibliographic data LD. DB [1] Includes database image data GD DB As such, database image data GD DB (1) and database image data GD DB (2) is shown. And, for example, database image data GD DB (1) Database image label GL DB (1) is assigned, database image data GD DB (2) Database image label GL DB (2) is assigned. In other words, one database bibliographic data LD DB GD is a database image data of multiple databases. DB It can include multiple database image data GD DB Each of them has a database image label GL DB It is possible to assign this. For example, if Figure 1 is divided into Figure 1(a) and Figure 1(b), then Figure 1(a) and Figure 1(b) can be assigned to different database image data GD. DB And each of them is assigned a database image label GL DBIt is possible to assign a different database image label GL to "Figure 1(a)" and "Figure 1(b)". DB It can be done this way.

[0080] Database Document Data DD DB This is a database image data GD DB Includes explanatory text. Database document data DD DB This includes database image labels GL DB This includes the database document data DD. DB The text contained within can be divided into multiple paragraphs. Figure 3 shows the database document data DD. DB In the database, image labels GL DB (1) The “Figure 1” is included in paragraph

[0001] , and the database image label GL DB (2) This shows an example in which "Figure 2" is included in paragraph

[0002] .

[0081] Next, as shown in step S02 in Figure 2, the text extraction unit 21 extracts the database image data GD DB The text explaining this is a database document (DD). DB Extract from. The extracted text is stored in the tag acquisition database (text data TTD). DB Let's assume that.

[0082] Figure 4(A) shows the document data (TTD) in the database for tag acquisition. DB This is a schematic diagram illustrating an example of how to obtain the data. Figures 4(B1) and 4(B2) show the document data TTD in the database for tag acquisition. DB This is a schematic diagram illustrating an example. Figures 4(A), 4(B1), and 4(B2) show database bibliographic data LD. DB [1] Includes database image data GD DB (1) and database image data GD DB (2) For each of these, the tag acquisition database document data TTD DB This shows an example of extracting data.

[0083] Tag retrieval database document data TTD DB This is a database image label GL DB Extraction can be performed based on the following: For example, database document data DD DB Among the paragraphs included, database image label GL DB At least a portion of the paragraph containing the tag is included in the tag retrieval database document data TTD. DB It can be included in the database image label GL. DB Paragraphs containing sentences with the subject as the subject are included in the tag acquisition database document data TTD. DB It can be included in the database image label GL. DB Paragraphs containing this as the first sentence, i.e., the opening sentence, are included in the tag acquisition database document data TTD. DB It can be included in the database image label GL. DB Paragraphs that begin with this tag are included in the tag acquisition database document data TTD DB It can be included.

[0084] In the example shown in Figure 4(A), the beginning of paragraph [0aa1] contains the database image label GL. DB (1) Since it contains "Figure 1", paragraph [0aa1] is the document data TTD in the tag acquisition database. DB (1) can be included. Also, at the beginning of paragraph [0bb1], the database image label GL DB (2) Since it contains "Figure 2", paragraph [0bb1] is the document data TTD in the tag acquisition database. DB (2) can be included. For example, database image data GD DB (1) Tag acquisition database document data TTD DB Tagged document data (TTD) DB (1) Database image data GD DB (2) Tag acquisition database document data TTD DB Tagged document data (TTD) DB (2)

[0085] In this specification, etc., database image label GL DB Based on database document data DD DB The text extracted from this source is sometimes referred to as the first database text.

[0086] Also, the tag acquisition database document data TTD DB This may be extracted based on the signed words contained in the first database document. For example, at least a portion of the paragraphs containing the signed words in the first database document may be extracted from the tag acquisition database document data TTD DB It may also be included. For example, a paragraph containing a sentence with a signed word as its subject in the first database document may be included in the tag acquisition database document data TTD. DB It may also be included. For example, a paragraph that contains a quoted word from the first database document in its first sentence, i.e., the opening sentence, may be included in the tag acquisition database document data TTD. DB It may also be included. For example, a paragraph containing a signed word from the first database document at the beginning may be included in the tag acquisition database document data TTD. DB It may be included.

[0087] In the example shown in Figure 4(A), the database image data GD DB Paragraph [0aa1], which is the first database document corresponding to (1), contains the coded word “display device 10000”. Also, the database image data GD DB Paragraph [0bb1], which is the first database document corresponding to (2), contains the coded word “transistor 10011”. Here, for example, in the word “display device 10000”, “10000” is the coded word, and in the word “transistor 10011”, “10011” is the coded word. The coded word may include alphabets, Greek letters, kana, or other characters.

[0088] The database image data GD shown in Figure 4(A) DB Therefore, for the sake of simplifying the diagrams, the symbols that would normally be included in the diagrams have been omitted. The same applies to subsequent diagrams that show image data.

[0089] At the beginning of paragraph [0aa2], the database image data GD DB (1) contains the tagged word "display device 10000" included in the first database sentence corresponding thereto. Therefore, paragraph [0aa2] is the tag acquisition database sentence data TTD DB (1) can be included. Also, at the beginning of paragraph [0bb2], the database image data GD DB (2) contains the tagged word "transistor 10011" included in the first database sentence corresponding thereto. Therefore, paragraph [0bb2] is the tag acquisition database sentence data TTD DB (2) can be included.

[0090] In this specification and the like, a sentence extracted from the database document data DD DB based on a tagged word may be referred to as a second database sentence.

[0091] Also, the tag acquisition database sentence data TTD DB may be extracted based on the tagged word included in the second database sentence. For example, the tag acquisition database sentence data TTD DB is extracted in the same manner as the method of extracting based on the tagged word included in the first database sentence, and the tag acquisition database sentence data TTD DB may be extracted based on the tagged word included in the second database sentence.

[0092] In the example shown in FIG. 4(A), paragraph [0aa2], which is the second database sentence corresponding to the database image data GD DB (1), contains the tagged word "pixel 10001". And at the beginning of paragraph [0aa3], "pixel 10001" is included. From the above, paragraph [0aa3] is the second database sentence corresponding to the database image data GD DB (1) and can be included as the tag acquisition database sentence data TTD DB (1).

[0093] Also, for example, even in a paragraph that includes at the beginning a tagged word included in the first database text or the second database text, a paragraph that is separated from the first database text by a predetermined number of paragraphs or more may not be included in the second database text. That is, for example, a paragraph that includes a tagged word included in the first database text or the second database text and is within less than a predetermined number of paragraphs from the first database text may be included in the second database text.

[0094] Even in a paragraph that includes at the beginning a tagged word included in the first database text or the second database text, for the database image label GL DB in the database document data DD DB extracted from the first database text which is a text far from the first database text, the explanation of the database image data GD DB corresponding to the database image label GL DB may not be sufficiently provided. Therefore, by not including paragraphs that are separated from the first database text by a predetermined number of paragraphs or more in the database text data for tag acquisition TTD DB it is possible to suppress a text in which the explanation of the database image data GD DB is not sufficiently provided from being included in the database text data for tag acquisition TTD DB As a result, in the process described later, the database image data GD DB whose represented concept, technical content, or features such as points of interest are similar to the image data input to the search system 10 can be searched with high accuracy.

[0095] Furthermore, the predetermined number of paragraphs can be different for paragraphs containing a coded word included in the first database document and paragraphs containing a coded word that is not included in the first database document but is included in the second database document. For example, the predetermined number of paragraphs in paragraphs containing a coded word included in the first database document can be greater than the predetermined number of paragraphs in paragraphs containing a coded word that is not included in the first database document but is included in the second database document. For example, consider the example shown in Figure 4(A), where the predetermined number of paragraphs in paragraphs containing a coded word included in the first database document is 7, and the predetermined number of paragraphs in paragraphs containing a coded word that is not included in the first database document but is included in the second database document is 5. In this case, if the coded word "display device 10000" included in paragraph [0aa1], which is the first database document, is included in the subject of a sentence constituting paragraph [0aa7] (not shown), then paragraph [0aa7] is used as the second database document for tag acquisition database document data TTD. DB (1) can be included. On the other hand, if the coded word “pixel 10001” which is not included in paragraph [0aa1] but is included in the second database document paragraph [0aa2] is included in the subject of a sentence that makes up paragraph [0aa7], but “display device 10000” is not included in the subject of a sentence that makes up paragraph [0aa7], then paragraph [0aa7] can be included in the tag acquisition database document data TTD DB (1) may be omitted.

[0096] Furthermore, the predetermined number of paragraphs may differ for each method of extracting the second database document. For example, a first threshold and a second threshold may be set, with the second threshold being greater than the first threshold. Paragraphs that are less than the first threshold away from the paragraph containing the first database document may be included in the second database document if they contain, for example, all paragraphs containing the signed word included in the first database document or the second database document. Paragraphs that are more than the first threshold but less than the second threshold away from the paragraph containing the first database document may be included in the second database document if they begin with, for example, a signed word included in the first database document or the second database document. Paragraphs that are more than the second threshold away from the paragraph containing the first database document may not be included in the second database document even if they begin with a signed word included in the first database document or the second database document.

[0097] Furthermore, the above code used for extracting the second database document is used in the database image data GD DB It may also be extracted from the database image data GD. For example, the text extraction unit 21 may extract from the database image data GD. DB The code is read from the document data TTD in the tag acquisition database, and at least a portion of the paragraph containing the coded word is extracted. DB It may also be included. For example, a paragraph containing a sentence with the punctuated word as its subject may be included in the tag acquisition database document data TTD. DB It may also be included. For example, a paragraph that contains the quoted word in the first sentence, i.e., the opening sentence, may be included in the tag acquisition database document data TTD. DB It may also be included. For example, paragraphs that begin with the quoted word may be included in the tag acquisition database document data TTD. DB It may also be included. Furthermore, the code used for extracting the second database document is the database image data GD DB When extracting from the above-mentioned signed words, use the database document data DD. DB Extraction from is not necessary.

[0098] Furthermore, for example, at least a portion of a paragraph containing a specific word is used in the tag acquisition database document data TTD. DB It may be included in the following: For example, a parallel conjunction such as "also" at the beginning, and paragraphs within a predetermined number of paragraphs from the first database document or the second database document, are included in the tag acquisition database document data TTD. DB It may be included.

[0099] In the example shown in Figure 4(A), the conjunction "also" is included at the beginning of both paragraph [0aa4] and paragraph [0bb3]. Paragraph [0aa4] is also part of the document data TTD in the tag acquisition database. DB Paragraphs [0aa1] through [0aa3] included in (1) are close to the document data TTD in the tag acquisition database. DB Paragraphs [0bb1] and [0bb2] included in (2) are separate. Therefore, paragraph [0aa4] is in the tag acquisition database document data TTD DB (1) can be included in paragraph [0bb3]. Also, paragraph [0bb3] is the document data TTD in the tag acquisition database. DB Paragraphs [0bb1] and [0bb2] included in (2) are close to the tag acquisition database document data TTD DB Paragraphs [0aa1] through [0aa3] included in (1) are separate. Therefore, paragraph [0bb3] is in the tag acquisition database document data TTD DB (2) can be included in (2).

[0100] In this specification, etc., database document data DD is based on words that are not labeled, such as parallel conjunctions. DB The text extracted from this source is sometimes referred to as a third type of database text.

[0101] Also, the tag acquisition database document data TTD DB This may or may not be extracted based on the signed words contained in the third database document. (Tag acquisition database document data TTD) DBWhen extracting text based on coded words contained in a third database document, the extracted text can be included, for example, in a second database document.

[0102] By the above method, the text extraction unit 21 extracts the database image data GD DB The text explaining this is a database document (DD). DB Extracted from, the document data TTD for tag acquisition database. DB This can be obtained. Figure 4(B1) shows the database image data GD DB (1) Tag acquisition database document data TTD DB (1) is a schematic diagram illustrating an example, and Figure 4(B2) shows the database image data GD DB (2) Tag acquisition database document data TTD DB (2) This is a schematic diagram showing an example. As shown in Figure 4(B1), paragraphs [0aa1] to [0aa4] are used in the tag acquisition database document data TTD DB (1) can be done. Also, as shown in Figure 4(B2), paragraphs [0bb1] to [0bb3] are used in the tag acquisition database document data TTD DB (2) This can be done.

[0103] Note: The document data (TTD) is used in the database for tag acquisition. DB An example of extracting data paragraph by paragraph has been shown, but the present invention is not limited to this. For example, database document data DD DB For each sentence contained within, the tag acquisition database document data TTD DB You may also extract the following. In this case, for example, by appropriately replacing "paragraph" with "sentence", the above tag acquisition database document data TTD DB You can refer to the explanation of the extraction method.

[0104] Subsequently, as shown in step S03 in Figure 2, the tag acquisition unit 23 acquires the tag acquisition database document data TTD DB Based on database tags TAG DBSpecifically, the tag acquisition unit 23 acquires the tag acquisition database document data TTD. DB Database tag TAG containing at least some of the words included DB The tag acquisition unit 23 acquires the tag acquisition database document data TTD. DB Morphological analysis is performed on the results, and the segmented words are tagged in the database. DB It can be included in the database tag TAG. For example, a word identified as a noun by morphological analysis can be included in the database tag TAG. DB It can be included in the tag acquisition database document data TTD acquired by the tag acquisition unit 23. DB This is a database image data GD DB It is granted to.

[0105] Figure 5(A) shows the database image data GD DB (1) Database tag TAG DB (1) is a schematic diagram illustrating an example, and Figure 5(B) shows the database image data GD DB (2) Database tag TAG DB (2) This is a schematic diagram showing an example. As shown in Figure 5(A), the nouns contained in paragraphs [0aa1] to [0aa4] are tagged with the database tag TAG DB (1) This can be done. Also, as shown in Figure 5(B), nouns included in paragraphs [0bb1] to [0bb3] can be set to database tags TAG DB (2) This can be done. Here, the symbols attached to nouns and other words are database tags. DB It can be excluded. For example, in the example shown in Figure 5(A), "display device" is used instead of "display device 10000" as the database tag TAG. DB (1) is included.

[0106] For example, all words extracted by morphological analysis are assigned to the database tag TAG. DB It is not necessary to include them; for example, all nouns extracted by morphological analysis can be included in the database tag TAG. DBIt does not have to be included. For example, among the extracted words, words with a high TF-IDF (Term Frequency-Inverse Document Frequency) can be used as database tags. DB This can be done. For example, words whose TF-IDF is less than or equal to a predetermined value will have the database tag TAG DB It is possible to exclude them from being included. Alternatively, for example, from the words extracted by morphological analysis, a predetermined number of words with high TF-IDF can be tagged with a database tag TAG. DB It can be done this way.

[0107] TF-IDF is calculated based on two indicators: word frequency (TF) and inverse document frequency (IDF). Therefore, words that appear frequently throughout a document will have a high TF but a low IDF. Consequently, words that appear frequently throughout a document are included in the document data (TTD) used for tag acquisition. DB A word that appears frequently in the text it is contained in will have a lower TF-IDF than a word that appears less frequently in other texts. For example, in database document data DD. DB The most frequently appearing words overall are: Database image data GD DB The term may not strongly represent the concept, technical content, or characteristics of a particular point of interest. Therefore, considering TF-IDF, the database tag TAG DB When you obtain it, for example, all the words extracted by morphological analysis will be included in the database tag TAG DB Compared to including it in the database image data GD DB Database tags that strongly represent the characteristics of the database. DB This allows you to obtain database image data GD that is similar to the image data input to the search system 10 in the process described later, in which the characteristics such as the represented concept, technical content, or points of interest are found. DB It can search for this with high accuracy.

[0108] By the above method, the processing unit 20 processes the database tag TAG DB It is possible to obtain the database tag TAG. DB This is a database image data GDDB It can be assigned to and registered in database 13.

[0109] [Searching database image data] Figure 6 shows the database image data GD DB This flowchart shows an example of how to search for database image data (GD). DB To search, first, as shown in step S11, image data GD and document data DD are input. Specifically, for example, bibliographic data including image data GD with an image label and document data DD is supplied to the input unit 11. For example, bibliographic data can be supplied to the input unit 11 from outside the search system 10. Alternatively, bibliographic data registered in the database 13 (database bibliographic data) can be supplied to the input unit 11. In this case, for example, the user of the search system 10 can supply the desired bibliographic data to the input unit 11 by specifying information to identify the database bibliographic data. Although not shown in the figure, the bibliographic data supplied to the input unit 11 in step S11 is denoted as bibliographic data LD. Also, the image label assigned to the image data GD is denoted as image label GL. The bibliographic data LD is the database bibliographic data LD shown in Figure 3. DB A similar configuration can be adopted.

[0110] Next, as shown in step S12 in Figure 6, the text describing the image data GD is extracted from the document data DD. The extracted text is designated as the tag acquisition text data TTD. The tag acquisition text data TTD is the tag acquisition database text data TTD shown in Figures 4(A), 4(B1), and 4(B2). DB It can be extracted using the same method as the extraction method for [previous method].

[0111] In this specification, etc., text extracted from document data DD based on image labels GL may be referred to as the first text. Text extracted from document data DD based on coded words may be referred to as the second text. Furthermore, text extracted from document data DD based on uncoded words such as parallel conjunctions may be referred to as the third text.

[0112] Subsequently, as shown in step S13 in Figure 6, the tag acquisition unit 23 acquires the tag TAG based on the tag acquisition document data TTD. Specifically, it acquires the tag TAG which includes at least some of the words contained in the tag acquisition document data TTD. The tag acquisition document data TTD acquired by the tag acquisition unit 23 is attached to the image data GD. The tag TAG is a database tag TAG. DB It can be obtained using the same method as the method for obtaining [the other item].

[0113] Next, as shown in step S14 in Figure 6, the tag similarity calculation unit 25 calculates the database tag TAG DB Calculate the similarity to the tag TAG. For example, all database tags TAG registered in database 13. DB The similarity to the tag TAG can be calculated for this. Alternatively, the database tag TAG registered in database 13 can be used. DB Among them, some database tags DB For this, similarity to the tag TAG may be calculated. For example, database bibliographic data LD DB , and when the document data LD is used as the document in the application, the database document data LD filed before the filing date of the document data LD DB The database tag assigned to it DB Similarity to the tag can only be calculated for this specific case.

[0114] Subsequently, as shown in step S15 in Figure 6, the output unit 15 outputs the information as search results based on the similarity score. Specifically, the output unit 15 outputs the database tags with the highest similarity score. DBDatabase image data GD DB The system outputs information related to the above. For example, the output unit 15 supplies the information to an external source of the search system 10.

[0115] For example, the output unit 15 outputs database tags TAGs whose similarity is equal to or greater than a predetermined value. DB Database image data GD DB , and the database image data GD DB Database bibliographic data including LD DB The output unit 15 can output a predetermined number of database tags, starting with the one with the highest similarity. DB Extract the relevant database tag TAG DB Database image data GD DB , and the database image data GD DB Database bibliographic data including LD DB The output unit 15 can output the data. The information relating to the data output by the output unit 15 can be displayed, for example, by a display device provided outside the search system 10. In this way, the information output by the output unit 15 can be presented to the user of the search system 10.

[0116] By the above method, the search system 10 searches for images that, although their feature quantities differ significantly, have similar characteristics such as the concepts, technical content, or points of interest they represent, in the database image data GD. DB You can search within that database. Additionally, you can search for literature containing the image in question using the database literature data LD. DB The search can be performed from among these documents. Therefore, the search system 10 can search for, for example, patent documents, papers, or industrial products related to or similar to the invention before filing. This allows for a prior art search related to the invention before filing. By understanding and reviewing the relevant prior art, the invention can be strengthened and made into a strong patent that is difficult for competitors to circumvent.

[0117] Furthermore, for example, the search system 10 can be used to search for patent documents, papers, or industrial products related to or similar to industrial products before their release. For example, database literature data LD DB If the company's patent documents are included, it can be confirmed whether sufficient patent applications have been filed internally for technologies related to industrial products that have not yet been released. Alternatively, the database literature data LD DB If the prior art includes information on another company's intellectual property, it can be used to verify whether a pre-release industrial product infringes on the other company's intellectual property rights. By understanding the relevant prior art and re-examining the technology related to the pre-release industrial product, it is possible to discover new inventions and develop them into strong patentable inventions that contribute to one's own business. Furthermore, the search may extend beyond pre-release industrial products to include industrial products that have already been released.

[0118] Furthermore, for example, the search system 10 can be used to search for patent documents, papers, or industrial products related to or similar to a specific patent. In particular, by searching based on the filing date of the patent, it is possible to easily and accurately investigate whether the patent contains any grounds for invalidation.

[0119] Furthermore, the above search method allows for the acquisition of tags that comprehensively include words representing the concepts, content, or points of interest of the image data GD, compared to, for example, a search system user specifying all the words to be included in the tags TAG assigned to the image data GD and not presenting the user with candidate words to include in the tags TAG. Therefore, the search system 10 can perform searches easily.

[0120] <Method using machine learning_1> The tag acquisition database document data TTD shown in step S02 of Figure 2. DB The text data TTD for tag acquisition shown in step S12 of Figure 6 can be acquired using a machine learning model. For example, a multilayer perceptron or a neural network model can be applied as the machine learning model. Applying a neural network model is particularly preferable because it allows for efficient processing of learning and inference.

[0121] Figure 7 is a schematic diagram illustrating an example of a training method for the Classifier CLS, a machine learning model that can be used, for example, to acquire text data (TTD) for tag acquisition. When applying a neural network model as the Classifier CLS, for example, a Convolutional Neural Network (CNN) model can be applied.

[0122] The CLS classifier is trained using the training literature data LD. L This can be done through supervised learning using the following: Learning literature data LD L This includes GD image data for training. L , and learning document data DD L It includes GD for training image data. L This includes the learning image label GL L It has been granted.

[0123] Learning document data DD L The text included contains GD training image data. L A label is assigned to indicate whether or not the text is explaining [the topic]. Since this label indicates whether or not the text is used for tag acquisition, it is referred to as the tag label TL in this specification. The tag label TL can be assigned to each paragraph, or to each sentence. Figure 7 shows the training image data GD. L The tag label TL, which indicates that the text explains the topic, is represented by "Y", and the training image data GD L The tag label TL is represented by "N" to indicate that this is not a text explaining [the topic].

[0124] Tag labels TL can be assigned in the same manner as shown in Figures 4(A), 4(B1), and 4(B2). For example, a tag label TL represented by "Y" can be assigned to text extracted as document data for tag acquisition, and a tag label TL represented by "N" can be assigned to text not extracted as document data for tag acquisition. Alternatively, tag labels TL may be assigned manually. Alternatively, tag labels TL may be assigned in the same manner as shown in Figures 4(A), 4(B1), and 4(B2), and then manually modified.

[0125] The tag label TL can be used as the ground truth label during training of the classifier CLS. Through training, the classifier CLS can obtain the training result LR1. The training result LR1 can be used, for example, as a weight coefficient.

[0126] Learning literature data LD L For example, database bibliographic data LD DB At least a portion of it can be used. In addition, literature data not registered in database 13 can be used as learning literature data LD. L It may also be used. Here, literature that is closely related to the literature expected to be entered into the search system 10 as literature data LD is used as learning literature data LD L When used in this way, the CLS classifier can perform the inference described later with high accuracy, which is preferable.

[0127] Also, for example, learning literature data LD L If the literature used contains multiple images, such as multiple drawings, then a separate training literature data (LD) will be created for each image. L It may also be divided. For example, training literature data LD L If the document used contains two images, and the labels of these images are "Figure 1" and "Figure 2", then only the image labeled "Figure 1" in the document will be used as training image data (GD). L The learning literature data LD included as L (1) and only the image in "Figure 2" are used as training image data GD L The learning literature data LD included as L(2) and can be divided into these two parts.

[0128] Here, different images extracted from the same document were used as training image data for GD L Learning literature data LD L Among them, the training document data DD L These can be the same or different. For example, learning literature data LD L (1) Training document data DD L (1) and learning literature data LD L (2) Training document data DD L (2) and may be the same or different. Learning document data DD L If they are different, for example, training image data GD L The text that explains this, and the texts in the vicinity, are used as learning document data DD. L This is preferable because it allows for efficient training of the classifier CLS. For example, training image data GD L (1) Of the paragraphs included in the extracted documents, the paragraphs that say "Figure 1" and the paragraphs in the vicinity thereof are used as training document data DD L (1) can be done. Also, the training image data GD L (2) Of the paragraphs included in the extracted documents, the paragraphs that say "Figure 2" and the paragraphs in the vicinity thereof are used as training document data DD L (2) This can be done.

[0129] Figures 8(A) and 8(B) are schematic diagrams illustrating an example of how to obtain tag acquisition text data (TTD) by performing inference with a trained classifier CLS. As shown in Figure 8(A), when literature data LD, which includes image data GD with image labels GL and document data DD, is supplied to the classifier CLS, inference result IR is assigned to the sentences contained in the document data DD. Specifically, an inference result IR indicating whether or not a sentence describes the image data GD is assigned to the sentences contained in the document data DD.

[0130] For example, a sentence that is inferred to be a sentence describing the image data GD is assigned an inference result IR represented by "Y", and a sentence that is inferred not to be a sentence describing the image data GD is assigned an inference result IR represented by "N". Then, sentences that are assigned an inference result IR represented by "Y" can be used as tag acquisition document data TTD. In other words, step S12 shown in Figure 6 can be performed. Figure 8(A) shows an example in which paragraphs [0yy1] and [0yy2] in the document data DD are assigned an inference result IR represented by "Y", and paragraph [0yy3] is assigned an inference result IR represented by "N". Figure 8(B) shows the tag acquisition document data TTD acquired based on the inference result IR. The tag acquisition document data TTD includes paragraphs [0yy1] and [0yy2], but does not include paragraph [0yy3]. Here, since the tag acquisition document data TTD is acquired by the document extraction unit 21, the classifier CLS can be said to be incorporated into the document extraction unit 21.

[0131] Note: The document data (TTD) is used in the database for tag acquisition. DB Also, the literature data LD is a database literature data LD. DB By reinterpreting and making the necessary substitutions as appropriate, the data can be obtained in the same way as shown in Figures 8(A) and 8(B). Furthermore, the following explanations can also be obtained by making the necessary substitutions as appropriate, thus obtaining the database bibliographic data LD. DB The methods shown in Figures 8(A) and 8(B) can be applied when performing inferences on this.

[0132] After obtaining the text data TTD for tag acquisition, the tags TAG can be acquired using the same method as described using Figures 5(A) and 5(B), for example. That is, step S13 shown in Figure 6 can be performed. Subsequently, the search system 10 can perform a search by performing the steps shown in Figures 6, for example, steps S14 and S15.

[0133] The methods shown in Figures 8(A) and 8(B) allow for the acquisition of text data for tag acquisition by considering the feature quantities of the image itself represented by the image data GD. Here, in the training of the classifier CLS shown in Figure 7, if the methods shown in Figures 4(A), 4(B1), and 4(B2) are not used when assigning tag labels TL, specifically, for example, if tag labels TL are assigned manually, the training image labels GL L If you use it for training, then the training image data GD L It does not need to be used for training. In this case, even in the inference shown in Figure 8(A), if the image label GL is used for inference, the image data GD does not need to be used for inference.

[0134] <Method using machine learning_2> The database tag shown in step S03 of Figure 2 is TAG DB The tags shown in step S13 of Figure 6 can be obtained using a machine learning model. For example, a multilayer perceptron or a neural network model can be applied as the machine learning model. Applying a neural network model is particularly preferable because it allows for efficient processing of learning and inference.

[0135] Figure 9(A) is a schematic diagram showing an example of a training method for a generator GEN, a machine learning model that can be used, for example, to acquire tags. When applying a neural network model as the generator GEN, a model that applies GANs (Generative Adversarial Networks), such as DCGAN (Deep Convolutional Generative Adversarial Network), can be applied.

[0136] The generator GEN is trained using training image data GD L Using the learning tag TAG L This can be done using supervised learning with the correct label as TAG. LThis can be obtained in the same way as shown in Figures 4(A), 4(B1), 4(B2), 5(A), and 5(B). Alternatively, the learning tag TAG L This can be assigned manually. Alternatively, a learning tag can be used. L After assigning the learning tag TAG in the same manner as shown in Figures 4(A), 4(B1), 4(B2), 5(A), and 5(B), L You may manually add, delete, and modify words included in the text.

[0137] Through the above learning process, the generator GEN can obtain the learning result LR2. The learning result LR2 can be used, for example, as a weight coefficient.

[0138] Figure 9(B) is a schematic diagram illustrating an example of how to obtain a tag TAG by performing inference with a trained generator GEN. As shown in Figure 9(B), when image data GD is supplied to the generator GEN, a tag TAG is generated as an inference result. For example, the generator GEN calculates the probability by inference that each candidate word to be included in the tag represents a feature such as a concept, technical content, or point of interest in the image data GD, and can then include words with a probability of a certain value or higher in the tag TAG.

[0139] As a result, the tag TAG can be obtained. In other words, step S13 shown in Figure 6 can be performed. Here, since the tag TAG is obtained by the tag acquisition unit 23, it can be said that the generator GEN is incorporated into the tag acquisition unit 23. Note that when obtaining the tag TAG using the generator GEN, it is not necessary to obtain the tag acquisition document data TTD. Therefore, step S12 shown in Figure 6 does not need to be performed. Also, when obtaining the tag TAG using the generator GEN, the document data DD and image label GL do not need to be input in step S11 shown in Figure 6.

[0140] After obtaining the tag, the search system 10 can perform a search by, for example, carrying out the steps shown in steps S14 and S15 in Figure 6.

[0141] Note: Database tag TAG DB Also, image data GD to database image data GD DB By reinterpreting it as such and making the necessary reinterpretations as appropriate, it can be obtained in the same way as the method shown in Figure 9(B).

[0142] <Search System_2> Figure 10 is an illustrative diagram showing the search system of this embodiment.

[0143] The search system shown in Figure 10 comprises a server 1100 and terminals (also called electronic devices). Communication between the server 1100 and each terminal can be performed via an internet connection 1110.

[0144] Server 1100 can perform calculations using data input from a terminal via the internet connection 1110. Server 1100 can also transmit the results of the calculations to the terminal via the internet connection 1110. This reduces the computational burden on the terminal.

[0145] Figure 10 shows information terminals 1300, 1400, and 1500 as terminals. Information terminal 1300 is an example of a portable information terminal such as a smartphone. Information terminal 1400 is an example of a tablet terminal. Information terminal 1400 can also be used as a notebook-type information terminal by connecting it to a casing 1450 with a keyboard. Information terminal 1500 is an example of a desktop-type information terminal.

[0146] By configuring the system in this way, users can access the server 1100 from information terminals 1300, 1400, and 1500, etc. Users can then receive services provided by the administrator of the server 1100 through communication via the internet line 1110. Examples of such services include a service using a search system according to one embodiment of the present invention. In such a service, artificial intelligence may be used on the server 1100. [Explanation of Symbols]

[0147] 10 Search System 11 Input section 13 Databases 15 Output section 20 Processing Units 21 Sentence extraction part 23 Tag acquisition section 25 Tag Similarity Calculation Unit 1100 Server 1110 Internet connection 1300 Information Terminals 1400 Information Terminals 1450 cabinets 1500 Information Terminals

Claims

1. The apparatus includes an input unit, a sentence extraction unit, a tag acquisition unit, and a tag similarity calculation unit, the text extraction unit has a function of, when database image data to which a database image label has been assigned and database document data including the database image label are supplied to the input unit, extracting database text data for tag acquisition from the database document data based on the database image label; the text extraction unit has a function of, when image data to which an image label has been assigned and document data including the image label are supplied to the input unit, extracting text data for tag acquisition from the document data based on the image label; the tag acquisition unit has a function of acquiring database tags including at least some of the words included in the tag acquisition database sentence data, the tag acquisition unit has a function of acquiring tags including at least some of the words included in the tag acquisition sentence data, The tag similarity calculation unit is a search system having a function of calculating the similarity of the database tag to the tag.

2. In claim 1, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, at least a part of a paragraph including the database image label as a first database sentence, and setting the first database sentence as the database sentence data for tag acquisition; The sentence extraction unit extracts at least a portion of a paragraph that includes the image label from among the paragraphs included in the document data as a first sentence, and uses the first sentence as sentence data for tag acquisition.

3. In claim 2, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, a paragraph that includes the database image label at the beginning as the first database sentence; The sentence extraction unit is a search system having a function of extracting, from among paragraphs included in the document data, a paragraph that includes the image label at the beginning as the first sentence.

4. In claim 2 or claim 3, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, at least a part of a paragraph including a coded word included in the first database sentence as a second database sentence, and including the second database sentence in the database sentence data for tag acquisition; The sentence extraction unit extracts at least a portion of a paragraph from among the paragraphs included in the document data that contains a coded word included in the first sentence as a second sentence, and includes the second sentence in the sentence data for tag acquisition.

5. In claim 4, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, paragraphs that begin with a coded word included in the first database sentence and are within a predetermined number of paragraphs from the first database sentence, as the second database sentence; The sentence extraction unit is a search system having a function of extracting, from among the paragraphs contained in the document data, paragraphs that contain a coded word contained in the first sentence at the beginning and are within a predetermined number of paragraphs from the first sentence as the second sentence.

6. In claim 2 or claim 3, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, a paragraph that is within a predetermined number of paragraphs from a paragraph included in the database sentence data for tag acquisition and that includes a parallel conjunction at the beginning as a third database sentence, and including the third database sentence in the database sentence data for tag acquisition; The sentence extraction unit extracts, from among the paragraphs included in the document data, a paragraph that is within a predetermined number of paragraphs from the paragraph included in the sentence data for tag acquisition and that contains a parallel conjunction at the beginning as a third sentence, and includes the third sentence in the sentence data for tag acquisition.

7. In claim 4, the sentence extraction unit has a function of extracting, from among paragraphs included in the database document data, paragraphs that are separated from the paragraph including the first database sentence by a distance equal to or greater than a first threshold value and less than a second threshold value, and that include a coded word included in the first database sentence at the beginning, as the second database sentence; The sentence extraction unit is a search system having a function of extracting, from among the paragraphs included in the document data, a paragraph that is more than the first threshold value and less than the second threshold value away from the paragraph containing the first sentence, and that includes a coded word contained in the first sentence at the beginning as the second sentence.

8. In claim 1, the sentence extraction unit has a function of extracting the tag acquisition sentence data by a machine learning model based on the image data and the document data, A search system in which the machine learning model is a model trained using training image data to which training image labels have been assigned and training document data that includes the training image labels.

9. In claim 8, A search system in which a tag label indicating whether or not the sentence represented by the learning document data is used for tag acquisition is assigned to the sentence.

10. In claim 9, A search system in which the tag labels are assigned based on the learning image labels.

11. In claim 9 or claim 10, A search system in which the tag label is assigned to each paragraph included in the learning document data.

12. 1. A method for searching database images that have been tagged with database tags containing a term, comprising: When image data to which an image label has been assigned and document data including the image label are input, tag acquisition sentence data is extracted from the document data based on the image label; acquiring tags that include at least some of the words included in the tag acquisition sentence data; A search method that calculates the similarity of the database tag to the tag.

13. In claim 12, extracting, from among paragraphs included in the document data, at least a part of a paragraph including the image label as a first sentence; A search method in which the first sentence is used as the tag acquisition sentence data.

14. In claim 13, A search method for extracting, from among paragraphs included in the document data, a paragraph that includes the image label at the beginning as the first sentence.

15. In claim 13 or claim 14, extracting, from among the paragraphs included in the document data, at least a part of a paragraph including a coded word included in the first sentence as a second sentence; A search method in which the second sentence is included in the tag acquisition sentence data.

16. In claim 15, A search method for extracting, from among the paragraphs contained in the document data, paragraphs that begin with a coded word contained in the first sentence and are within a predetermined number of paragraphs from the first sentence as the second sentence.

17. In claim 13 or claim 14, extracting, from among the paragraphs included in the document data, a paragraph that is within a predetermined number of paragraphs from the paragraph included in the tag acquisition sentence data and that includes a parallel conjunction at the beginning, as a third sentence; A search method in which the third sentence is included in the tag acquisition sentence data.

18. In claim 15, A search method for extracting, from among the paragraphs included in the document data, paragraphs that are at least a first threshold value and less than a second threshold value away from the paragraph containing the first sentence, paragraphs that contain a coded word included in the first sentence at the beginning as the second sentence.

19. In claim 12, extracting the tag acquisition sentence data using a machine learning model based on the image label and the document data; A search method in which the machine learning model is a model trained using training image labels and training document data including the training image labels.

20. In claim 19, A search method in which a tag label indicating whether or not the sentence represented by the learning document data is used for tag acquisition is assigned to the sentence.

21. In claim 20, A search method in which the tag labels are assigned based on the learning image labels.

22. In claim 20 or claim 21, A search method in which the tag label is assigned to each paragraph included in the learning document data.

Images