Document data processing system

Abstract

It allows natural language input as query text, enables searching across multiple documents, and presents the reader with sections of the text that are highly relevant to the input. [Solution] The document data processing system includes a document reading unit that reads multiple target documents, a document splitting unit that divides each of the multiple target documents into multiple blocks, a first distributed representation acquisition unit that acquires word distributed representations for each block, a first distributed representation storage unit that stores the distributed representations acquired by the first distributed representation acquisition unit for each target document and each block, a query text reading unit that reads a query text, a second distributed representation acquisition unit that extracts words contained in the query text and acquires word distributed representations, a second distributed representation storage unit that stores the distributed representations acquired by the second distributed representation acquisition unit, and a similarity calculation unit that compares the word distributed representations contained in the query text with the word distributed representations contained in each block and calculates the similarity for each block.

Description

Technical Field

[0001] One aspect of the present invention relates to a document data processing method and a document data processing system. One aspect of the present invention relates to a document search method, a document search system, a document reading assistance meth od, and a document reading assistance system.

Background Art

[0002] Generally, when identifying the document most relevant to the information desired by a user from a large volume of documents, or when identifying sentences or paragraphs describing such information, search using text is sometime s performed. In addition, search may also be performed using document classification information such as the international patent classification in patent documents in some cases. After appropriately using such searches to narrow down the number of documents to a certain range, manual review of the content may be performed. For digitized documents, there is also a method of browsing documents while searching with keyword words to find desired information. In addition, methods for performing structural analysis of documents in accordance with set rules have been proposed (Patent Document 1).

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Non-Patent Literature

[0004]

Non-Patent Document 1

[0005] As described above, a primary search using keywords and classifications narrows down the number of documents to a certain number. Identifying the document containing the desired information from among several documents, and determining the degree of relevance from multiple documents. Identifying high points is a laborious task. In such tasks, keywords Another method is to use text search to find sentences or paragraphs containing keywords throughout the entire document. However, sometimes it is not possible to find the desired information efficiently. Reasons for not being able to find it efficiently include The problem is that there are too many results for a given keyword, making it take too long to find the information you want. Examples include not being able to find suitable keywords. Also, documents that do not follow rules. When performing structural analysis, the structure to be read is limited, so sentences with various structures It is difficult to handle the writing. One aspect of the present invention solves at least one of these problems. It is.

[0006] One aspect of the present invention enables natural language input as a query sentence and allows searching across multiple documents. A document data processing system that enables searching and presents the reader with sections of the text that are highly relevant to the input text. One of the objectives is to provide a method for processing system or document data.

[0007] Furthermore, the description of these problems does not preclude the existence of other problems. One aspect of the present invention is It is not necessarily required to resolve all of these issues. Specifications, drawings, invoices. It is possible to extract other issues from the descriptions in the sections. [Means for solving the problem]

[0008] One aspect of the present invention is a document reading unit that reads multiple target documents, and each of the multiple target documents A document splitting unit that divides the document into multiple blocks, and a first unit that obtains word embeddings for each block. The distributed representations acquired by the distributed representation acquisition unit and the first distributed representation acquisition unit are processed for each target document, and A first distributed representation storage unit that stores data in blocks, and a query text reader that reads the query text. This section extracts words from the query text and obtains the distributed representations of those words. A second distributed representation acquisition unit, and a second unit that stores the distributed representation acquired in the second distributed representation acquisition unit. The distributed representation holding unit of 2, and the distributed representations of words contained in the query text, and multiple blocks The similarity is calculated by comparing the embedded word representations contained in each block with the corresponding block. A document data processing system including a calculation unit, the similarity calculation unit includes units included in a block. The system searches for words that match the words contained in the query text, and then searches for the matching words. The similarity between the word embeddings in the block and the word embeddings in the query text is calculated. It is a document data processing system that performs calculations.

[0009] One aspect of the present invention involves the steps of reading multiple target documents and processing each of the multiple target documents in multiple ways. a step of dividing into a plurality of blocks, a step of acquiring distributed representations of words for each block, a step of reading a query sentence, extracting words included in the query sentence, a step of acquiring distributed representations of words included in the query sentence, and comparing the distributed representations of words included in the query sentence with distributed representations of words included in each of the plurality of blocks to calculate a similarity for each block, which is a document data processing method comprising: in the step of calculating a similarity for each block, words matching words included in the query sentence are searched from among words included in the block, and for the matched words, the similarity between the distributed representation of the word in the block and the distributed representation of the word in the query sentence is calculated, which is a document data processing method.

[0010] A display method of scores as a result of calculating similarities can be appropriately determined depending on the purpose of work. For example, sentences can be displayed on a screen in the order of blocks with high similarity. This is useful when it is desired to find one or a plurality of most relevant documents from among a plurality of entire target documents. Alternatively, when it is desired to evaluate each of a plurality of target documents, the block having the highest similarity in each of the target documents is displayed, or a predetermined number of top blocks with high similarity can be displayed. It is also possible to display the blocks.

[0011] Each of the plurality of blocks may include one or a plurality of paragraphs of the target document.

[0012] Each of the plurality of blocks can include one or a plurality of sentences.

[0013] The calculation of similarity may be performed only for predetermined parts of speech.

[0014] ​The similarity can also be calculated using cosine similarity.

[0015] If there are multiple matching words in the query text and block, a distribution table is created for each word. The sum of the current similarity scores may be used as the score for that block. [Effects of the Invention]

[0016] According to one aspect of the present invention, natural language input is enabled as a query sentence, and from among multiple documents... Furthermore, a method for processing document data that presents the reader with sections of the input text that are highly relevant to the text, and a document We can provide a data processing system.

[0017] Furthermore, the description of these effects does not preclude the existence of other effects. One aspect of the present invention is It is not necessarily required to have all of these effects. It is possible to extract effects other than those listed above. [Brief explanation of the drawing]

[0018] [Figure 1] Figure 1 shows an example of a document data processing system. [Figure 2] Figure 2 is a flowchart showing an example of a document data processing method. [Figure 3] Figure 3 is a flowchart showing an example of a document data processing method. [Figure 4] Figure 4 is a diagram illustrating word embeddings. [Figure 5] Figure 5 illustrates an example of a method for calculating similarity. [Figure 6] Figure 6 shows an example of the hardware for a document data processing system. [Figure 7] Figure 7 shows an example of the hardware for a document data processing system. [Modes for carrying out the invention]

[0019] Embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description. Without departing from the spirit and scope of the present invention, its form and details may be modified in various ways. It will be easily understood by those skilled in the art to obtain this. Therefore, the present invention is as shown in the embodiments below. The interpretation is not limited to the content stated herein.

[0020] In the configuration of the invention described below, the same part or part having a similar function is included. The same reference numerals are used consistently across different drawings, and explanations of their repetition are omitted. When referring to the function of [this], the hatch pattern is the same, and sometimes no specific symbol is assigned.

[0021] Furthermore, the position, size, and extent of each component shown in the drawings are, for the sake of ease of understanding, actually The location, size, and range may not be described. Therefore, the disclosed invention is not always Furthermore, it is not limited to the location, size, scope, etc., disclosed in the drawings.

[0022] (Embodiment 1) In this embodiment, a document data processing system and a document data processing method according to one aspect of the present invention are provided. This will be explained using Figures 1 to 5.

[0023] In the document data processing method of this embodiment, first, multiple documents to be processed (target documents) are processed. ) obtain. Multiple documents are documents collected by some method, and the method of obtaining them is This is not limited to any specific method or means. For example, by using a general search service The documents can be those collected by the system, or they can be documents collected by the user using their own methods. Furthermore, the number of documents to be processed should be determined considering the capacity and load of the computer and memory used for processing. The user can decide as needed. Each of the target documents consists of multiple blocks (e.g., paragraphs). The document is divided into blocks, and word embeddings are obtained for each block. Data is created that contains embedded representations of each word.

[0024] On the other hand, it retrieves query text to obtain information of interest to the user, and further includes in the query text Obtain the embedded representation of the word.

[0025] Next, we search for words within the block that match the words in the query sentence. Search. Then, for the matching words, find the word's distributed representation in the block and the query statement. Calculate the similarity (e.g., cosine similarity) between the word embeddings in the chapter. If there are multiple words, the sum of the similarity scores of the distributed representations for each word is used as the block score. Blocks with relatively high scores are considered to be highly relevant to the query text. This allows us to identify blocks from the entire data that have a high relationship or similarity to the information in question. It can be determined that blocks are arranged in descending order of score and in descending order of relevance. The checklist can be displayed on the user's screen.

[0026] In the document data processing method of this embodiment, multiple targets are processed by inputting a question sentence in natural language. You can provide relevant sections from the document that are related to the question. Because different distributed representations are used depending on the text, the more relevant or similar blocks are to the question. This allows for the presentation of a search query. This enables, for example, primary searches using keywords or classifications. By creating a set through searching, and then processing the documents included in that set, efficient reading can be achieved. It becomes possible to perform solutions and searches. In other words, the document data processing system of this embodiment and Document data processing methods can be used for document searching, document comprehension support, and other purposes.

[0027] The question can contain one or more sentences. The selection of keywords to use in the search is important. Because it is essential, users can find the desired information from the document with minimal effort.

[0028] Unless otherwise specified in this specification, a document is a description of an event in natural language, and electricity The documents are childized and machine-readable. Examples include patent applications, case law, contracts, terms and conditions, and product information. This includes, but is not limited to, manuals, novels, publications, white papers, and technical documents. In this specification, etc., "text" includes one or more sentences.

[0029] In this specification, a word is the smallest linguistic unit that possesses a sound, meaning, and grammatical function. However, we may also seek embedded representations for subwords obtained by further dividing a word. For example, English The word "transformer" is a combination of "transform" and "er". It is also possible to decompose it into broaders and assign a distributed representation to each. Or, two or more broaders. It is also possible to assign distributed representations to linked phrases of words. In this specification, a subword obtained by splitting a word is also called a word. The given phrase, word, or subword is sometimes also called a token.

[0030] In this embodiment, even if the word is the same word, the distribution of surrounding words is also This is obtained using a language model that yields different distributed representations depending on the context. Alternatively, the same It is obtained using a language model that can produce different distributed representations depending on the context, even for single words. As word embeddings, the position of a word in a sentence and its segment (information about the connection between words in a sentence) A language model can be used that provides a distributed representation with embedded token information. A language model with attention-grabbing capabilities that learns from both sides of text and acquires distributed representations. You may also use "ru". Even the same word can have different meanings depending on the distribution of surrounding words or the context. One example of a language model that can produce dispersive representations is BERT (Bidirectional En). coder Representations from Transformers) (See Non-Patent Document 1) can be cited.

[0031] Figure 4 shows the results for six English sentences containing "carbon" in each sentence. The distributed representation of "carbon" obtained by BERT is plotted on the XY coordinate system. Therefore, the three plots (squares) on the left half contain "carbon" as an impurity in the material. The text is written in a way that the three plots (diamonds) on the right half represent "carbo" as the negative electrode material. This is a text about "n". Figure 4 shows that even though it is the same "carbon", the context and sentence can differ. This is an example demonstrating that different distributed representations can be obtained.

[0032] Even the same word can have different word embeddings depending on the context in which it appears. By using this method, blocks highly relevant to the information the user needs can be displayed with high accuracy. It can be found. For example, if the query text contains "carbon" as a negative electrode material... If this was the case, the score of the block containing "carbon" as the negative electrode material would be Conversely, the score of blocks containing "carbon" as an impurity is relatively high. It is expected to decrease in proportion.

[0033] [Document Data Processing System] Figure 1 is a block diagram showing the configuration of the document data processing system 100.

[0034] The document data processing system 100 uses information from personal computers and other devices used by users. It may be provided in the processing unit. Alternatively, the document data processing system 100 may be provided on the server. A processing unit is provided, and the system is configured to be accessed and used from client PCs via the network. That's fine.

[0035] The document data processing system 100 includes a document reading unit 101, a question input unit 102, and a document splitting unit. 103, distributed representation acquisition unit 104a, distributed representation acquisition unit 104b, distributed representation storage unit 105a, Distributed representation storage unit 105b, word selection unit 106, similarity calculation unit 107, score display unit 108 It also includes a text display unit 109.

[0036] The document reading unit 101 reads multiple documents to be read.

[0037] The multiple documents read by the document reading unit 101 are a collection of documents gathered by some method. For example, it could be a collection of documents gathered via the internet. Also, it could be used by users. This can be a document stored on a personal computer, or on a network-connected storage device. It may also be a document saved on the page.

[0038] The question input section 102 is where the user enters the text they specify for the search.

[0039] The method for inputting a question (also called a query text) is to directly enter any text, or You may also copy and paste text from a document file. A portion of the document read by 101 is arbitrarily selected by the user and loaded into the question input unit 102. A mechanism that allows this would also be acceptable.

[0040] The document splitting unit 103 divides each of the multiple documents read by the document reading unit 101 into multiple blocks. Divide into blocks.

[0041] Treat each paragraph as a block, and each sentence separated by a period or punctuation mark as a block. Alternatively, the document may be divided into blocks of a specified number of paragraphs or sentences. In this case, there is a document format that includes paragraph numbers from the beginning, and according to those paragraph numbers It may be divided into blocks.

[0042] The distributed representation acquisition unit 104a processes each document read by the document reading unit 101 in blocks. Then, obtain the embedded representations of the words contained within the block.

[0043] The distributed representation acquisition unit 104b generates a distributed representation of the words contained in the text entered into the question input unit 102. Acquire the current.

[0044] The distributed representation acquisition unit 104a and the distributed representation acquisition unit 104b basically use the same language model. It is preferable.

[0045] The distributed representation storage unit 105a and the distributed representation storage unit 105b use the acquired distributed representation as data. To retain. The data structure of the retaining part of the distributed representation retaining part 105a is shown in Table 1. Table 2 shows the data structure of the holding part of 105b.

[0046] [Table 1]

[0047] [Table 2]

[0048] The distributed representation storage unit 105a has a data area for each of the multiple documents, and further data for each block. There is a region. The data region of each block contains data extracted from the text that each block contains. The word and the corresponding distributed representation are stored. The data structure shown assumes a single query statement, but it can also handle multiple query statements as input. The embedded representations of words in the text may be preserved.

[0049] The word selection unit 106 selects words from the input question text to be used for calculating similarity. This is the part that does that.

[0050] Select all words, select specific parts of speech such as nouns, or allow the user to freely select words. You may choose to select at least one word, and even if you select only one word, it may be important to consider the context of the sentence. Because different distributed representations can be obtained, scoring is possible.

[0051] The similarity calculation unit 107 uses the data obtained by the distributed representation acquisition unit 104a and the distributed representation acquisition unit 104b. The similarity of each block to the question sentence is calculated using word embeddings.

[0052] The score display unit 108 can display the score calculated by the similarity calculation unit 107. .

[0053] The document display unit 109 can display the document read by the document reading unit 101. The display unit 109 may also display the text entered into the question input unit 102.

[0054] It is preferable that the score display unit 108 and the text display unit 109 are synchronized. For example, the score Sort the text blocks in descending order of score, displaying only blocks with a score above a specified value. The display method for the target document may be changed based on the score value, etc.

[0055] [Document Data Processing Method] Figures 2 and 3 illustrate the processing flow performed by the document data processing system 100, respectively. This is a flowchart. In other words, Figures 2 and 3 are documents of one aspect of the present invention, respectively. This could be considered a flowchart illustrating an example of a data processing method.

[0056] [Step S1: Retrieve multiple target documents] First, the document reading unit 101 of the document data processing system 100 is used to read multiple documents. Load it.

[0057] [Step S2: Divide the target document into multiple blocks] Next, the document splitting unit 103 divides each of the multiple target documents into multiple blocks.

[0058] [Step S3: Obtain word embeddings for each block] Next, the distributed representation acquisition unit 104a receives text block by block and acquires distributed representations of words. Specifically, the target document is input into a language model such as BERT block by block, and the words are divided... Dispersive representations are obtained. The distributed representations obtained by the distributed representation acquisition unit 104a are, for each target document, Furthermore, each block is stored in the distributed representation holding unit 105a.

[0059] [Step S4: Retrieve the query statement] Furthermore, the query text is obtained in the question input unit 102 of the document data processing system 100. The query text may be text entered arbitrarily by the user, and it may reflect the user's interest in the target document. It is also acceptable for the text to be in a high position. In Figure 2, step S3 is followed by step S4 and step An example of performing step S5 is shown, but as shown in Figure 3, steps S1 to S3 and step Steps S4 and S5 can be performed independently, and the order does not matter.

[0060] [Step S5: Obtain word embeddings from the query text] Next, the distributed representation acquisition unit 104b receives a query sentence and acquires word distributed representations. In practice, the query text is input into a language model such as BERT, and word embeddings are obtained. The distributed representations obtained by the distributed representation acquisition unit 104b are stored in the distributed representation holding unit 105b. It will be done.

[0061] [Step S6: Calculate the block score] Next, the similarity calculation unit 107 calculates the words included in each block and the words included in the query text. It searches for matching words between them, and only if a word matches, it connects the distributed representations of the matching word. By calculating the sine similarity and then summing the cosine similarities within the block, the block's size is determined. Obtain the core.

[0062] In the word selection unit 106, select the words from the query text to be used for similarity calculation. Furthermore, similarity calculations may be performed only for the selected words.

[0063] In this embodiment, we show an example of calculating similarity using cosine similarity, but other similarity A method for calculating degrees may also be used.

[0064] Figure 5 explains how to calculate the score for each block. In Figure 5, the query text is Then, block 1, block 2, block 3, and block of target document 1 and target document 2 Here is an example of comparing 4. First, in each block of the target document, find the words that match the words in the query sentence. Search for a word and calculate the cosine similarity of its distributed representation only for matching words. If there are multiple matching words in a single block, the cosine similarity of each word is calculated. The score for the block is calculated by adding these values. For example, the score for target document 1 shown in Figure 5 In Block 1, two words, W1 and W2, in the query statement match. In this case, the target The score for block 1 of document 1 is the cosine similarity of word W1 and the cosine similarity of word W2. It becomes a harmony.

[0065] [Step S7: Output the calculated score] Then, the blocks with the highest calculated scores are the blocks that are most likely to contain the information you are looking for. This can be presented to the user. The method of presentation involves setting a predetermined threshold, and the threshold A method for presenting blocks that exceed a certain value, and presenting the block with the highest score in each document. A method of showing, or a predetermined number of blocks with the highest scores among multiple blocks as a whole. Methods such as presenting the answer are examples. These methods may also be combined as appropriate.

[0066] As described above, in the document data processing system and document data processing method of this embodiment, The system provides a set of documents to be read and texts related to the information needed. The system then presents blocks from the collection of documents that are highly relevant to the information the user needs. This eliminates the need for keyword selection, allowing users to search for desired information within documents. It becomes easier to take out.

[0067] In the document data processing system and document data processing method of this embodiment, the same word This also employs a language model that yields different word embeddings depending on the included sentence. This allows us to find blocks that are highly relevant to the information the user needs with high accuracy. It is possible.

[0068] This embodiment can be appropriately combined with other embodiments. Furthermore, this specification Furthermore, if multiple configuration examples are shown within a single embodiment, the configuration examples may be combined as appropriate. It is possible to do so.

[0069] (Embodiment 2) In this embodiment, Figures 6 and 7 show a document data processing system according to one aspect of the present invention. I will explain.

[0070] The document data processing system of this embodiment uses the document data processing method shown in Embodiment 1. This allows for easy searching and retrieval of desired information from documents.

[0071] <Example of document data processing system configuration 1> Figure 6 shows a block diagram of the document data processing system 200. In the drawings, the components are classified by function and shown as independent blocks in a block diagram. However, it is difficult to completely separate the actual components by function, and one component An element can be involved in multiple functions. Also, one function can be involved in multiple components. It is also possible that, for example, the processing performed by the processing unit 120 is executed on different servers depending on the process. It can happen.

[0072] The document data processing system 200 has at least a processing unit 120. The document shown in Figure 6 The data processing system 200 further includes an input unit 110, a storage unit 130, and a database 14 It has a 0, a display unit 150, and a transmission line 160.

[0073] [Input section 110] The input unit 110 receives a question (query text) from outside the document data processing system 200. It is supplied. In addition, the input unit 110 receives the target document from outside the document data processing system 200. A set of the following may be supplied. The set of target documents and the query text supplied to the input unit 110 are Each of these is transmitted via the transmission line 160 to the processing unit 120, the storage unit 130, or the database. It will be supplied to 140.

[0074] The target documents and query statements include, for example, text data, audio data, or image data. The data is entered in this format. Preferably, the target document is entered as text data.

[0075] Query text can be entered using a keyboard, touch panel, or other key input methods. Force, voice input using a microphone, reading from recording media, scanner, camera, etc. Examples include image input and data acquisition using communication.

[0076] The document data processing system 200 has a function to convert audio data into text data. It may be. For example, the processing unit 120 may have that function. Or, document data The processing system 200 may further include a voice conversion unit having the said function.

[0077] The document data processing system 200 may have an optical character recognition (OCR) function. This allows for the recognition of characters contained in image data and the creation of text data. For example, the processing unit 120 may have the function. Alternatively, the document data processing system The M200 may further have a character recognition unit having the said function.

[0078] [Processing step 120] The processing unit 120 receives data supplied from the input unit 110, storage unit 130, database 140, etc. It has the function of performing calculations using data. The processing unit 120 stores the calculation results in the storage unit 130. It can be supplied to the database 140, the display unit 150, and the like.

[0079] The processing unit 120 has the function of dividing a document into multiple blocks. For example, it divides a document into chapters. Even if it has a function to divide into multiple blocks, such as paragraphs or every specified number of sentences, good.

[0080] The processing unit 120 has the function of obtaining word embeddings. For example, blocks of the target document It is possible to obtain embedded representations of words contained within the query text, as well as words contained within the query text.

[0081] The processing unit 120 has the function of extracting words from the query text. You can select which words from the list to use for calculating similarity.

[0082] The processing unit 120 has the function of calculating the similarity between word embeddings.

[0083] The processing unit 120 may use a transistor having a metal oxide in the channel formation region. Because the transistor has an extremely low off-current, it can be used as a memory element. By using it as a switch to hold the charge (data) that has flowed into a capacitive element, This ensures that data retention can be maintained for a long period. Processing unit 120 utilizes this characteristic. By using it in at least one of the registers and cache memory, when necessary The processing unit 120 is operated only in this case, and in other cases, the information from the previous processing is saved to the memory element. By doing so, the processing unit 120 can be turned off. In other words, normally off computer This enables routing and reduces the power consumption of the document data processing system. .

[0084] In this specification, a transistor using an oxide semiconductor in the channel formation region is referred to as O xide Semiconductor transistor, also called OS transistor. The channel formation region of the OS transistor preferably has a metal oxide.

[0085] The metal oxide in the channel-forming region preferably contains indium (In). If the metal oxide in the Nell-forming region is an indium-containing metal oxide, the OS Transis The carrier mobility (electron mobility) of the ion becomes higher. Also, the metallic acid present in the channel-forming region The oxide is preferably an oxide semiconductor containing element M. Element M is aluminum (Al). It is preferable that it be gallium (Ga) or tin (Sn). Other applicable elements M The elements include boron (B), silicon (Si), titanium (Ti), iron (Fe), and nickel. Kel (Ni), Germanium (Ge), Yttrium (Y), Zirconium (Zr), Mo Ribdenum (Mo), Lanthanum (La), Cerium (Ce), Neodymium (Nd), Hafniu Examples include fluorine (Hf), tantalum (Ta), and tungsten (W). However, as element M... In some cases, it is acceptable to combine multiple of the aforementioned elements. Element M, for example, can be combined with oxygen. It is an element with high bonding energy. For example, its bonding energy with oxygen is higher than that of indium. It is an element. Furthermore, the metal oxides that the channel-forming region contains include zinc (Zn). This is preferable. Zinc-containing metal oxides may be prone to crystallization.

[0086] The metal oxides present in the channel-forming regions are not limited to indium-containing metal oxides. The semiconductor layer is made of materials such as zinc tin oxide and gallium tin oxide, which do not contain indium. These included metal oxides containing zinc, metal oxides containing gallium, and metal oxides containing tin. That's fine.

[0087] Furthermore, the processing unit 120 may use a transistor that includes silicon in its channel formation region. stomach.

[0088] Furthermore, the processing unit 120 includes a transistor containing an oxide semiconductor in the channel formation region, and a channel A transistor containing silicon in the flannel formation region may be used in combination with the other transistor.

[0089] The processing unit 120 is, for example, an arithmetic circuit or a central processing unit (CPU). It has an processing unit, etc.

[0090] The processing unit 120 includes a DSP (Digital Signal Processor) and a GP (Ground Processing Unit). It has a microprocessor such as U (Graphics Processing Unit). It is acceptable to do so. Microprocessors are FPGAs (Field Programmable Arrays). Field Programmable Array), FPAA (Field Programmable A Programmable Logic Dev (PLD) such as a rectangular array The configuration may be implemented by ice. The processing unit 120 is determined by the processor. By interpreting and executing instructions from various programs, various data processing and programmatic processes are performed. It is possible to perform the action. The programs that can be executed by the processor are those that the processor possesses. It is stored in at least one of the memory area and the storage unit 130.

[0091] The processing unit 120 may have main memory. The main memory may be volatile memory such as RAM. It has at least one of Mori and non-volatile memory such as ROM.

[0092] For example, RAM can be DRAM (Dynamic Random Access Memory). mory), SRAM (Static Random Access Memory), etc. This is used, and a memory space is virtually allocated and used as the workspace for the processing unit 120. The operating system and application programs stored in the memory unit 130. Program modules, program data, and lookup tables are used for execution. These are then loaded into RAM. These data, programs, and programs loaded into RAM are then loaded into RAM. Each program module is directly accessed and operated by the processing unit 120.

[0093] The ROM contains BIOS (Basic Input / Output) which does not require rewriting. It can store the System and firmware, etc. As for ROM, SCRROM, OTPROM (One Time Programmable Read) Only Memory), EPROM (Erasable Programmable Examples include Read Only Memory. EPROMs include ultraviolet light UV-EPROM (Ultra-Violet) enables the erasure of stored data by irradiation. Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmability) Examples include Read Only Memory (e) and flash memory.

[0094] [Storage section 130] The memory unit 130 has the function of storing the program to be executed by the processing unit 120. The memory unit 130, for example, receives the calculation results generated by the processing unit 120 and the input input to the input unit 110. It may also have a function to store the data. Specifically, the storage unit 130 is a processing unit 1 It is preferable that the system has a function to store the word embeddings obtained in step 20.

[0095] The storage unit 130 has at least one of volatile memory and non-volatile memory. The unit 130 may have, for example, volatile memory such as DRAM or SRAM. Part 130 is, for example, ReRAM (Resistive Random Access). Memory (also called resistive random-access memory), PRAM (Phase change R) andom Access Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresis (also known as magnetically resistive random access memory) Alternatively, it may have non-volatile memory such as flash memory. Also, storage unit 130 These include hard disk drives (HDDs) and solid-state drives. Recording media such as state drives (Solid State Drive: SSD) It's okay to have live performances.

[0096] [Database 140] The document data processing system may have a database 140. For example, the database -140 has the function of storing multiple documents. For example, it can store documents in database 140. A document data processing method according to one aspect of the present invention may be used on a collection of documents. Note that the storage unit 130 and the database 140 do not necessarily have to be separated from each other. The document data processing system has the functions of both a storage unit 130 and a database 140. It may have a memory unit.

[0097] Furthermore, the memory of the processing unit 120, the storage unit 130, and the database 140 is Therefore, it can be considered an example of a non-temporary computer-readable storage medium.

[0098] [Display section 150] The display unit 150 has the function of displaying the calculation results from the processing unit 120. Unit 150 has the function of displaying the target document. The display unit 150 also displays the query text. It may have the function of doing so.

[0099] The document data processing system 200 may also have an output unit. The output unit may be external It has the function of supplying data.

[0100] [Transmission path 160] The transmission line 160 has the function of transmitting various data. Input unit 110, processing unit 120, Data transmission and reception between the memory unit 130, the database 140, and the display unit 150 are performed via transmission path 1 This can be done via 60. For example, data such as the target document can be transmitted via transmission line 160. And then it is sent and received.

[0101] <Example of document data processing system configuration 2> Figure 7 shows a block diagram of the document data processing system 210. System 10 includes a server 220 and a terminal 230 (such as a personal computer).

[0102] The server 220 includes a communication unit 161a, a transmission line 162, a processing unit 120, and a storage unit 170. Although not shown in Figure 7, the server 220 may also have input / output units, etc. .

[0103] Terminal 230 includes a communication unit 161b, a transmission line 164, a processing unit 180, a storage unit 130, and a display. It has a section 150. Although not shown in Figure 7, the terminal 230 further has a database, etc. It is acceptable to have it.

[0104] The user of the document data processing system 210 enters a question (question) into the input unit 110 of the terminal 230. Enter the text. The question text is sent from the communication unit 161b of terminal 230 to the communication unit 220 of server 220. It will be sent to 161a.

[0105] The question received by the communication unit 161a is stored in the storage unit 170 via the transmission line 162. Alternatively, the question may be supplied directly from the communication unit 161a to the processing unit 120.

[0106] The document segmentation, distributed representation acquisition, and similarity calculation described in Embodiment 1 are each highly Processing power is required. The processing unit 120 of the server 220 processes the processing of the terminal 230. It has higher processing power compared to unit 180. Therefore, these processes are performed by processing unit 12, respectively. It is preferable to perform this with 0.

[0107] Then, the processing unit 120 calculates the score of the block. The score is calculated based on the transmission line 162. The score is stored in the memory unit 170 via [a certain method]. Alternatively, the score is transmitted directly from the processing unit 120 via [a certain method]. The score may be supplied to unit 161a. The score is sent from the communication unit 161a of server 220 to terminal 23. The data is transmitted to communication unit 161b of terminal 0. The score is displayed on the display unit 150 of terminal 230.

[0108] [Transmission lines 162 and 164] Transmission lines 162 and 164 have the function of transmitting data. Communication unit 161a, processing Data transmission and reception between the processing unit 120 and the storage unit 170 are performed via the transmission line 162. This is possible. Input unit 110, communication unit 161b, processing unit 180, storage unit 130, and display unit 1 Data transmission and reception between 50 can be performed via the transmission line 164.

[0109] [Processing Unit 120 and Processing Unit 180] The processing unit 120 uses the data supplied from the communication unit 161a and the storage unit 170, etc. It has the function of performing calculations. The processing unit 180 consists of a communication unit 161b, a storage unit 130, and a display unit. It has the function of performing calculations using data supplied from 150, etc. Processing unit 120 and The processing unit 180 can refer to the description of the processing unit 120. The processing unit 120 is compared to the processing unit 180. High processing capacity is preferable in all aspects.

[0110] [Storage section 130] The memory unit 130 has the function of storing the program to be executed by the processing unit 180. The memory unit 130 contains the calculation results generated by the processing unit 180, the data input to the communication unit 161b, It also has a function to store data entered into the input unit 110.

[0111] [Storage section 170] The storage unit 170 stores multiple documents, calculation results generated by the processing unit 120, and the communication unit 161a. It has a function to store input data and other information.

[0112] [Communication section 161a and communication section 161b] Using communication units 161a and 161b, data is exchanged between the server 220 and the terminal 230. It can send and receive data. Communication units 161a and 161b include a hub and a log A router, modem, etc. can be used. Data can be transmitted and received using either a wired connection or wireless (e.g.) For example, radio waves, infrared rays, etc. may be used.

[0113] Note that communication between server 220 and terminal 230 is via the World Wide Web (WWW). The foundations of the internet, intranets, extranets, and PANs (Personal Internet Access) LAN (Local Area Network), LAN (Local Area Network) ), CAN (Campus Area Network), MAN (Metropolis tan Area Network), WAN (Wide Area Network) , in computer networks such as GANs (Global Area Networks) This can also be done by connecting.

[0114] This embodiment can be combined with other embodiments as appropriate. [Explanation of Symbols]

[0115] W1: word, W2: word, 1: block, 2: block, 3: block, 4: block, 100: Document data processing system, 101: Document reading unit, 102: Question text input unit, 103 :Document segmentation unit, 104a: Distributed representation acquisition unit, 104b: Distributed representation acquisition unit, 105a: Distributed Expression retention unit, 105b: Distributed expression retention unit, 106: Word selection unit, 107: Similarity calculation unit, 108: Score display unit, 109: Text display unit, 110: Input unit, 120: Processing unit, 130 : Storage unit, 140: Database, 150: Display unit, 160: Transmission line, 161a: Communication unit 161b: Communication unit, 162: Transmission line, 164: Transmission line, 170: Storage unit, 180: Processing Department, 200: Document data processing system, 210: Document data processing system, 220: Server B, 230: Terminal

Claims

[Claim 1] A document reading unit that reads multiple target documents. A document splitting unit that divides each of the aforementioned multiple target documents into multiple blocks, A first distributed representation acquisition unit that acquires word distributed representations for each of the aforementioned blocks, A first distributed representation holding unit stores the distributed representations acquired by the first distributed representation acquisition unit for each target document and each block. Query text reading unit that reads the query text, A second distributed representation acquisition unit extracts words contained in the query text and obtains distributed representations of the words contained in the query text. A second distributed representation holding unit that stores the distributed representation acquired in the second distributed representation acquisition unit, and A similarity calculation unit compares the word embeddings in the query text with the word embeddings in each of the multiple blocks and calculates the similarity for each block. This is a document data processing system that includes, The document data processing system includes a similarity calculation unit which searches for words in the block that match words in the query text, and calculates the similarity between the word embeddings in the block and the word embeddings in the query text for the matching words.

Images