Data processing device

Abstract

The present invention provides a novel data processing device. The data processing device has: a function for accepting document data and a genre list in which a plurality of genre items and other items are listed; a function for splitting up the text data of the document data to create a plurality of chunk data and transmitting first prompts each containing one of the plurality of chunk data to a language model to acquire a plurality of first summary data corresponding to the plurality of chunk data; a function for acquiring second summary data, which is a summary of the document data, by transmitting a second prompt containing the plurality of first summary data to the language model; and a function for acquiring any one of the plurality of genre items and other items that corresponds to the document data and the second summary data by transmitting a third prompt containing the second summary data and the genre list to the language model.

Description

Data Processing Unit

[0001] The present invention relates to a data processing device, a data processing system, a data processing method, a data processing program, etc. that utilize a large language model (LLM).

[0002] The above technical field is one embodiment of the present invention, and the present invention is not limited to the above technical field. Other embodiments of the present invention can include, for example, a semiconductor device, a display device, a light-emitting device, a power storage device, a memory device, an electronic device, a lighting device, an input device (e.g., a touch sensor), an input / output device (e.g., a touch panel), a driving method thereof, or a manufacturing method thereof.

[0003] When searching for desired data from among a plurality of data using keywords, genres (classifications), etc., there is a method of clustering the data into a plurality of clusters in advance. In recent years, a method of clustering data using AI (Artificial Intelligence) technology incorporating machine learning techniques has been proposed (see, for example, Patent Document 1).

[0004] International Publication No. 2020 / 075485

[0005] When multiple data sets include technical documents such as papers (including academic papers, dissertations, doctoral theses, essays, journal articles, etc.) and conference proceedings, it is necessary to understand the contents in advance in order to find the desired data. For example, when searching for desired data from a mixture of multiple technical documents, it may be necessary to understand the selection criteria of the publishing organization or the selection criteria of the type of conference in order to know what keywords, genres, etc. to search for. As a result, it takes time to obtain the desired data from multiple data sets, which could reduce search efficiency.

[0006] In the case of technical documents such as papers and conference proceedings, there is a variation in the information that can be confirmed, such as the presence or absence of an abstract, the number of characters, etc. This results in variation in the content of the abstract, which is important for efficiently understanding the content, and there is a risk that search accuracy will be impaired.

[0007] In order to categorize multiple data by genre, it is effective to create a genre list in advance and then categorize the data according to the genre list. However, if the pre-assigned genre list does not contain an appropriate genre, the amount of data that cannot be categorized by genre (for example, data that is assigned the "Other" genre) will increase. This may significantly impair the convenience of creating genre lists and categorizing data.

[0008] An object of one embodiment of the present invention is to provide a novel data processing device or the like.Another object of one embodiment of the present invention is to provide a highly convenient data processing device or the like.Another object of one embodiment of the present invention is to provide a data processing device or the like with high search efficiency.Another object of one embodiment of the present invention is to provide a data processing device or the like with high search accuracy.

[0009] The present invention does not necessarily solve all of these problems. Furthermore, the description of these problems does not preclude the existence of other problems of the present invention. For example, problems other than these can be extracted from the description of the specification, drawings, and claims.

[0010] One aspect of the present invention is a data processing device having the following functions: a function for accepting document data and a genre list that lists items of a plurality of genres and other items; a function for dividing text data of the document data into a plurality of chunks of data and acquiring a plurality of first summary data corresponding to the plurality of chunks of data by sending a first prompt including one of the plurality of chunks of data to a language model; a function for acquiring second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data; and a function for acquiring one of the items of a plurality of genres and other items corresponding to the document data and the second summary data by sending a third prompt including the second summary data and the genre list to the language model.

[0011] a language model to acquire a plurality of first summary data corresponding to the plurality of chunk data by dividing text data of the document data and sending a first prompt including one of the plurality of chunk data; a language model to acquire a second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data; a language model to acquire one of the plurality of genre items and the other items corresponding to the document data and the second summary data by sending a third prompt including the second summary data and the genre list; a language model to acquire a plurality of genre items and the other items corresponding to the document data and the second summary data; a language model to acquire a third prompt including the second summary data and the genre list; a language model to acquire a plurality of genre items and the other items corresponding to the document data and the second summary data by vectorizing the second summary data from which the other items have been acquired and clustering the second summary data; a language model to acquire a plurality of genre items and the other items corresponding to the plurality of genre items and the other items corresponding to the plurality of chunk data by clustering the second summary data;

[0012] One aspect of the present invention is a data processing device having the following functions: a function of accepting document data and a genre list that lists a plurality of genre items and other items; a function of dividing text data of the document data into a plurality of chunk data, and acquiring a plurality of first summary data corresponding to the plurality of chunk data by sending a first prompt including one of the plurality of chunk data to a language model; a function of acquiring second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data; a function of acquiring any one of the plurality of genre items and other items corresponding to the document data and the second summary data by sending a third prompt including the second summary data and the genre list to the language model; and a function of extracting a plurality of keywords contained in the second summary data by sending a fifth prompt including any one of the second summary data from which the other items have been acquired, selecting keywords present in another second summary data from which the other items have been acquired as genre items, and updating the genre list.

[0013] In one aspect of the present invention, the data processing device preferably divides the text data of the document data into a plurality of chunks of data in accordance with the upper limit of the number of tokens in the first prompt.

[0014] In one aspect of the present invention, the data processing device preferably divides text data of the document data into a plurality of chunks of data according to paragraphs, chapters, a table of contents, or line breaks in the document data.

[0015] In one aspect of the present invention, a data processing device is preferred which has a first memory unit and a second memory unit, and in which document data and second summary data obtained from items of multiple genres are stored in the first memory unit, and second summary data obtained from multiple chunk data, multiple first summary data, and other items are stored in the second memory unit.

[0016] One aspect of the present invention can provide a novel data processing device, etc. Another aspect of the present invention can provide a data processing device, etc. that is highly convenient. Another aspect of the present invention can provide a data processing device, etc. that is highly efficient in search. Another aspect of the present invention can provide a data processing device, etc. that is highly accurate in search.

[0017] The present invention does not necessarily have to have all of these effects. Furthermore, the description of these effects does not preclude the existence of other effects of the present invention. For example, other effects can be extracted from the description of the specification, drawings, and claims.

[0018] FIGS. 1A to 1C are block diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 2 is a flowchart illustrating a data processing device of one embodiment of the present invention. FIGS. 3A and 3B are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 4 is a flowchart illustrating a data processing device of one embodiment of the present invention. FIGS. 5A and 5B are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIGS. 6A to 6E are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIGS. 7A and 7B are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIGS. 8A to 8C are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 9 is a schematic diagram illustrating a data processing device of one embodiment of the present invention. FIG. 10 is a flowchart illustrating a data processing device of one embodiment of the present invention. FIGS. 11A and 11B are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIGS. 12A and 12B are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 13A is a flowchart illustrating a data processing device of one embodiment of the present invention, and FIG. 13B is a schematic diagram illustrating a data processing device of one embodiment of the present invention. 14A to 14C are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 15 is a flowchart illustrating a data processing device of one embodiment of the present invention. FIG. 16A to 16C are schematic diagrams illustrating a data processing device of one embodiment of the present invention. FIG. 17 is a schematic diagram illustrating a data processing device of one embodiment of the present invention.

[0019] The following description will discuss embodiments of the present invention with reference to the accompanying drawings. However, it will be readily understood by those skilled in the art that the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited to the following description of the embodiments.

[0020] In the drawings, the position, size, range, etc. of each component may not accurately represent the actual components, and therefore the position, size, range, etc. of each component are not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0021] In this specification, a language model is based on the Transformer architecture and is incrementally trained to become a conversational (also called dialog-based) model. In other words, a conversational generative model is a lower-level concept of a language model. A language model is also generally called a large-scale language model (LLM).

[0022] In this specification, the words "first" and "second" are used for the convenience of understanding the technical content or to identify each component. Therefore, the words "first" and "second" do not limit the number of each component. Furthermore, the words "first" and "second" do not limit the order of each component. Furthermore, the words "first" and "second" or identifying symbols used in this specification may not match the words or identifying symbols in the claims.

[0023] In this specification and the like, when the same reference numeral is used for multiple elements, and particularly when it is necessary to distinguish between them, the reference numeral may be accompanied by an identifying symbol such as "A," "B," "_1," "_2," or "[n]." Furthermore, when explaining matters common to multiple elements to which the identifying numerals are attached, or when it is not necessary to distinguish between them, the elements may be described without the identifying symbol.

[0024] In this specification, a document refers to a written expression of human intent using characters or symbols. A document may be divided into multiple chapters or multiple paragraphs. A chapter may also consist of multiple paragraphs.

[0025] (Embodiment 1) In this embodiment, an example of a data processing device according to one aspect of the present invention will be described. Data processing according to one aspect of the present invention is processing for categorizing document data such as technical literature, papers, and academic conference proceedings based on genres assigned by a user. By changing the target configuration, the data processing device can be called a data processing system, a data processing method, or a data processing program.

[0026] <Configuration Example of Data Processing Device> A configuration example of a data processing device according to one embodiment of the present invention will be described. Fig. 1A is a block diagram illustrating a data processing device according to one embodiment of the present invention. Fig. 1B is a block diagram illustrating an input device external to the data processing device. Fig. 1C is a block diagram illustrating a data generation device external to the data processing device. Fig. 2 is a flowchart illustrating the operation of the data processing device. Fig. 3A is a schematic diagram illustrating document data input to the data processing device. Fig. 3B is a schematic diagram illustrating a genre list input to the data processing device.

[0027] The data processing device 100 shown in FIG. 1A includes, as an example, an input unit 10, a storage unit 20, a storage unit 30, a data processing unit 40, an output unit 60, and a transmission unit 90.

[0028] In the block diagrams in this specification, components are classified by function and shown as independent blocks. However, in actual integrated circuits, it is difficult to separate components by function, and there may be cases where a single integrated circuit is involved in multiple functions, or where a single function is involved across multiple integrated circuits. For example, although the memory unit 20 and the memory unit 30 are shown as memory units in separate blocks having different functions, they may also be shown as memory units in a single block having different functions.

[0029] The input unit 10 has a function of accepting data from an input device 50 external to the data processing device 100. The input device 50 is a data server or an information terminal such as a desktop computer, a notebook computer, a smartphone, or a tablet computer. For example, if the data input to the input device 50 is document data, the input device 50 is a data server, and if the data input to the input device 50 is list data entered by a user, the input device 50 is an information terminal such as a desktop computer, a notebook computer, a smartphone, or a tablet computer.

[0030] For example, when the provider of a service using the data processing device and the user receiving the service belong to the same organization, such as a company, data transmission between the data processing device 100 and the input device 50 is preferably performed using a network, such as a local area network (LAN), established within the organization. This allows data to be transmitted between the data processing device 100 and the input device 50 more securely than when data is transmitted via the Internet. This also prevents information within the organization from leaking to the outside. Alternatively, data transmission and reception between the data processing device 100 and the input device 50 can be performed using the Internet, which is the foundation of the World Wide Web (WWW).

[0031] 1B, document data DOC and a genre list JL are input to the data processing device 100. The input unit 10 accepts the document data DOC and the genre list JL. The input unit 10 transmits the accepted data to the storage unit 20 via the transmission unit 90.

[0032] The step of receiving the document data DOC and the genre list JL corresponds to step S11 shown in FIG.

[0033] Document data DOC is computerized, machine-readable data that has been collected for a specific purpose, such as papers, academic conference proceedings, technical reports, patent documents, contracts, and product manuals.

[0034] The document data DOC can be schematically illustrated as document data DOC_1 to DOC_3 shown as an example in FIG. 3A . The document data DOC_1 to DOC_3 each have a different amount of data. Document data DOC_1 is composed of document data DOC_1_1 to DOC_1_2. Document data DOC_2 is composed of document data DOC_2_1. Document data DOC_3 is composed of document data DOC_3_1 to DOC_3_k (k is an integer equal to or greater than 3). In addition to the different amounts of data shown in the document data DOC_1 to DOC_3 shown in FIG. 3A , technical documents such as academic papers and conference proceedings often have different formats, such as paragraphs, chapters, tables of contents, and line breaks, depending on the publisher or academic society. Therefore, a simple search using keywords or the like can take a long time to find the desired document data, potentially reducing search efficiency.

[0035] The genre list JL is data that lists items related to multiple genres as well as other items. The items related to multiple genres are subdivided genres that allow users to search for desired data from a mixture of multiple technical documents and understand their contents. The multiple genres included in the genre list JL are preferably data that are assigned in advance by the user. This configuration makes it easier for users to access technical documents assigned to the desired data genre without having to understand the selection criteria of the issuing organization or the selection criteria based on the type of academic society. This reduces the time required to obtain desired document data from document data containing multiple technical documents, thereby improving search efficiency.

[0036] It is preferable that the genre list JL is configured so that genres are updated by the data processing device 100. The updated genre list JL is updated with genres obtained by genre generation other than the genres JD in the genre list JL that the user had previously envisioned. This configuration makes it easier for the user to understand genres in technical documents that the user is not yet aware of.

[0037] An example of the genre list JL is shown in Figure 3B. The genre list JL is composed of "# genre list." In the example of Figure 3B, "# genre list" corresponds to "1. Display," "2. Memory," "3. Processor," and "4. AI." The genre items in Figure 3B are shown as genres JD_1 to JD_4.

[0038] The genre list JL includes an "Other" item. In the example of FIG. 3B, the "Other" item corresponds to "5. Other." The "Other" item in FIG. 3B is illustrated as "Genre JD_OTH." By providing the "Other" item in the genre list JL in addition to items related to multiple genres, it is possible to reduce incorrect genre classification when there is document data that does not correspond to multiple genres assigned by the user.

[0039] The storage units 20 and 30 are memory areas capable of storing programs and / or data, etc. A typical example of a program is a program executed by the data processing unit 40. Data stored in the storage unit 20 includes document data DOC transmitted from the input unit 10, a genre list JL, and summary data generated in accordance with the document data DOC. Data stored in the storage unit 30 includes a portion of document data currently being processed by the data processing unit 40, summary data, etc. This configuration allows the user to access data associated with the document data DOC without accessing the storage unit 30 where the currently being processed data is stored.

[0040] The storage unit 20 and the storage unit 30 include at least one of a volatile memory and a non-volatile memory. Examples of the volatile memory include a dynamic random access memory (DRAM) and a static random access memory (SRAM). Examples of the non-volatile memory include a resistive random access memory (ReRAM), a phase change random access memory (PRAM), a ferroelectric random access memory (FeRAM), a magnetoresistive random access memory (MRAM), and a flash memory.

[0041] The data processing unit 40 has a function of performing data processing such as calculation, analysis, and inference. The data processing unit 40 acquires data from the storage unit 20 and the storage unit 30. The data processing unit 40 has a function of using the data from the storage unit 20 and the storage unit 30 to generate a prompt (question sentence) containing an instruction sentence, a sentence, etc. to be transmitted to the outside of the data processing device 100.

[0042] The data processing unit 40 has at least an arithmetic circuit to realize the above functions. The arithmetic circuit may include, for example, a central processing unit (CPU). The data processing unit 40 may also include a graphics processing unit (GPU) in addition to or instead of the CPU.

[0043] The data processing unit 40 may have a microprocessor such as a DSP (Digital Signal Processor) in addition to a CPU or GPU. Since a DSP is specialized for digital signal processing, it is preferable to install a DSP to control peripheral circuits of the CPU or GPU. The microprocessor may also be implemented by a PLD (Programmable Logic Device) that operates on hardware such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array).

[0044] The data processing unit 40 has the function of performing a plurality of processes using the data in the storage units 20 and 30. The data processing unit 40 has the function of generating prompts for executing processes such as summary data generation, genre classification, and genre generation. Fig. 1A illustrates the configuration for generating prompts, such as a summary data generation unit 41 and a genre classification unit 42, within the data processing unit 40.

[0045] The summary data generation unit 41 has a function of outputting a prompt for generating summary data corresponding to each of the plurality of document data DOC in the data generation device 70, and performing data processing of the document data DOC for the prompt. The summary data is data generated by the data generation device 70 after data processing is performed for each of the plurality of document data DOC by the data processing device 100.

[0046] The step of generating the summary data ABD based on the document data DOC corresponds to step S12 shown in FIG.

[0047] The configuration including a summary data generator allows for the generation of summary data based on uniform standards for each of multiple document data. This configuration allows for the association (also called linking) of summary data from technical documents such as papers and academic conference proceedings to document data as summary data generated based on uniform standards. This configuration eliminates variation in the content of summary data, which is important for efficiently understanding the content of document data, and improves readability.

[0048] The genre sorting unit 42 assigns one of the genres described in the genre list JL to each piece of document data DOC, and has the function of outputting a prompt to cause the data generation device 70 to generate a genre corresponding to the document data DOC, and performing data processing of the summary data and document data DOC for the prompt.

[0049] The step of acquiring the genre JD items based on the genre list JL assigned by the user based on the summary data ABD corresponds to step S13 shown in FIG.

[0050] The genre classification unit 42 allows each of a plurality of document data DOC to be assigned a genre based on the genre list JL assigned by the user, using summary data generated based on uniform criteria. As a result, the document data DOC can be associated with the summary data generated based on uniform criteria and any of the genres listed in the genre list JL assigned to the data processing device by the user based on the summary data. Therefore, by selecting a genre, the content of the document data can be understood from the summary data generated based on uniform criteria, thereby improving search efficiency.

[0051] The summary data ABD associated with the document data DOC in step S12 and the genre JD associated with the document data DOC in step S13 are stored in the storage unit 20. This step corresponds to step S14 shown in FIG.

[0052] One aspect of the present invention is particularly suited to data processing for searching document data that is difficult to search because it is a collection of document data of similar genres, particularly when the document data is a mixture of technical documents such as papers and conference proceedings of similar genres from different publishers and academic societies. This is because technical documents such as papers and conference proceedings of similar genres are categorized into genres (fields) according to different conventions for each publisher and academic society. By re-sorting the genres (fields) by technical field according to a certain convention, as in one aspect of the present invention, it is possible to perform genre (field) classification by technical field according to a certain convention, as with patent documents. Therefore, the data that has undergone this data processing can be used to improve search efficiency when searching for desired document data.

[0053] 1A, in addition to the summary data generating unit 41 and the genre sorting unit 42, a genre generating unit 43 is also shown within the data processing unit 40. The data processing unit 40 included in the data processing device 100 preferably includes the genre generating unit 43.

[0054] The genre generation unit 43 has the function of outputting a prompt to generate a new genre that is not included in the genre list JL initially entered by the user when an item of other genres (genre JD_OTH in Figure 3B) is obtained by a prompt from the genre classification unit 42, and performing data processing of the summary data for the prompt.

[0055] The configuration including the genre generation unit 43 allows a new genre to be generated for each piece of summary data to which other genres have been assigned. Since the genre list can be updated based on summary data generated based on uniform criteria, the number of document data DOC and summary data to which other genres have been assigned can be reduced, and the convenience of creating genre lists and sorting by genre can be improved.

[0056] The output unit 60 has a function of transmitting a prompt containing instructions, sentences, etc. to be sent to a data generation device 70 external to the data processing device 100. As shown in Fig. 1C, the data processing device 100 receives a prompt PD from the output unit 60. The prompt PD is data including, in addition to instructions added by the data processing unit 40, text data generated by processing document data DOC in the data processing unit 40, and / or a genre list JL.

[0057] The output unit 60 has a function of receiving data generated by a data generation device 70 external to the data processing device 100. As shown in FIG. 1C , the data processing device 100 receives generated data GD (also referred to as an answer sentence) generated by the output unit 60 in response to the instruction sentence of the prompt PD. The output unit 60 acquires the generated data GD. The output unit 60 transmits the acquired data to the data processing unit 40, the memory unit 20, or the memory unit 30 via the transmission unit 90.

[0058] For example, when a provider of a service using the data generating device 70 and a user receiving the service belong to the same organization such as a company, data reception between the data processing device 100 and the data generating device 70 is preferably performed using a network established within the organization. This allows data to be received between the data processing device 100 and the data generating device 70 more securely than when data is received via the Internet. This also prevents information within the organization from leaking to the outside. Alternatively, data transmission and reception between the data processing device 100 and the data generating device 70 may be performed using the Internet.

[0059] The transmission unit 90 has a function of transmitting data. Data can be transmitted and received between the input unit 10, the storage units 20 and 30, the data processing unit 40, and the output unit 60 via the transmission unit 90.

[0060] The data generating device 70 can perform processing using a language model. For example, it can execute processing using a natural language processing model such as BERT (Bidirectional Encoder Representations from Transformer) or T5 (Text-to-Text Transfer Transformer). The data generating device 70 can execute processing using a general-purpose language processing model that can perform various natural language processing tasks. The data generating device 70 is a device that performs processing using a language model, and may be simply referred to as a language model.

[0061] The data generation device 70 can also perform processing using a model that utilizes an LLM (such as a document generation model or a dialogue model). For example, the LLM can be GPT-3, GPT-3.5, GPT-4, LaMDA (Language Model for Dialogue Applications), PaLM (Pathways Language Model), Llama2, or the like. Of course, a model built within an organization can be used as the LLM.

[0062] The data generating device 70 can input generated data GD, which is a response sentence to a prompt PD (question sentence) output from the data processing device 100, to the data processing device 100 through processing using a language model.

[0063] The prompt PD created by the data processing unit 40 will be described with reference to Figures 4 to 17. The following describes the prompts for executing processes such as summary data generation, genre classification, and genre generation performed by the data processing unit 40, and the data processing of document data DOC.

[0064] <Configuration Example of Summary Data Generation> FIG. 4 is a flowchart illustrating summary data generation performed by summary data generation unit 41 of data processing unit 40. FIG. 5A is a schematic diagram illustrating chunk data CHD in document data DOC. FIG. 5B is a schematic diagram of storage unit 30 in which chunk data CHD is stored. FIGS. 6A to 6E are schematic diagrams of prompt PD_1 including chunk data CHD and first summary data CAB corresponding to chunk data CHD, which are input from data processing device 100 to data generation device 70. FIG. 7A is a schematic diagram of storage unit 30 in which first summary data CAB corresponding to chunk data CHD is stored. FIG. 7B is a schematic diagram of prompt PD_1 including text data OD1 and chunk data CHD. FIG. 8A is a schematic diagram of prompt PD_2 including multiple pieces of first summary data CAB, which are input from data processing device 100 to data generation device 70, and summary data ABD corresponding to multiple pieces of first summary data CAB. Fig. 8B is a schematic diagram of prompt PD_2 including text data OD2 and a plurality of first summary data CAB. Fig. 8C is a schematic diagram of storage unit 20 in which summary data ABD (corresponding to second summary data) corresponding to the plurality of first summary data CAB is stored in association with document data DOC. Fig. 9 is a schematic diagram illustrating the first summary data and second summary data used to obtain summary data for each of a plurality of document data DOC_1 to DOC_3.

[0065] As shown in FIG. 4, the data processing unit 40 described in FIG. 1A has the following steps for generating summary data: generating chunk data CHD (step S21), saving the chunk data CHD (step S22), generating first summary data CAB based on prompt PD_1 (step S23), saving the first summary data CAB (step S24), generating summary data ABD (second summary data) based on prompt PD_2 (step S25), and saving the summary data ABD (step S26).

[0066] Chunk data is text data that groups together connected parts of document data. First summary data is data that summarizes chunk data and corresponds to a summary of a portion of the document data. Second summary data is data that is aggregated based on multiple pieces of first summary data.

[0067] 5A shows document data DOC as an example of chunk data CHD in step S21. The document data DOC has multiple paragraphs DOC_P1 to DOC_P6. As shown in FIG. 5A, if the document data DOC is text data with multiple columns, paragraphs DOC_P3 and DOC_P4 that are in the same heading (paragraph, chapter, table of contents, or line break) are treated as separate chunks.

[0068] Chunk data CHD is text data in which paragraphs under the same heading (paragraph, chapter, table of contents, or line break) are treated as connected parts within document data DOC and organized into a single block. This configuration eliminates content variation when generating primary summary data, improving readability. In the document data DOC of Figure 5A, paragraphs DOC_P3 and DOC_P4 are merged as meaningful blocks and illustrated as chunk data CHD_1 to CHD_5. Chunk data CHD_1 to CHD_5 can be considered data obtained by dividing document data DOC.

[0069] 5B illustrates chunk data CHD_1 to CHD_5 stored in the storage unit 30 as an example of saving chunk data CHD in step S22. The chunk data CHD is data that is being processed to obtain summary data ABD of document data DOC. By separating the storage unit 30, which is not accessed by the user, from the storage unit 20 in which document data DOC is stored, the process of obtaining chunk data CHD can be sped up.

[0070] 6A to 6E are schematic diagrams illustrating an example of step S23 in which the data processing device 100 transmits a prompt PD_1 having chunk data CHD_1 to CHD_5 to the data generating device 70 to generate first summary data CAB_1 to CAB_5. The prompt PD_1 is sometimes referred to as the first prompt. The prompt PD_1 is a prompt in which an instruction sentence and chunk data CHD are written so that the first summary data CAB becomes the answer sentence of the data generating device 70.

[0071] A prompt PD_1 having any one of chunk data CHD_1 to CHD_5 is transmitted multiple times from the data processing device 100 to the data generating device 70. This configuration makes it possible to prevent the prompt PD_1 from exceeding the upper limit of the number of tokens. Each of the first summary data CAB_1 to CAB_5 corresponding to the multiple chunk data CHD_1 to CHD_5 can be summary data of text data with coherent content.

[0072] 7A illustrates first summary data CAB_1 to CAB_5 stored in storage unit 30 as an example of saving first summary data CAB in step S24. First summary data CAB is data that is being processed to obtain summary data ABD of document data DOC. By separating storage unit 30, which is not accessed by the user, from storage unit 20, in which document data DOC is stored, the process of obtaining first summary data CAB can be sped up.

[0073] FIG. 7B is a schematic diagram illustrating prompt PD_1. Prompt PD_1 is composed of, for example, "#instruction" and "#sentence." Text data OD1 is written in "#instruction." Any one of chunk data CHD_1 to CHD_5 is written in "#sentence." Text data OD1 is a fixed phrase. An example of the fixed phrase could be, "Please summarize the following text in 400 characters or less. Please try to include as many numerical values ​​as possible from the original text."

[0074] 8A is a schematic diagram illustrating an example of step S25 in which data processing device 100 transmits prompt PD_2 containing first summary data CAB_1 through CAB_5 (corresponding to a plurality of first summary data CAB) to data generating device 70 to generate summary data ABD (second summary data) that is a summary of document data DOC. Prompt PD_2 is sometimes referred to as the second prompt. Prompt PD_2 is a prompt that includes an instruction statement and a plurality of first summary data CAB so that summary data ABD serves as a response statement for data generating device 70. In the case of first summary data CAB generated from chunk data that includes the contents of original document data DOC, the generated summary data ABD is summary data that conforms to the contents of document data DOC.

[0075] For a prompt PD_2 having multiple first summary data CAB, it is preferable to transmit the first summary data CAB that the prompt PD_2 has to the data generating device 70 in multiple installments so that the prompt PD_2 does not exceed the upper limit of the number of tokens.

[0076] FIG. 9 is a schematic diagram for explaining how the summary data ABD is obtained for each of the plurality of document data DOC_1 to DOC_3 via the chunk data CHD and the first summary data CAB.

[0077] 9, summary data ABD_2 is obtained via chunk data CHD_1 to CHD_5 and first summary data CAB_1 to CAB_5. The first summary data CAB_1 to CAB_5 can be called primary summary data used to obtain summary data ABD_2, and summary data ABD_2 can be called secondary summary data.

[0078] 9 is document data with a larger amount of text than document data DOC_2. In this case, the text data of the original document data DOC_1 is divided, for example, into chunk data CHD_1 to CHD_10, and first summary data CAB_1 to CAB_10 is obtained based on prompt PD_1. If the number of tokens that can be processed together with prompt PD_2 for first summary data CAB_1 to CAB_10 exceeds the number of tokens that can be processed, the first summary data CAB_1 to CAB_5 and first summary data CAB_6 to CAB_10 are processed separately. First summary data CAB_1-5 can be obtained from first summary data CAB_1 to CAB_5 based on prompt PD_2. First summary data CAB_6-10 can be obtained from first summary data CAB_6 to CAB_10 based on prompt PD_2. For document data DOC_1, summary data ABD_1 is obtained via chunk data CHD_1 to CHD_10, first summary data CAB_1 to CAB_10, first summary data CAB_1-5, and first summary data CAB_6-10. In this case, first summary data CAB_1 to CAB_10 can be called primary summary data, first summary data CAB_1-5 and first summary data CAB_6-10 can be called secondary summary data, and summary data ABD_1 can be called tertiary summary data.

[0079] 9 is document data with a larger amount of text than document data DOC_1 and DOC_2. In this case, the text data of original document data DOC_3 is divided, for example, into chunk data CHD_1 to CHD_k, and first summary data CAB_1 to CAB_k is obtained based on prompt PD_1. Like document data DOC_1, first summary data CAB_1 to CAB_k exceeds the number of tokens that can be processed together with prompt PD_2. In this case, the data is divided and processed, for example, into first summary data CAB_1 to CAB_5, first summary data CAB_6 to CAB_10, first summary data CAB_k-4 to CAB_k, etc. From first summary data CAB_1 to CAB_5, first summary data CAB_1-5 can be obtained based on prompt PD_2. The first summary data CAB_6 to CAB_10 can be obtained as first summary data CAB_6-10 based on the prompt PD_2. The first summary data CAB_k-4 to CAB_k can be obtained as first summary data CAB_k-4-k based on the prompt PD_2.

[0080] In document data DOC_3, first summary data CAB_P and first summary data CAB_Q can be obtained by generating first summary data CAB based on prompt PD_2 multiple times (e.g., N-1 times (N is a natural number greater than or equal to 3)), such as chunk data CHD_1 to CHD_k, first summary data CAB_1 to CAB_k, first summary data CAB_1-5, CAB_6-10 to, CAB_k-4-k, etc. In document data DOC_3, first summary data CAB_P and first summary data CAB_Q can be obtained. In this case, the first summary data CAB_1 to CAB_10 can be called primary summary data, the first summary data CAB_1-5, the first summary data CAB_6-10, and the first summary data CAB_k-4-k can be called secondary summary data, the first summary data CAB_P and the first summary data CAB_Q can be called (N-1)th summary data, and the summary data ABD_3 can be called Nth summary data.

[0081] As shown in the document data DOC_1 to DOC_3 in FIG. 9 , when generating summary data using the data processing device 100, document data with different amounts of text is processed. To obtain summary data ABD from the document data, chunk data CHD and first summary data CAB, each divided into different numbers, are processed multiple times. This configuration allows summary data to be generated based on uniform standards. As a result, summary data for technical documents such as papers and academic conference proceedings can be generated based on uniform standards. Furthermore, this eliminates variation in the content of summary data, which is important for efficiently understanding the content, thereby improving readability.

[0082] FIG. 8B is a schematic diagram illustrating prompt PD_2. Prompt PD_2 is composed of, for example, "#instruction" and "#sentence." Text data OD2 is written in "#instruction." Multiple pieces of first summary data CAB (for example, first summary data CAB_1 to CAB_5) are written in "#sentence." Text data OD2 is a fixed phrase. Note that the fixed phrase can be, for example, "Please summarize the following text in 400 characters or less. Please try to include as many numerical values ​​as possible that appear in the original text."

[0083] FIG. 8C illustrates summary data ABD_1 to ABD_3 stored in the storage unit 20 as an example of saving summary data ABD (second summary data) in step S26. Summary data ABD_1 to ABD_3 can be stored corresponding to document data DOC_1 to DOC_3, respectively. Summary data ABD is data accessed by the user during a search. By providing separate storage units for the storage unit 20 accessed by the user and the storage unit 30 storing data in the middle of processing, summary data generation and user access can be separated. This prevents delays in user access due to the processing involved in generating summary data, thereby speeding up the process of accessing summary data ABD.

[0084] If the document data DOC is written in a foreign language, the summary data ABD will also be written in a foreign language. If the summary data ABD stored in the storage unit 20 is written in a foreign language, it is preferable to translate it into the user's native language. This configuration improves readability when the user accesses the storage unit 20 to search for information.

[0085] In addition to the above-described configuration, methods such as the MAP Reduce method and the Refine method can also be applied as a method for creating summary data based on document data DOC. In addition to the above-described configuration, the method for creating summary data can also be a combination of the above-described methods.

[0086] As described above, a data processing device according to one embodiment of the present invention includes a summary data generator 41, which can generate summary data based on uniform standards for each of multiple document data DOC. This configuration allows the summary data of technical documents, such as papers and academic conference proceedings, to be generated based on uniform standards. This configuration eliminates variations in the content of summary data, which is important for efficiently understanding the content, and improves readability.

[0087] <Configuration Example of Genre Sorting> Figure 10 is a flowchart explaining genre sorting performed by the genre sorting unit 42 of the data processing unit 40. Figure 11A is a schematic diagram of summary data ABD input from the data processing device 100 to the data generating device 70, a prompt PD_3 including a genre list JL, and a genre JD corresponding to the summary data ABD. Figure 11B is a schematic diagram of prompt PD_3 including text data OD3, summary data ABD, and a genre list JL. Figure 12A is a schematic diagram of the storage unit 20 in which summary data ABD and genre JD associated with document data DOC are stored. Figure 12B is a schematic diagram of the storage unit 30 in which summary data ABD assigned to other genres is stored.

[0088] As shown in FIG. 10, the data processing unit 40 described in FIG. 1A performs genre classification by obtaining a genre JD based on the prompt PD_3 (step S31), saving the genre JD (step S32), determining whether there is a plurality of summary data assigned to other genres (step S33), generating a genre JD (step S34), determining whether the generated genre JD is appropriate (step S35), and updating the genre list JL (step S36).

[0089] 11A is a schematic diagram illustrating an example of step S31 in which the data processing device 100 transmits a prompt PD_3 having summary data ABD and a genre list JL to the data generating device 70, and data is generated for selecting an item in the genre JD described in the genre list JL corresponding to the summary data ABD. The prompt PD_3 is sometimes referred to as a third prompt. The prompt PD_3 is a prompt that describes an instruction sentence, the summary data ABD, and the genre list JL so that one of the items in the genre JD described in the genre list JL becomes the answer sentence of the data generating device 70.

[0090] FIG. 11B is a schematic diagram illustrating prompt PD_3. Prompt PD_3 is composed of, for example, "#instruction," "#sentence," and "#genre list." Text data OD3 is written in "#instruction." Summary data ABD corresponding to document data DOC is written in "#sentence." A genre list JL including multiple genres JD (genres JD_1 to JD_4 and JD_OTH are exemplified) is written in "#genre list." Text data OD3 is a fixed phrase. Note that the fixed phrase can be, for example, "Please select which of genres 1 to 5 the content of the following sentence belongs to. Please answer only with the genre number."

[0091] 12A illustrates genres JD_1 to JD_3 stored in the storage unit 20 as an example of genre JD storage in step S32. Genres JD_1 to JD_3 can be stored corresponding to document data DOC_1 to DOC_3 and summary data ABD_1 to ABD_3, respectively. Genre JD, summary data ABD, and document data DOC are data accessed by the user during a search. By separating the storage unit 20 accessed by the user from the storage unit 30 that stores data in the middle of processing, the process of accessing genre JD can be sped up.

[0092] 12B illustrates an example of a case where there is a plurality of summary data items for which the "other genre" item has been acquired in step S33, where the summary data items ABD_4 to ABD_6 are associated with the genre JD_OTH. The summary data items ABD_4 to ABD_6 for which the "other genre" item has been acquired can be stored in the storage unit 30.

[0093] When there are multiple pieces of summary data ABD from which the genre JD_OTH item has been acquired, as shown in Figure 12B, it may be difficult to understand the contents of the summary data ABD from which the genre JD_OTH item has been acquired and the document data DOC. In this case, a preferred configuration is to use the genre generation method described below to generate genres into which the summary data ABD from which the genre JD_OTH item has been acquired can be classified and then sorted by genre. In this configuration, the summary data ABD from which the genre JD_OTH item has been acquired is data that is being processed for genre generation and becomes a memory unit 30 that the user does not access. By separating the memory unit 30 that the user does not access from the memory unit 20 in which the document data DOC is stored, the genre generation process can be sped up.

[0094] As described above, the storage unit 20 stores the summary data ABD and genre JD items associated with the document data. The storage unit 30 stores the chunk data CHD, the primary summary data, and the summary data ABD from which other genre items have been acquired (added). This configuration allows the user to access the storage unit 20 in which the data associated with the document data DOC is stored, without accessing the storage unit 30 in which the data being processed is stored.

[0095] As described above, the data processing device according to one embodiment of the present invention includes a genre classification unit 42, which allows it to generate genres for each of a plurality of document data DOC based on a genre list assigned by the user, using summary data generated based on uniform criteria. Because genre classification is performed based on summary data generated based on uniform criteria, search efficiency can be improved.

[0096] One aspect of the present invention is particularly suited to data processing for searching document data that is difficult to search because it is a collection of document data of similar genres, particularly when document data DOC is a mixture of technical documents such as papers and conference proceedings of similar genres from different publishers and academic societies. This is because technical documents such as papers and conference proceedings of similar genres are categorized into genres (fields) according to different conventions for each publisher and academic society. By re-sorting the genres (fields) by technical field according to a certain convention, as in one aspect of the present invention, it is possible to perform genre (field) classification by technical field according to a certain convention, as with patent documents. Therefore, the data that has undergone this data processing can be used to improve search efficiency when searching for desired document data.

[0097] <Configuration Example 1 of Genre Generation> Figure 13A is a flowchart illustrating genre generation performed by the genre generation unit 43 of the data processing unit 40, corresponding to step S34 described in Figure 10. Figure 13B is a schematic diagram of a cluster associated with vectorization used in genre generation and summary data CCAB at the center of the cluster CC. Figure 14A is a schematic diagram of summary data CCAB input from the data processing device 100 to the data generating device 70, a prompt PD_4 including the summary data CCAB, and a generated genre JD_N. Figure 14B is a schematic diagram of a prompt PD_4 including a directive and summary data CCAB. Figure 14C is a schematic diagram of a genre list JL_N in which the generated genre JD_N is added to the genre list JL illustrated in Figure 3B.

[0098] In the data processing unit 40 described in FIG. 1A, as shown in FIG. 13A, the steps for generating genres include vectorizing and clustering the summary data ABD from which the items of genre JD_OTH have been obtained (step S41), selecting the summary data ABD at the center of cluster CC as summary data CCAB (step S42), and generating genre JD_N based on prompt PD_4 (step S43).

[0099] A cluster CC is a set of summary data obtained by vectorizing the summary data ABD from which the genre JD_OTH items have been acquired. Various methods can be used to vectorize the summary data ABD. For example, Bag-of-Words, BERT, etc. can be used. To determine the summary data CCAB at the center of a cluster CC, the center point of the cluster CC of the summary data ABD from which the genre JD_OTH items have been acquired can be calculated, and the summary data ABD at vector coordinates closest to the center point can be selected as the summary data CCAB.

[0100] 13B is a diagram showing an example of steps S41 and S42, which schematically illustrates clusters CC_1 to CC_3 obtained by vectorizing summary data ABD from which genre JD_OTH items have been acquired. The summary data ABD from which genre JD_OTH items have been acquired can be roughly divided into, for example, three clusters CC_1 to CC_3. Central summary data CCAB_1 to CCAB_3 are selected from each of the clusters CC_1 to CC_3.

[0101] 14A is a schematic diagram illustrating an example of step S43 in which the data processing device 100 transmits a prompt PD_4 having summary data CCAB to the data generating device 70 to generate a new genre JD_N corresponding to the summary data CCAB. The prompt PD_4 is sometimes referred to as the fourth prompt. The prompt PD_4 is a prompt that includes an instruction to respond with a new genre JD_N and the summary data CCAB.

[0102] FIG. 14B is a schematic diagram illustrating prompt PD_4. Prompt PD_4 is composed of, for example, "#instruction" and "#sentence." Text data OD4 is written in "#instruction." Summary data CCAB selected in step S42 is written in "#sentence." Text data OD4 is a fixed phrase. An example of the fixed phrase could be, "Please tell me the genre name based on the content of the following sentence."

[0103] It is preferable that the "# sentence" included in prompt PD_4 instructs the user to answer a new genre JD_N by combining not only the summary data CCAB but also other nearby summary data within the cluster CC to which the summary data CCAB belongs. This configuration allows for a highly accurate answer. Furthermore, by repeatedly replacing other nearby summary data within the cluster CC and obtaining an answer with the same instruction, a highly accurate answer can be obtained. If multiple answers are obtained by multiple instruction statements, it is also possible to inquire of the data generation device 70 about the optimal solution.

[0104] FIG. 14C is a schematic diagram of a genre list JL_N when the genre list JL shown in FIG. 3B is updated based on the above-described genre generation step. The genre list JL_N is composed of "# genre list." The "# genre list" includes each genre JD described in the genre list JL as well as an added genre JD_N. In the example of FIG. 14C, the added genre item corresponds to "5. Battery." The numbers assigned to the genres are also updated. The genre list update corresponds to step S36 shown in FIG. 10.

[0105] It is preferable to inquire of the data generating device 70 whether the genre JD_N obtained by steps S41 to S43 is valid. This step corresponds to step S35 shown in Fig. 10. Step S35 can be omitted.

[0106] <Configuration Example 2 of Genre Generation> Figure 15 is another flowchart illustrating genre generation performed by the genre generation unit 43 of the data processing unit 40, corresponding to step S34 described in Figure 10. Figure 16A is a schematic diagram of summary data ABD input from the data processing device 100 to the data generating device 70, a prompt PD_5 including the summary data ABD, and a generated keyword list KWL. Figure 16B is a schematic diagram of a directive and prompt PD_5 including the summary data ABD. Figure 16C is a schematic diagram of the generated keyword list KWL. Figure 17 is a schematic diagram illustrating matching of the keyword list KWL with another summary data ABD_7.

[0107] As shown in FIG. 15, the data processing unit 40 described in FIG. 1A performs the steps of genre generation, which are to extract keywords KW based on the prompt PD_5 (step S51), associate the extracted keywords KW with summary data ABD and store them (step S52), determine whether the extracted keywords KW exist in other summary data ABD (step S53), and select the extracted keywords KW into a genre list JL_N (step S54).

[0108] 16A is a schematic diagram illustrating an example of step S51, in which the data processing device 100 transmits a prompt PD_5 having summary data ABD from which the item of genre JD_OTH has been acquired to the data generating device 70, and a keyword list KWL having a plurality of keywords KW corresponding to the summary data ABD is generated. The prompt PD_5 is sometimes referred to as the fifth prompt. The prompt PD_5 is a prompt that includes an instruction to answer a new genre JD_N and summary data CCAB.

[0109] The keyword list KWL containing the plurality of keywords KW obtained in step S51 is stored in association with the summary data ABD. The keyword list KWL containing the keywords KW is preferably stored in the storage unit 30, similar to the chunk data, primary summary data, etc. described above.

[0110] FIG. 16B is a schematic diagram illustrating prompt PD_5. Prompt PD_5 is composed of, for example, "#instruction" and "#sentence." Text data OD5 is written in "#instruction." Any one of the summary data ABD from which the items of genre JD_OTH have been acquired is written in "#sentence." Text data OD5 is a fixed phrase. Note that the fixed phrase can be, for example, "Please extract the main keywords from the content of the following sentence. Please list up to three keywords in order of importance. Please answer with only the keywords."

[0111] An example of a keyword list KWL having a plurality of keywords KW can be illustrated in FIG. 16C . The keyword list KWL is composed of “#keywords” having a plurality of keywords KW. In the example of FIG. 16C , the keywords KW correspond to “1. oxide,” “2. nitride,” and “3. metal.” The keywords KW in FIG. 16C are illustrated as keywords KW_1 to KW_3.

[0112] FIG. 17 is a schematic diagram illustrating, as an example of step S53, the matching of the keyword list KWL with another summary data ABD_7. In the schematic diagram shown in FIG. 17, the summary data ABD_7 contains the same term "nitride" as the keyword KW_2. Meanwhile, the summary data ABD_7 contains no or few terms corresponding to the keywords KW_1 and KW_3. In this case, as an example of step S54, the keyword KW_2 is selected as the genre JD_N. If the summary data ABD_7 does not contain any of the keywords KW_1 to KW_3, genre generation using the generated keyword list KWL is terminated.

[0113] The summary data ABD_7 may be another summary data ABD from the summary data ABD from which the genre JD_OTH item was acquired. It is also possible to make the summary data ABD_7 belong to the same cluster by the above-mentioned clustering.

[0114] As described above, the data processing device according to one embodiment of the present invention includes a genre generation unit 43, which allows it to generate a new genre for each piece of summary data to which other genres have been assigned. Because the genre list can be updated based on summary data generated based on uniform criteria, it is possible to reduce the number of document data DOC and summary data to which other genres have been assigned, and to improve the convenience of creating genre lists and sorting by genre.

[0115] ABD: summary data, CAB: first summary data, CC: cluster, CCAB: summary data, CHD: chunk data, DOC: document data, GD: generated data, JD: genre, JL: genre list, KW: keyword, KWL: keyword list, PD: prompt, 10: input unit, 20: memory unit, 30: memory unit, 40: data processing unit, 41: summary data generation unit, 42: genre classification unit, 43: genre generation unit, 50: input device, 60: output unit, 70: data generation device, 90: transmission unit, 100: data processing device

Claims

A function of receiving document data and a genre list in which items of a plurality of genres and other items are listed; a function of dividing text data of the document data into a plurality of chunk data, and transmitting a first prompt including one of the plurality of chunk data to a language model, thereby obtaining a plurality of first summary data corresponding to the plurality of chunk data; a function of obtaining second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data to the language model; a function of acquiring any one of the items of the plurality of genres and the other items corresponding to the document data and the second summary data by transmitting a third prompt including the second summary data and the genre list to the language model; Data processing device. 　 A function of receiving document data and a genre list in which items of a plurality of genres and other items are listed; a function of dividing text data of the document data into a plurality of chunk data, and transmitting a first prompt including one of the plurality of chunk data to a language model, thereby obtaining a plurality of first summary data corresponding to the plurality of chunk data; a function of obtaining second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data to the language model; a function of acquiring any one of the items of the plurality of genres and the other items according to the document data and the second summary data by transmitting a third prompt including the second summary data and the genre list to the language model; and a function of vectorizing and clustering the second summary data from which the other items have been acquired; a function of selecting, from the second summary data classified into a plurality of clusters by the clustering and from which the other items have been acquired, the second summary data that is at the center of the cluster; a function of acquiring a genre item corresponding to the cluster by transmitting a fourth prompt including the selected second summary data to the language model, and updating the genre list; Data processing device. 　 A function of receiving document data and a genre list in which items of a plurality of genres and other items are listed; a function of dividing text data of the document data into a plurality of chunk data, and transmitting a first prompt including one of the plurality of chunk data to a language model, thereby obtaining a plurality of first summary data corresponding to the plurality of chunk data; a function of obtaining second summary data that is a summary of the document data by sending a second prompt including the plurality of first summary data to the language model; a function of acquiring any one of the items of the plurality of genres and the other items according to the document data and the second summary data by transmitting a third prompt including the second summary data and the genre list to the language model; a function of extracting a plurality of keywords contained in the second summary data by sending a fifth prompt including any one of the second summary data from which the other items have been acquired to the language model, selecting the keywords present in the other second summary data from which the other items have been acquired as genre items, and updating the genre list; Data processing device. 　 In any one of claims 1 to 3, the text data of the document data is divided into the plurality of chunk data in accordance with an upper limit of the number of tokens of the first prompt; Data processing device. 　 In any one of claims 1 to 3, the text data of the document data is divided into the plurality of chunk data according to paragraphs, chapters, a table of contents, or line breaks in the document data; Data processing device. 　 In any one of claims 1 to 3, A first storage unit and a second storage unit are included, the document data and the second summary data from which the plurality of genre items have been acquired are stored in the first storage unit; the second summary data from which the plurality of chunk data, the plurality of first summary data, and the other items have been acquired is stored in the second storage unit; Data processing device.

Images