Data generation method

Abstract

This data generation method for generating transmission data to be transmitted to a generative AI 5 includes the following. A plurality of partial materials are generated by dividing materials 3 to be referenced by the generative AI 5, and are stored in a database 2. A summary of each of the plurality of partial materials is obtained, and the plurality of summaries are associated with the plurality of partial materials and stored in the database 2. Summary vectors V1-1 to V1-N1 generated respectively from the plurality of summaries are obtained, and the plurality of summary vectors V1-1 to V1-N1 are associated with the plurality of summaries and stored in a vector database 21 included in the database 2. An input of an instruction sentence 4 is received, and an instruction vector Vp generated from the instruction sentence 4 is obtained. A similarity search is performed on the vector database 21, and the partial materials corresponding to summary vectors V1-10, V1-11, and V1-12 that have high similarity to the instruction vector Vp are extracted from the database 2. Transmission data including the instruction sentence 4 and the extracted partial materials is generated.

Description

Data generation method

[0001] The present invention relates to a data generation method for generating transmission data to be transmitted to a generative artificial intelligence (AI).

[0002] Today, generative AI, which generates text data in response to instructions, is rapidly advancing. The data generated by cutting-edge generative AI is now so accurate that it is indistinguishable from text written by humans, and it can be generated very quickly.

[0003] Operating such a high-performance generative AI requires a huge amount of energy, including the power consumed by the computers and the cooling systems in the computer room. The more instructions received from more users and the larger the data associated with each instruction, the more energy is required to process them, and costs increase.

[0004] For this reason, many generation AIs have an upper limit on the amount of data (number of tokens) that can be sent and received per instruction. For example, if you want the generation AI to create data based on the information in a document, and the information in the document has not yet been learned by the generation AI, the data for that information must be provided to the generation AI. If the amount of data in that document exceeds the upper limit on the number of tokens, the document cannot be provided to the generation AI.

[0005] Even if there is no upper limit on the number of tokens, it is desirable to keep the amount of data exchanged small to prevent energy waste.

[0006] In one aspect of the present invention, a data generation method for generating transmission data to be sent to a generation AI includes: generating a plurality of partial documents by dividing a document to be referenced by the generation AI and storing the document in a database; obtaining a summary of each of the plurality of partial documents, associating the plurality of summaries with the plurality of partial documents and storing the plurality of summary vectors in the database; obtaining a summary vector generated from each of the plurality of summaries, associating the plurality of summary vectors with the plurality of summaries and storing the plurality of summary vectors in a vector database included in the database; accepting input of an instruction statement and obtaining the instruction vector generated from the instruction statement; performing a similarity search of the vector database to extract from the database partial documents corresponding to summary vectors that have a high similarity to the instruction vector; and generating the transmission data including the instruction statement and the extracted partial documents.

[0007] In another aspect of the present invention, a data generation method for generating transmission data to be sent to a generation AI includes: generating N1 partial documents by dividing a document to be referenced by the generation AI and storing the document in a database; obtaining a summary of each of the N1 partial documents as a primary summary and storing the N1 primary summaries in association with the N1 partial documents in the database; generating N2 primary summary groups from the N1 primary summaries, each primary summary group containing fewer than N1 primary summaries; obtaining a summary of each of the N2 primary summary groups as a secondary summary and storing the N2 secondary summaries in association with the N2 primary summary groups in the database; obtaining a summary vector generated from each of the N2 secondary summaries and storing the N2 summary vectors in association with the N2 secondary summaries in a vector database included in the database; accepting input of a directive statement and obtaining the directive vector generated from the directive statement; The method includes: performing a similarity search of the vector database to extract from the database primary summaries included in a primary summary group corresponding to a summary vector having a high similarity to the instruction vector; and generating the transmission data including the instruction text and the extracted primary summaries.

[0008] 5 is a block diagram of a system configuration for realizing a data generation method. FIG. 5 is a flowchart of a data generation method. FIG. 5 is a flowchart showing details of S200 in the first embodiment. FIG. 5 is a flowchart showing details of S400 in the first embodiment. FIG. 5 is a flowchart showing details of S500 in the first embodiment. FIG. 5 shows segmented texts obtained by finely dividing document 3. FIG. 5 shows segmented texts obtained by finely dividing document 3. FIG. 5 shows segmented texts obtained by finely dividing document 3. FIG. 5 shows a primary summary summarizing each partial document formed by grouping several segmented texts. FIG. 5 shows a primary summary summarizing each partial document formed by grouping several segmented texts. FIG. 5 shows a primary summary summarizing each partial document formed by grouping several segmented texts. FIG. 5 shows an example of a keyword list created in S410a of FIG. 4. FIG. 5 shows an example of data handled in S420a and S430a of FIG. 4. FIG. 5 shows an example of partial documents extracted in S510a of FIG. 5 shows an example of data generated by generation AI5. FIG. 5 is a flowchart showing details of S200 in the second embodiment. FIG. 5 is a flowchart showing details of S400 in the second embodiment. FIG. 5 is a flowchart showing details of S500 in the second embodiment. 10 shows primary summary groups formed by grouping several primary summaries, and secondary summaries that summarize each primary summary group.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each embodiment described below shows an example of the present invention and does not limit the content of the present invention. Furthermore, not all of the configurations and operations described in each embodiment are necessarily essential as the configurations and operations of the present invention. Note that the same components are given the same reference numerals, and redundant explanations will be omitted.

[0010] <1. Overview of the embodiment> In the embodiment of the present invention, an example is shown in which a generative AI generates a response to the instruction, "What is the lesson to be learned from The Spider's Thread?" "The Spider's Thread" is the title of a short story by Ryunosuke Akutagawa (1892-1927). It is assumed that the content of "The Spider's Thread" has not been learned by the generative AI, and that the entire text data of "The Spider's Thread" exceeds the upper limit of the number of tokens of the generative AI. Therefore, it is necessary to reduce the transmission data sent to the generative AI to below the upper limit of the number of tokens.

[0011] In the first embodiment, the data generating device 1 that generates transmission data acquires the entire text data of "The Spider's Thread" as a material (FIG. 2, S100), and divides the material to generate a plurality of partial materials (FIG. 3, S210a). The data generating device 1 acquires a plurality of primary summaries that respectively summarize the plurality of partial materials (S220a), acquires summary vectors generated from each of the plurality of primary summaries (S250a), and stores them in the vector database 21.

[0012] The data generating device 1 acquires a directive vector Vp generated from the directive statement "What is the lesson to be learned from the spider's thread?" (FIG. 2, S400). The data generating device 1 executes a similarity search of the vector database 21 using the directive vector Vp, and extracts partial data corresponding to summary vectors that have a high degree of similarity to the directive vector Vp (FIG. 5, S510a). Transmission data (S520a) including the directive statement and the extracted partial data is transmitted to the generation AI.

[0013] In the second embodiment, the data generating device 1 generates N2 primary summary groups (less than N1) from the N1 primary summaries (FIG. 16, S230b). The data generating device 1 obtains N2 secondary summaries that respectively summarize the N2 primary summary groups (S240b), obtains summary vectors generated from each of the N2 secondary summaries (S250b), and stores them in the vector database 21.

[0014] The data generation device 1 performs a similarity search of the vector database 21 using the instruction vector Vp and extracts primary summaries corresponding to the summary vectors that have a high similarity to the instruction vector Vp (FIG. 18, S510b). Transmission data (S520b) including the instruction statement and the extracted primary summaries is sent to the generation AI.

[0015] In the third embodiment, the hierarchical structure of the summaries is deeper.

[0016] 1 is a block diagram of a system configuration for realizing a data generation method. The configuration shown in FIG. 1 is common to the first to third embodiments. The data generation method is realized by a data generation device 1 and a database 2.

[0017] The data generating device 1 is a computer device including a memory 11 that stores a computer program for implementing a data generating method, and a CPU 12 that executes the computer program.

[0018] Database 2 stores text data such as a plurality of partial documents and summaries, and summary vectors. The summary vectors are stored in a vector database 21 included in database 2. Vector database 21 stores data in the form of high-dimensional vectors and is configured to enable high-speed similarity searches. Text data such as partial documents and summaries may be stored in a relational database (not shown) included in database 2. The partial documents and summaries stored in the relational database and the summary vectors stored in vector database 21 are associated with each other by a common ID.

[0019] The data generating device 1 is configured to be able to acquire the document 3 and the instruction 4 from an external device. The document 3 is acquired, for example, from a storage unit or a server connected via a network. The instruction 4 is acquired, for example, from an input device such as a keyboard. The document 3 is not limited to being a novel. For example, if the instruction 4 requests information regarding product handling, the document 3 may be a product instruction manual. If the instruction 4 requests legal information, the document 3 may be a collection of laws and regulations or a collection of court decisions. The document 3 may also be audio data, and the data generating device 1 may convert the audio data into text data and store it in the database 2.

[0020] The data generation device 1 is configured to send transmission data including instructions to the generation AI 5 and receive data generated by the generation AI 5 from the generation AI 5. The generation AI 5 is a generation AI that uses an encoder-decoder model such as a Transformer.

[0021] 2 is a flowchart of a data generation method, showing the processing procedure of the data generation device 1. The processing procedure shown in FIG. 2 is common to the first to third embodiments.

[0022] In S100, the data generating device 1 acquires the material 3. Here, the text data of the entire text of "The Spider's Thread" is acquired from the digital library "Aozora Bunko" on the Internet.

[0023] In S200, the data generating device 1 divides the document 3 to obtain a plurality of summary vectors, and stores the summary vectors in the vector database 21. Details of S200 in the first embodiment will be described later with reference to Fig. 3, details of S200 in the second embodiment will be described later with reference to Fig. 16, and details of S200 in the third embodiment will be described later with reference to Fig. 20.

[0024] In S300, the data generating device 1 receives input of instruction statement 4 and acquires instruction statement 4. In S400, the data generating device 1 acquires an instruction vector generated from instruction statement 4. Details of S400 in the first embodiment will be described later with reference to FIG. 4, and details of S400 in the second and third embodiments will be described later with reference to FIG. 17.

[0025] In S500, the data generating device 1 generates transmission data based on the instruction vector and the summary vector. Details of S500 in the first embodiment will be described later with reference to Fig. 5, and details of S500 in the second and third embodiments will be described later with reference to Fig. 18.

[0026] In S600, the data generating device 1 transmits the transmission data to the generation AI 5. After S600, the data generating device 1 ends the processing of this flowchart.

[0027] 4. First Embodiment 4-1. Processing for Slicing Document 3 and Obtaining Summary Vectors Fig. 3 is a flowchart showing the details of S200 in the first embodiment. The processing shown in Fig. 3 corresponds to a subroutine of S200 in Fig. 2.

[0028] In S210a, the data generation device 1 divides the material 3 to be referenced by the generation AI 5 to generate multiple partial materials. The partial materials are obtained by dividing the material 3 into pieces with a number of characters that is less than the upper limit of the number of tokens of the generation AI 5. N1 partial materials are generated. The processing of S210a will be described below with reference to FIGS. 6 to 11.

[0029] 6 to 8 show segmented texts obtained by cutting up document 3 into small pieces. The segmented texts are stored in database 2. The segmented texts are generated, for example, according to the following rules (a) and (b): (a) Cut at the line break point. (b) If the distance from the first cutting position to the next second cutting position exceeds 200 characters, cut at the end of the sentence just before the distance from the first cutting position reaches 200 characters.

[0030] An ID is assigned to each of the multiple text segments. The ID may be any ID that can uniquely identify the text segments and the summaries described below. In the first to third embodiments, the ID includes a summary level (0, 1, 2, ...) and a serial number within each summary level. The text segments shown in Figures 6 to 8 are segments of the original text before summarization, and have a summary level of 0. The text segment with ID "0-1" contains only a line break code and is therefore not shown. The text segment with ID "0-2" is titled "The Spider's Thread," the text segment with ID "0-3" is written by "Akutagawa Ryunosuke," and the main text of the novel begins with the text segment with ID "0-10." If N0 is a natural number and the number of text segments is N0, then ID "0-N0" shown in Figure 8 corresponds to the last text segment.

[0031] 9 to 11 show primary summaries that are generated for each partial document, which is made up of several grouped text segments. The primary summaries are stored in database 2. The partial documents are generated, for example, according to the following rules (a) and (b): (a) The first text segment is taken to be one partial document, up to the last text segment that reaches 500 characters. (b) The next text segment is taken to be the first text segment of the next partial document.

[0032] The partial materials are stored in the database 2 as a list of IDs of the divided texts, as shown in the "Parent" column in Figures 9 to 11. IDs of divided texts that consist only of line break codes (such as "0-1") are excluded. A parent is the text that is the source of the summary. Note that divided texts do not have parents, so the "Parent" columns in Figures 6 to 8 are left blank.

[0033] Although the case where several divided texts (FIGS. 6 to 8) are grouped to generate partial materials ("parent" in FIG. 9 to FIG. 11) has been described, the present invention is not limited to this. Partial materials may also be generated by cutting directly from Material 3 at the end of the sentence just before it reaches 500 characters from the beginning.

[0034] 3 again, in S220a, the data generation device 1 obtains a summary of each of the multiple partial documents as a primary summary. The summaries can be created by, for example, a generation AI. If the data generation device 1 sends to the generation AI an instruction such as "Please summarize in 200 characters or less" along with one partial document, the primary summary can be obtained from the generation AI. The generation AI that generates the summary does not have to be the same generation AI as the generation AI 5 that responds to the instruction, "What lesson can be learned from the spider's thread?"

[0035] 9 to 11, the primary summaries are stored in the database 2 in association with the partial documents. If N1 is a natural number and the number of partial documents is N1, then N1 primary summaries will be created. The partial document and primary summary corresponding to ID "1-N1" shown in FIG. 11 will be the last partial document and primary summary.

[0036] In S250a of FIG. 3, the data generating device 1 requests the generation AI to generate a summary vector for each of multiple summaries (primary summaries) and obtains the summary vector from the generation AI. The summary vector is a high-dimensional vector that conforms to the specifications of the generation AI that generates the summary vector. The number of dimensions is, for example, 1536. The generation AI that generates the summary vector does not have to be the same generation AI as the generation AI 5 that answers the instruction, "What lesson can we learn from the spider's thread?"

[0037] 9 to 11, N1 summary vectors V1-1 to V1-N1 are created. The summary vectors are associated with the primary summaries and stored in a vector database 21 included in the database 2.

[0038] After S250a in FIG. 3, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0039] 4-2. Process for Obtaining a Pointing Vector Generated from Directive Statement 4 Fig. 4 is a flowchart showing the details of S400 in the first embodiment. The process shown in Fig. 4 corresponds to a subroutine of S400 in Fig. 2.

[0040] In S410a, the data generating device 1 acquires keywords from each of the multiple primary summaries to create a keyword list. Keywords can be generated by, for example, a generation AI. The data generating device 1 can acquire keywords for a summary from the generation AI by sending the summary along with an instruction such as "Please output keywords" to the generation AI. The generation AI that generates the keywords does not have to be the same as the generation AI 5 that responds to the instruction "What is the lesson to be learned from the spider's thread?" Alternatively, the keywords can be generated by the data generating device 1 or another external device. For example, words included in the summaries can be extracted, and the extracted words whose TFIDF (term frequency-inverse document frequency) exceeds a threshold can be used as keywords.

[0041] As shown in Figures 9 to 11, keywords are stored in the database 2 in association with the primary summaries. As in Figures 9 to 11, "vectors" and "keywords" are also shown in Figures 6 to 8. The vectors and keywords of the segmented texts shown in Figures 6 to 8 are not used in the first to third embodiments, but since performing the same processing for each hierarchy of summary levels (0, 1, 2, ...) is desirable from the standpoint of automation and can be convenient for checking whether the data generating device 1 is operating normally, the vectors and keywords of the segmented texts are created.

[0042] FIG. 12 shows an example of the keyword list created in S410a of FIG. 4. The keyword list is created from the keywords created for the N1 primary summaries shown in FIGS. 9 to 11. The generation AI can select the keywords created for the N1 primary summaries to include in the keyword list. The data generation device 1 can obtain the keyword list from the generation AI by sending an instruction such as "Please narrow down the keywords to 30" along with the keywords created for the N1 primary summaries to the generation AI. The generation AI that generates the keyword list does not have to be the same as the generation AI 5 that responds to the instruction, "What is the lesson to be learned from the spider's thread?" Alternatively, the keyword list can be created by the data generation device 1 or another external device.

[0043] Referring again to FIG. 4, in S420a, the data generating device 1 obtains a paraphrase of the instruction sentence 4 using keywords included in the keyword list, and in S430a, obtains an instruction vector generated from the paraphrase.

[0044] 13 shows an example of data handled in S420a and S430a of Fig. 4. The instruction "What lesson can be learned from the spider's thread?" is instruction 4 acquired in S300 of Fig. 2.

[0045] The correction instruction "What is the lesson to be learned from the novel 'The Spider's Thread'?" can be generated by, for example, a generation AI. If the data generation device 1 sends an instruction such as "Please revise 'What is the lesson to be learned from The Spider's Thread?' so that the question is clearer," to the generation AI, the correction instruction can be obtained from the generation AI. In the example shown in FIG. 13, the correction instruction makes it clear that "The Spider's Thread" does not refer to a physical thread but to the title of a novel. The generation AI that generates the correction instruction does not have to be the same generation AI as the generation AI 5 that answers the instruction "What is the lesson to be learned from The Spider's Thread?"

[0046] The paraphrases are, for example, the keywords "novel, spider's thread, lesson" extracted from the correction instruction. The paraphrases can be generated by, for example, a generation AI. If the data generation device 1 sends an instruction such as "Use the keyword list to output the keywords for 'What is the lesson to be learned from the novel "Spider's Thread"?'" along with the keyword list to the generation AI, the paraphrases can be obtained from the generation AI. The generation AI that generates the paraphrases does not have to be the same generation AI as the generation AI 5 that responds to the instruction "What is the lesson to be learned from Spider's Thread?" Alternatively, when extracting keywords as paraphrases, the data generation device 1 or another external device can extract them.

[0047] Figure 13 also shows the instruction vector Vp obtained in S430a of Figure 4. The instruction vector Vp is a high-dimensional vector generated using the same rules as the summary vector and has the same number of dimensions as the summary vector. The higher the similarity or relevance between the paraphrase wording and the summary (primary summary), the greater the similarity between the instruction vector Vp and the summary vector. The generation AI that generates the instruction vector Vp is the same generation AI that generated the summary vector, but it does not have to be the same generation AI as the generation AI5 that answers the instruction statement, "What lesson can we learn from the spider's thread?"

[0048] After S430a in FIG. 4, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0049] 4-3. Process for generating transmission data based on instruction vectors and summary vectors Fig. 5 is a flowchart showing the details of S500 in the first embodiment. The process shown in Fig. 5 corresponds to a subroutine of S500 in Fig. 2.

[0050] In S510a, the data generating device 1 executes a similarity search of the vector database 21, and extracts from the database 2 partial materials corresponding to summary vectors that have a high similarity to the instruction vector Vp.

[0051] FIG. 14 shows an example of the partial material extracted in S510a of FIG. 5. The summary vectors with a high degree of similarity to the instruction vector Vp were summary vector V1-10 corresponding to ID "1-10" shown in FIG. 10 and summary vectors V1-11 and V1-12 corresponding to IDs "1-11" and "1-12" shown in FIG. 11. As shown in FIG. 10, the partial material corresponding to ID "1-10" consists of segmented texts with IDs "0-30," "0-31," and "0-32." As shown in FIG. 11, the partial material corresponding to ID "1-11" consists of segmented texts with IDs "0-33" and "0-34." The partial material corresponding to ID "1-12" consists of segmented texts with IDs "0-38" and "0-39." Reading these segmented texts from database 2 yields the results shown in FIG. 14.

[0052] 5 again, in S520a, the data generating device 1 generates transmission data including instruction statement 4 and the extracted partial material. Instruction statement 4 may be rephrased as follows: "The following sentence is a part of the novel "The Spider's Thread." What is the lesson to be learned from this novel "The Spider's Thread"?"

[0053] The transmitted data may further include summaries (primary summaries) corresponding to the extracted partial materials. In the above example, the summaries corresponding to the extracted partial materials are the summaries corresponding to IDs "1-10," "1-11," and "1-12." Although the extracted partial materials alone may not be able to connect the context, the generation AI has the ability to rephrase and summarize each partial material in natural language, and thus including summaries in the transmitted data may result in more appropriate results.

[0054] The transmission data may further include partial materials before and after the extracted partial material. For example, if a partial material corresponding to ID "1-10" is extracted, the transmission data may include partial materials corresponding to ID "1-9" and ID "1-11." Including the partial materials before and after the extracted partial material may clarify the context and result in more appropriate results.

[0055] After S520a in FIG. 5, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0056] 15 shows an example of data generated by the generation AI 5. Even though the generation AI 5 cannot refer to the entire text of Document 3, the generation AI 5 can provide an appropriate answer because the transmitted data includes partial materials related to the instruction sentence 4.

[0057] 4-5. Effects of the First Embodiment (1) According to the first embodiment, the data generation method for generating transmission data to be sent to the generation AI 5 includes the following steps: From the material 3 to be referenced by the generation AI 5, a plurality of partial materials are generated by dividing the material 3 and stored in the database 2 (FIG. 3, S210a). A summary of each of the partial materials is obtained, and the plurality of summaries are associated with the plurality of partial materials and stored in the database 2 (S220a). Summary vectors V1-1 to V1-N1 generated from each of the plurality of summaries are obtained, and the plurality of summary vectors V1-1 to V1-N1 are associated with the plurality of summaries and stored in the vector database 21 included in the database 2 (S250a). Input of a directive statement 4 is accepted, and a pointer vector Vp generated from the directive statement 4 is obtained (FIG. 4). A similarity search is performed on the vector database 21 to extract from the database 2 partial materials corresponding to the summary vectors V1-10, V1-11, and V1-12 that have a high degree of similarity to the pointer vector Vp (FIG. 5, S510a). Transmission data including instruction statement 4 and the extracted partial material is generated (S520a).

[0058] According to this, by extracting partial materials highly relevant to the instruction statement 4, it is possible to identify the parts of the document 3 that should be referenced, and appropriate results can be obtained from the generation AI 5 while reducing the amount of data sent to the generation AI 5. However, even if partial materials are vectorized, the resulting vector may not be weighted to the main points. By vectorizing the summary and evaluating the similarity between the instruction vector Vp and the summary vector, it is possible to extract partial materials highly relevant to the instruction statement 4.

[0059] Furthermore, because partial materials highly related to instruction sentence 4 are extracted based on the similarity of vectors, it is not necessary for material 3 and instruction sentence 4 to be in the same language. For example, "The Spider's Thread" is a material in Japanese, but instruction sentence 4 may be in English.

[0060] Furthermore, according to the first embodiment, the information contained in Document 3 is not learned by the generation AI 5, so even if Document 3 contains confidential information, in order to maintain confidentiality, it is sufficient to limit the users of Database 2 to specific people, and there is no need to limit the users of Generation AI 5.

[0061] Furthermore, according to the first embodiment, even if there is no upper limit on the number of tokens, there is an effect of reducing the transmission data sent to the generation AI 5. If, for example, there were 100 users seeking information about "The Spider's Thread," and all 100 of them were to individually send the entire text data of "The Spider's Thread" to the generation AI 5, the load on the generation AI 5 would be enormous. By storing partial materials, summaries, and summary vectors in the database 2 in advance and sending the partial materials extracted in accordance with the instruction statement 4 to the generation AI 5, the energy consumption by the generation AI 5 can be reduced.

[0062] (2) According to the first embodiment, the transmission data further includes summaries (primary summaries) corresponding to the extracted partial materials (S520a).

[0063] This allows the generation AI 5 to obtain more appropriate results by using not only the partial material but also the corresponding primary summary as the transmitted data.

[0064] (3) According to the first embodiment, keywords from each of multiple summaries (primary summaries) are obtained to create a keyword list (FIG. 4, S410a), a paraphrase of instruction sentence 4 is obtained using the keywords included in the keyword list (S420a), and an instruction vector Vp generated from the paraphrase is obtained (S430a).

[0065] According to this, the instruction vector Vp is created using a keyword list obtained from multiple primary summaries, so that partial materials can be extracted by highlighting the similarity with the primary summaries that are closely related to the instruction sentence 4 among the multiple primary summaries, and transmission data suitable for transmission to the generation AI 5 can be generated.

[0066] 5. Second Embodiment In the first embodiment, one-stage summarization is performed, whereas in the second embodiment, two-stage summarization is performed as follows.

[0067] 5-1. Processing for Dividing Document 3 and Obtaining Summary Vectors Fig. 16 is a flowchart showing details of S200 in the second embodiment. The processing shown in Fig. 16 corresponds to a subroutine of S200 in Fig. 2.

[0068] In S210b, the data generation device 1 divides the material 3 to be referenced by the generation AI 5 to generate N1 partial materials. The process of S210b is similar to the process of S210a in the first embodiment described with reference to Figure 3. A modification of S210b in the third embodiment will be described later with reference to Figure 20.

[0069] In S220b, the data generating device 1 obtains a summary of each of the N1 partial documents as a primary summary. The process of S220b is similar to the process of S220a in the first embodiment described with reference to Fig. 3. The primary summaries are stored in the database 2 in association with the partial documents.

[0070] In S230b, the data generating device 1 generates N2 primary summary groups, which is less than N1, from the N1 primary summaries, where N2 is a natural number.

[0071] 19 shows a primary summary group formed by grouping several primary summaries. Primary summary groups are generated, for example, according to the following rules (a) and (b): (a) The first primary summary to the last primary summary just before it reaches 500 characters constitute one primary summary group. (b) The next primary summary is made the first primary summary of the next primary summary group.

[0072] The primary summary groups are stored in the database 2 as a list of IDs of the primary summaries, as shown in the "Parent" column of Figure 19. Each primary summary group consists of one or more primary summaries, and the number of primary summaries included in each primary summary group is less than N1.

[0073] 16 again, in S240b, the data generating device 1 obtains a summary of each of the N2 primary summary groups as a secondary summary. The summaries can be generated by, for example, a generation AI.

[0074] Secondary summaries are also shown in Figure 19 (details omitted). The secondary summaries are stored in database 2 in association with the primary summary groups. N2 secondary summaries are created. The primary summary group and secondary summary corresponding to ID "2-N2" shown in Figure 19 are the final primary summary group and secondary summary.

[0075] 16, the data generating device 1 obtains a summary vector generated from each of the N2 secondary summaries. The generation AI that generates the summary vector is the same as the generation AI that generates the instruction vector Vp, but it does not have to be the same as the generation AI5 that answers the instruction sentence, "What lesson can be learned from the spider's thread?"

[0076] 19, N2 summary vectors V2-1 to V2-N2 are created. The summary vectors are associated with the secondary summaries and stored in the vector database 21 included in the database 2.

[0077] After S250b in FIG. 16, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0078] 5-2. Process for Obtaining a Directive Vector Generated from Directive Statement 4 Figure 17 is a flowchart showing the details of S400 in the second embodiment. The process shown in Figure 17 corresponds to a subroutine of S400 in Figure 2. The process shown in Figure 17 differs from the process of the first embodiment described with reference to Figure 4 in that keywords for each of the N2 secondary summaries are obtained in S410b. As shown in Figure 19, the keywords are stored in database 2 in association with the secondary summaries. Other points are the same as in the first embodiment.

[0079] 5-3. Process for generating transmission data based on instruction vector and summary vector Fig. 18 is a flowchart showing the details of S500 in the second embodiment. The process shown in Fig. 18 corresponds to a subroutine of S500 in Fig. 2.

[0080] In S510b, the data generating device 1 performs a similarity search of the vector database 21, determines a primary summary group corresponding to the summary vectors that have a high similarity to the instruction vector Vp, and extracts the primary summaries included in the primary summary group from the database 2. The summary vectors that have a high similarity to the instruction vector Vp are, for example, the summary vectors corresponding to IDs "2-5" and "2-6," which correspond to the primary summaries IDs "1-9," "1-10," "1-11," and "1-12."

[0081] In S520b, the data generating device 1 generates transmission data including the instruction statement 4 and the extracted primary summary.

[0082] The transmitted data may further include secondary summaries corresponding to the primary summary groups containing the extracted primary summaries, such as the secondary summaries corresponding to IDs "2-5" and "2-6" in the above example.

[0083] After S520b in FIG. 18, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0084] 5-4. Effects of the Second Embodiment (4) According to the second embodiment, the data generation method for generating transmission data to be sent to the generation AI 5 includes the following: From the material 3 to be referenced by the generation AI 5, N1 partial materials are generated by dividing the material 3 and stored in the database 2 (FIG. 16, S210b). A summary of each of the N1 partial materials is obtained as a primary summary, and the N1 primary summaries are associated with the N1 partial materials and stored in the database 2 (S220b). From the N1 primary summaries, N2 primary summary groups are generated, each containing fewer than N1 primary summaries (S230b). A summary of each of the N2 primary summary groups is obtained as a secondary summary, and the N2 secondary summaries are associated with the N2 primary summary groups and stored in the database 2 (S240b). Summary vectors V2-1 to V2-N2 generated from each of the N2 secondary summaries are obtained, and the N2 summary vectors V2-1 to V2-N2 are associated with the N2 secondary summaries and stored in vector database 21 included in database 2 (S250b). Input of directive statement 4 is accepted, and a reference vector Vp generated from directive statement 4 is obtained (FIG. 17). A similarity search is performed on vector database 21, and primary summaries included in the primary summary group corresponding to summary vectors V2-5 and V2-6 that have a high degree of similarity to reference vector Vp are extracted from database 2 (FIG. 18, S510b). Transmission data including directive statement 4 and the extracted primary summaries is generated (S520b).

[0085] According to this, by extracting a primary summary that is highly relevant to the instruction statement 4, it is possible to obtain an appropriate result from the generation AI 5 while reducing the amount of data transmitted to the generation AI 5. Also, by vectorizing the secondary summary and evaluating the similarity between the instruction vector Vp and the summary vector, it is possible to extract a primary summary that is highly relevant to the instruction statement 4. Furthermore, by performing two-stage summarization, it is possible to further reduce the amount of data transmitted compared to one-stage summarization.

[0086] (5) According to the second embodiment, the transmitted data further includes a secondary summary corresponding to the primary summary group including the extracted primary summary (S520b).

[0087] This allows for more appropriate results to be obtained from the generated AI5 by using not only the primary summary but also the corresponding secondary summary in the transmitted data.

[0088] (6) According to the second embodiment, keywords from each of the N2 secondary summaries are obtained to create a keyword list (FIG. 17, S410b), a paraphrase of instruction sentence 4 is obtained using the keywords included in the keyword list (S420a), and an instruction vector Vp is generated from the paraphrase (S430a).

[0089] According to this, an instruction vector Vp is generated using a keyword list obtained from N2 secondary summaries, so that a primary summary can be extracted by highlighting the similarity with the secondary summary that is closely related to instruction sentence 4 among the N2 secondary summaries, and transmission data suitable for transmission to generation AI 5 can be generated.

[0090] 6. Third Embodiment While one-level summarization is performed in the first embodiment and two-level summarization is performed in the second embodiment, three-level summarization or more is performed in the third embodiment by generating N1 partial materials through one or more levels of summarization. For example, N1 partial materials are generated from N0 zero-level summaries obtained by summarizing N0 segmented texts, respectively.

[0091] <6-1. Process of Dividing Document 3 to Generate N1 Partial Documents> Fig. 20 is a flowchart showing details of S210b in the third embodiment. The process shown in Fig. 20 corresponds to a subroutine of S210b in Fig. 16.

[0092] In S211c, the data generating device 1 divides the material 3 to generate N0 pieces of divided text.

[0093] In S212c, the data generating device 1 obtains summaries of the N0 divided texts as zero-order summaries.

[0094] In S213c, the data generating device 1 generates N1 zero-order summary groups, which are less than N0, from the N0 zero-order summaries as N1 partial materials.

[0095] After S213c, the data generating device 1 ends the processing of this flowchart and returns to the processing shown in FIG.

[0096] Here, we have explained the case where N1 partial documents are generated from the divided text obtained by dividing Document 3 through one stage of summarization, but N1 partial documents may also be generated through two or more stages of summarization.

[0097] 6-2. Other Processing In other respects, the third embodiment is similar to the second embodiment. As in the second embodiment, N2 primary summary groups (S230b) are generated from N1 primary summaries (FIG. 16, S220b) obtained by summarizing each of the N1 partial materials, and summary vectors (S250b) are generated from N2 secondary summaries (S240b) obtained by summarizing each of the N2 primary summary groups. The primary summaries included in the primary summary groups corresponding to the summary vectors with the highest similarity to the indicator vector Vp are then extracted (FIG. 18, S510b).

[0098] <6-3. Effects of the Third Embodiment> According to the third embodiment, by summarizing in three or more stages, it is possible to further reduce the amount of transmitted data compared to two stages of summarization. Even if the document 3 is enormous, appropriate results can be obtained by transmitting data up to the upper limit of the number of tokens to the generation AI 5. Furthermore, if the overall summary of document 3 falls below the upper limit of the number of tokens due to multi-stage summarization, a macroscopic perspective can be provided to the generation AI 5 by including the overall summary of document 3 in the transmitted data.

[0099] 7. Modifications In the first to third embodiments, the instruction 4 requests information about the document 3, but the present invention is not limited to this. If the document 3 includes text data of a statement made by a real person or a fictional character, the instruction 4 may request the generation of text that resembles a statement made by a specific person or character. The instruction 4 may also specify a scene in which such a statement is made. Based on the similarity between the instruction vector and the summary vector, the data generation device 1 extracts text data of a statement made by the person or character in a specific scene as partial material and transmits it to the generation AI 5. Based on the transmitted data, the generation AI 5 can generate text incorporating the person's or character's unique catchphrases, dialect, etc.

Claims

1. A data generation method for generating transmission data to be sent to a generation AI, comprising: generating a plurality of partial materials by dividing a material to be referenced by the generation AI from the material, and storing the plurality of partial materials in a database; obtaining a summary of each of the plurality of partial materials, and storing the plurality of summaries in the database in association with the plurality of partial materials; obtaining a summary vector generated from each of the plurality of summaries, and storing the plurality of summary vectors in a vector database included in the database in association with the plurality of summaries; accepting input of an instruction statement and obtaining the instruction vector generated from the instruction statement; performing a similarity search of the vector database to extract from the database partial materials corresponding to summary vectors that have a high similarity to the instruction vector; and generating the transmission data including the instruction statement and the extracted partial materials.

2. A data generating method according to claim 1, wherein the transmission data further includes a summary corresponding to the extracted partial material.

3. A data generation method as described in claim 1, comprising the steps of: obtaining keywords for each of the plurality of summaries to create a keyword list; obtaining a paraphrase phrase that rephrases the instruction sentence using the keywords included in the keyword list; and obtaining the instruction vector generated from the paraphrase phrase.

4. A data generation method for generating transmission data to be sent to a generation AI, comprising the steps of: generating N1 partial documents by dividing a document to be referenced by the generation AI, and storing the N1 partial documents in a database; obtaining a summary of each of the N1 partial documents as a primary summary, and storing the N1 primary summaries in the database in association with the N1 partial documents; generating N2 primary summary groups from the N1 primary summaries, each primary summary group containing fewer than N1 primary summaries; obtaining a summary of each of the N2 primary summary groups as a secondary summary, and storing the N2 secondary summaries in association with the N2 primary summary groups in the database; obtaining a summary vector generated from each of the N2 secondary summaries, and storing the N2 summary vectors in a vector database included in the database in association with the N2 secondary summaries; accepting input of an instruction statement and obtaining the instruction vector generated from the instruction statement; performing a similarity search of the vector database to extract from the database primary summaries included in a primary summary group corresponding to a summary vector having a high similarity to the instruction vector; and generating the transmission data including the instruction text and the extracted primary summaries.

5. A data generating method according to claim 4, wherein the transmitted data further comprises secondary summaries corresponding to the primary summary group from which the extracted primary summary is derived.

6. A data generation method as described in claim 4, comprising the steps of: obtaining keywords for each of the N2 secondary summaries to create a keyword list; obtaining a paraphrase phrase that rephrases the instruction sentence using the keywords included in the keyword list; and obtaining the instruction vector generated from the paraphrase phrase.

Images