Data processing apparatus, data processing method, and storage medium

Abstract

To provide a technique for creating prompt data for making a generation model generate an appropriate answer sentence.SOLUTION: One aspect of the present disclosure relates to a data processing device including an acquisition unit that acquires processing target data and a question sentence related to the processing target data, a prompt creation unit that determines an agent suitable for the question sentence for the processing target data and creates prompt data for causing the agent to answer the question sentence for the processing target data, and a model interface unit that transmits the prompt data to a generation model and acquires an answer sentence from the generation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a data processing device, a data processing method, and a program. [Background technology]

[0002] With the emergence of the ChatGPT (registered trademark) service, generative models are being actively utilized in various fields of business and society. In a typical use case, a user provides a question to a generative model and receives an answer to that question from the model.

[0003] The questions provided to the generative model are created as text data called prompt data. The accuracy of the answers to the questions depends on how the prompt data is created. Therefore, in order for the generative model to generate appropriate answers to the questions, it is necessary to create the prompt data appropriately. Various methods have been proposed for incorporating prompt data to enable the generative model to generate more accurate answers. For example, it is known to be useful to specify a "role" for the generative model in the prompt data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7416390 [Patent Document 2] Patent No. 7396762 Summary of the Invention [Problem to be solved by the invention]

[0005] While it is useful to assign a "role" to a generative model, there is room for further investigation into what kind of "role" should be effectively instructed to the generative model.

[0006] In view of the above problems, one object of the present disclosure is to provide a technique for creating prompt data that allows a generative model to generate appropriate answer sentences. [Means for solving the problem]

[0007] One aspect of the present disclosure relates to a data processing device having an acquisition unit that acquires data to be processed and a question regarding the data to be processed, a prompt creation unit that determines an agent suitable for the question regarding the data to be processed and creates prompt data for the agent to respond to the question regarding the data to be processed, and a model interface unit that sends the prompt data to a generation model and acquires the answer from the generation model. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a technique for creating prompt data that causes a generative model to generate appropriate answer sentences. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an interaction process according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a block diagram illustrating a hardware configuration of a data processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a functional configuration of a data processing device according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating prompt data according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating prompt data according to one embodiment of the present disclosure. [Figure 7] 7A and 7B are diagrams illustrating prompt data according to one embodiment of the present disclosure. [Figure 8]FIG. 8 is a flowchart illustrating an interaction process according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In the following embodiments, a data processing device is disclosed that utilizes generative models to answer questions about data to be processed.

[0012] [Summary of the Disclosure] In the embodiment described below, when data to be processed and a question to a generative model regarding the data to be processed are received, the generative model selects an agent suitable for answering the question and has the generative model answer the question from the viewpoint of the selected agent. For example, as shown in FIG. 1, when a video and a question regarding the contents of the video are received, the generative model determines a sewing expert and an object detection expert as agents to answer the question and generates prompts for each agent. The generative model then generates a response to the video and question from the viewpoint of the determined agent.

[0013] In the example shown in FIG. 1, a video of a person knitting and a question about what the knitter is doing are given. When the generative model receives the video and the question, it determines an agent suitable for generating an answer to the question about the video. In this example, the generative model determines an expert in sewing and an expert in object detection, and generates a prompt for each agent to answer the question. The generative model then answers the question from the perspective and position of each agent.

[0014] In this way, an agent suited to the data to be processed and the question can be made to generate an answer to the question in a generative model, enabling a more appropriate answer to be given from an expert's perspective or standpoint.

[0015] [Information Processing Systems] In an embodiment described below, as shown in Fig. 2, an information processing system 10 includes a user device 20, a generative model 30, and a data processing device 100. The user device 20 uses the generative model 30 via the data processing device 100 to perform interactive processing with the generative model 30.

[0016] The user device 20 is, for example, an information processing device such as a personal computer, a smartphone, or a tablet, and is operated by a user who uses the generative model 30.

[0017] The generative model 30 is, for example, any machine learning model that realizes interactive processing with a user, such as ChatGPT, etc. Typically, such a generative model 30 is managed and operated by a model management server (not shown) or the like.

[0018] The data processing device 100 is an information processing device, such as a local server or a cloud server, that is communicatively connected to the user device 20 and a model management server, and mediates the user's use of the generative model 30. Note that the functions and processing of the data processing device 100 described below may be realized by, for example, an application installed on the user device 20.

[0019] In the interactive processing according to this embodiment, in step S1, the user device 20 receives from the user data to be processed and a question regarding the generative model 30 regarding the data to be processed, and then transmits the received data to be processed and the question to the data processing device 100. The data to be processed may be, for example, a video, and the question may be a question regarding what a person or the like is doing in the video.

[0020] In step S2, the data processing device 100 creates prompt data (primary) requesting the determination of an agent suitable for answering a question regarding the data to be processed, and transmits the created prompt data to the generative model 30.

[0021] In step S3, the generative model 30 receives prompt data and determines an agent. At this time, multiple agents may be determined, and a coordinator agent may be further provided to consolidate the answer sentences from the determined multiple agents. Such a coordinator agent may aggregate the answer sentences from the multiple agents, or may be configured to have each agent refer to answer sentences from other agents so that the answer sentences from the multiple agents converge into one. Furthermore, the generative model 30 generates prompt data that instructs each agent to generate an answer sentence to the question sentence, and prompt data that instructs the coordinator agent to consolidate the answer sentences from the agents.

[0022] In step S4, the data processing device 100 transmits a response request using the agent. Specifically, the data processing device 100 transmits prompt data (secondary) for each agent generated in step S3 to the generative model 30.

[0023] In step S5, the generative model 30 transmits to the data processing device 100 the answer sentences from the agents and / or the answer sentences compiled by the coordinator agent.

[0024] In step S6, the data processing device 100 transmits the answer sentence acquired from the generative model 30 to the user device 20.

[0025] In the illustrated embodiment, steps S2 to S5 are executed sequentially between the data processing device 100 and the generative model 30, but the dialogue processing according to the present disclosure is not limited to this, and, for example, steps S2 to S5 may be executed collectively within the generative model 30. That is, the data processing device 100 may request the generative model 30 to select and use an agent to generate an answer to the question, and the answer generated within the generative model 30 may be returned to the data processing device 100.

[0026] Here, the data processing device 100 is realized by a computing device such as a server, a personal computer, a smartphone, or a tablet, and may have a hardware configuration such as that shown in Fig. 3. That is, the data processing device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0027] Programs or instructions for realizing various functions and processes described below in the data processing device 100 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory.

[0028] When the storage medium is set in the drive device 101, the program or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the program or instructions do not necessarily have to be installed from the storage medium, but may be downloaded from any external device via a network or the like.

[0029] The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

[0030] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or instruction is activated, reads and stores the program, instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0031] The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processes of the data processing device 100 described below in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc. stored in the memory device 103.

[0032] The user interface (UI) device 105 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a touch panel, and realizes an interface between a user and the data processing device 100. For example, a user operates the data processing device 100 by operating a GUI (Graphical User Interface) displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0033] The communication device 106 is realized by various communication circuits that execute communication processes with external devices, the Internet, a communication network such as a LAN (Local Area Network), and the like.

[0034] However, the above-described hardware configuration is merely an example, and the data processing device 100 according to the present disclosure may be realized by any other suitable hardware configuration.

[0035] [Data processing device] Next, a data processing device 100 according to an embodiment of the present disclosure will be described. Fig. 4 is a block diagram showing a functional configuration of the data processing device 100 according to an embodiment of the present disclosure. As shown in Fig. 4, the data processing device 100 includes an acquisition unit 110, a prompt creation unit 120, and a model interface unit 130. For example, one or more functional units of the acquisition unit 110, the prompt creation unit 120, and the model interface unit 130 may be realized by one or more processors 104 executing one or more programs or instructions stored in the memory device 103.

[0036] The acquiring unit 110 acquires data to be processed and a question related to the data to be processed. For example, the data to be processed may be a video, and the question may be related to the content of the video. Specifically, as shown in FIG. 1, the video may be a video of C knitting, and the question may be to identify one of options 0 to 4 related to the behavior of C.

[0037] For example, option 0 might be "Adjusting C is intended to produce a more complex pattern," option 1 might be "Adjusting C is intended to create a more vibrant and colorful pattern," option 2 might be "Adjusting C is intended to maintain tension in the yarn and prevent finger pain," option 3 might be "Adjusting C is intended to create a more textured pattern," and option 4 might be "Fine adjustments to C are intended to create a more symmetrical pattern."

[0038] The data to be processed is not limited to video, but may be other types of data such as audio, images, text, or a mixture of two or more types of data. The question is not limited to a selection from options, but may request the generation of an answer in any format by the generative model 30.

[0039] The prompt creation unit 120 determines an agent appropriate for the question about the data to be processed, and creates prompt data (primary) for the agent to respond to the question about the data to be processed. For example, if the video is of C knitting and the question is to identify one of options 0 to 4 regarding C's behavior, the prompt creation unit 120 can create prompt data such as that shown in FIG.

[0040] 5, prompt data portion P1 instructs the generative model 30 to identify what C is doing in the video from among options 0 to 4. The prompt creation unit 120 can write prompt data portion P1 based on the received processing target data and question text, and as in this example, when a video of C knitting and options related to C's actions in the video are given, the prompt creation unit 120 may create prompt data portion P1 as shown in the figure.

[0041] Furthermore, prompt data portion P2 instructs the generative model 30 to select multiple (two) agents with different specialties to respond to the question and generate prompt data for each agent to respond to the question. Prompt data portion P3 presents examples of prompts to be responded to by each agent. For interactive processing in which the target data is a video and the question is related to the video, the prompt creation unit 120 may use prompt data portions P2 and P3 as a standard, or may modify prompt data portions P2 and P3 depending on the video and question. For example, the number of agents may be changed to another number depending on the received video and question, or the number of agents may not be specified in the prompt data and may be determined by the generative model 30.

[0042] Finally, prompt data section P4 shows the prompt to be provided to each agent, where prompt generator 120 specifies the agent's name and the output format of the prompt for that agent.

[0043] Note that the term "agent" here may be replaced with "expert" or the like, and may refer to an entity that is knowledgeable in a particular field or area. By specifying an agent in this manner, a "role" can be assigned to the generative model 30 so that it functions as the agent. The generative model 30 may use servers and databases for video analysis, speech recognition, and text analysis to determine the agent that will generate a response to the data to be processed and the question. Each agent may then use servers and databases for video analysis, speech recognition, and text analysis to respond to prompts for that agent.

[0044] After creating prompt data for the processing target and the question in this way, the prompt creating unit 120 sends the created prompt data to the model interface unit 130.

[0045] The model interface unit 130 transmits prompt data to the generative model 30 and acquires an answer sentence from the generative model 30. That is, upon acquiring prompt data from the prompt creation unit 120, the model interface unit 130 transmits the acquired prompt data and video to the model operation server of the generative model 30, and receives an answer sentence from the generative model 30 via the prompt data from the model operation server. The video may be embedded in the prompt data, or may be transmitted separately to the generative model 30, or may be stored on a server or the like accessible from the generative model 30.

[0046] For example, when the model interface unit 130 provides the prompt data shown in Fig. 5 to the generative model 30, it can obtain from the generative model 30 answer sentences indicating determined agents and prompts for each agent as shown in Fig. 6. In the example shown in Fig. 6, the generative model 30 selects two agents, a "text tile agent" and an "ergonomics specialist," in accordance with the prompt data portion P4 in Fig. 5, and creates prompts for each agent, such as "You are a textile artist. Watch the video from the perspective of a professional textile artist and answer the following questions based on your expertise. Think about it step by step," and "You are an ergonomics expert. Watch the video from the perspective of an ergonomics expert and answer the following questions based on your expertise. Think about it step by step."

[0047] 6 is obtained from the generative model 30, the prompt creation unit 120 creates prompt data (secondary) as shown in Figures 7A and 7B, and provides the created prompt data to the model interface unit 130. Here, the prompt data shown in Figure 7A is from the "text tile agent," and the prompt data shown in Figure 7B is from the "ergonomics specialist."

[0048] When the prompt data for each agent is acquired from the prompt creation unit 120, the model interface unit 130 transmits the acquired prompt data to the generative model 30 and acquires the answer sentence from the generative model 30. At this time, the video may be embedded in the prompt data, or may be transmitted separately to the generative model 30, or may be stored on a server or the like accessible from the generative model 30.

[0049] The generative model 30 generates an answer sentence for each agent's prompt data and returns it to the data processing device 100. For example, the model interface unit 130 may combine an answer sentence for the prompt data of the "textile agent" obtained from the generative model 30 with an answer sentence for the prompt data of the "ergonomics specialist" (for example, by concatenating the two answer sentences), and transmit the combined answer sentence to the user device 20.

[0050] However, the response sentences from each agent are not necessarily the same. For this reason, the prompt creation unit 120 may set a coordinator agent that aggregates the response sentences from multiple agents, and aggregates the response sentences. The coordinator agent acquires response sentences from multiple agents, and if it determines that the acquired response sentences are significantly different, it aggregates these response sentences.

[0051] Specifically, the prompt creation unit 120 may create a prompt that instructs the coordinator agent to aggregate the response sentences of multiple agents, and provide the prompt for the coordinator agent along with the prompt for each agent to the model interface unit 130. Based on the prompt, the generative model 30 passes the response sentences from each agent to the coordinator agent, and causes the coordinator agent to generate an response sentence that aggregates these response sentences.

[0052] As a specific method of aggregation, for example, the coordinator agent may aggregate the response sentences from multiple agents by majority vote. For example, if three agents are selected, two of them select option 0, and the remaining agent selects option 2, the coordinator agent may respond to user device 20 with option 0 by majority vote.

[0053] Furthermore, when the coordinator agent determines that it is not possible to aggregate the answer statements from multiple agents, for example, when the majority rule of the answer statements does not hold, it may provide the agents with the answer histories from the multiple agents and have the multiple agents re-reply to the question statement for the data to be processed. For example, in order to aggregate the answer statements from each agent, the coordinator agent may instruct each agent to refer to the answer statements of other agents and to generate an answer statement taking into account the answer statements it referenced. In this way, by referring to the answer statements of other agents, each agent will generate an answer statement that takes these into account, which may enable the aggregation of answer statements.

[0054] According to the above-described data processing device 100, an agent suited to the data to be processed and the question can be caused to generate a reply to the question, and a more appropriate reply can be generated from an expert's point of view.

[0055] [Interactive processing] Next, an interaction process with the generative model 30 according to an embodiment of the present disclosure will be described. Fig. 8 is a flowchart showing an interaction process with the generative model 30 according to an embodiment of the present disclosure. The interaction process is performed by the above-described data processing device 100, and more specifically, may be realized by one or more processors 104 of the data processing device 100 executing one or more programs or instructions stored in one or more memory devices 103.

[0056] 8, in step S101, the data processing device 100 acquires data to be processed and a question. For example, the data processing device 100 may acquire a video and a question related to the video from the user device 20.

[0057] In step S102, the data processing device 100 creates prompt data (primary) for determining an agent suitable for the question regarding the data to be processed. Here, an "agent" is an expert in a certain field or area, and upon receiving the prompt data, the generative model 30 determines one or more agents suitable for answering the question from the received video and question.

[0058] In step S103, the data processing device 100 creates prompt data (secondary) for causing the agents to respond to the question sentence for the data to be processed. That is, once the agents are determined in step S102, the data processing device 100 creates prompt data for causing each agent to respond to the question sentence.

[0059] In step S104, the data processing device 100 transmits prompt data to the generative model 30 and acquires an answer sentence from the generative model 30. For example, when multiple agents are selected, the data processing device 100 acquires an answer sentence from each agent, synthesizes the acquired answer sentences, and provides the synthesized answer sentences to the user device 20.

[0060] Here, in order to aggregate response sentences from multiple agents, the data processing device 100 may set up a coordinator agent, aggregate the response sentences obtained from the multiple agents in the coordinator agent, and provide the aggregated response sentences to the user device 20.

[0061] According to the above-described dialogue processing, an answer to a question can be generated by an agent that is suited to the data to be processed and the question, and a more appropriate answer can be generated from an expert's perspective.

[0062] (Appendix 1) an acquisition unit that acquires processing target data and a question sentence related to the processing target data; a prompt creation unit that determines an agent suitable for the question regarding the processing target data, and creates prompt data for causing the agent to respond to the question regarding the processing target data; a model interface unit that transmits the prompt data to a generative model and acquires an answer sentence from the generative model; A data processing device comprising: (Appendix 2) 2. The data processing device according to claim 1, wherein the prompt creation unit creates prompt data for determining the agent. (Appendix 3) 2. The data processing device according to claim 1, wherein the prompt creation unit determines a plurality of agents suitable for the question sentence for the processing target data. (Appendix 4) 4. The data processing device according to claim 3, wherein the prompt creating unit sets a coordinator agent that aggregates answer sentences from the plurality of agents. (Appendix 5) 5. The data processing device according to claim 4, wherein the coordinator agent aggregates response statements from the plurality of agents by majority vote. (Appendix 6) The data processing device described in Appendix 4, wherein, when the coordinator agent determines that it is not possible to aggregate the response statements from the multiple agents, it provides the multiple agents with a response history from the multiple agents and has the multiple agents re-reply to the question statement for the data to be processed. (Appendix 7) acquiring data to be processed and a question regarding the data to be processed; determining an agent suitable for the question regarding the processing target data, and creating prompt data for causing the agent to respond to the question regarding the processing target data; sending the prompt data to a generative model and obtaining an answer sentence from the generative model; A data processing method performed by one or more computers. (Appendix 8) acquiring data to be processed and a question regarding the data to be processed; determining an agent suitable for the question regarding the processing target data, and creating prompt data for causing the agent to respond to the question regarding the processing target data; sending the prompt data to a generative model and obtaining an answer sentence from the generative model; A program that causes one or more computers to execute the following.

[0063] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims. [Industrial Applicability]

[0064] The present disclosure is useful for interaction processing using generative models. [Explanation of symbols]

[0065] 10 Information Processing Systems 20 User Device 30 Generative Models 100 Data processing device 110 Acquisition Department 120 Prompt Creation Department 130 Model Interface Section

Claims

1. an acquisition unit that acquires processing target data and a question sentence related to the processing target data; a prompt creation unit that determines an agent suitable for the question regarding the processing target data, and creates prompt data for causing the agent to respond to the question regarding the processing target data; a model interface unit that transmits the prompt data to a generative model and acquires an answer sentence from the generative model; A data processing device comprising:

2. The data processing device according to claim 1 , wherein the prompt creating unit creates prompt data for determining the agent.

3. The data processing device according to claim 1 , wherein the prompt creating unit determines a plurality of agents suitable for the question sentence for the processing target data.

4. 4. The data processing device according to claim 3, wherein the prompt creating section sets a coordinator agent that aggregates the responses of the plurality of agents.

5. 5. The data processing device according to claim 4, wherein said coordinator agent aggregates reply statements from said plurality of agents by majority vote.

6. 5. The data processing device according to claim 4, wherein when the coordinator agent determines that it is not possible to aggregate the response statements from the plurality of agents, it provides the plurality of agents with a response history from the plurality of agents and causes the plurality of agents to re-reply to the question statement for the data to be processed.

7. acquiring data to be processed and a question regarding the data to be processed; determining an agent suitable for the question regarding the processing target data, and creating prompt data for causing the agent to respond to the question regarding the processing target data; sending the prompt data to a generative model and obtaining an answer sentence from the generative model; A data processing method executed by one or more computers.

8. acquiring data to be processed and a question regarding the data to be processed; determining an agent suitable for the question regarding the processing target data, and creating prompt data for causing the agent to respond to the question regarding the processing target data; sending the prompt data to a generative model and obtaining an answer sentence from the generative model; A program that causes one or more computers to execute the following.

Images