Data processing device, data processing method, and program

Abstract

An aspect of the present disclosure relates to a data processing device comprising: an acquisition unit that acquires processing target data and a question text relating to the processing target data; a prompt creation unit that determines an expert agent suitable for the question text and creates prompt data for the expert agent to provide an answer to the question text; and a model interface unit that transmits the prompt data to a generative model and acquires an answer text from the generative model.

Description

Data processing device, data processing method and program

[0001] The present disclosure relates to a data processing device, a data processing method, and a program.

[0002] With the advent of the ChatGPT (registered trademark) service, generative models are being actively utilized in various fields of business and society. In a typical use, a user provides a question to a generative model and receives an answer to the question from the generative model.

[0003] The questions provided to the generative model are created as text data called prompt data. The accuracy of the answers to the questions depends on how the prompt data is created. Therefore, in order for the generative model to generate appropriate answers to the questions, it is necessary to create the prompt data appropriately. Various methods have been proposed for introducing prompt data to enable the generative model to generate more accurate answers. For example, it is known to be useful to specify a "role" for the generative model in the prompt data.

[0004] Patent No. 7416390 Patent No. 7396762

[0005] While it is useful to assign a "role" to a generative model, there is room for further study as to what kind of "role" should be effectively instructed to the generative model.

[0006] In view of the above problems, one object of the present disclosure is to provide a technique for creating prompt data that allows a generative model to generate appropriate answer sentences.

[0007] One aspect of the present disclosure relates to a data processing device having an acquisition unit that acquires data to be processed and a question related to the data to be processed, a prompt creation unit that determines an expert agent suitable for the question and creates prompt data for the expert agent to answer the question, and a model interface unit that transmits the prompt data to a generative model and acquires the answer from the generative model.

[0008] According to the present disclosure, it is possible to provide a technique for creating prompt data that causes a generative model to generate appropriate answer sentences.

[0009] FIG. 1 is a schematic diagram illustrating an example of a dialogue processing according to an embodiment of the present disclosure; FIG. 2 is a schematic diagram illustrating an information processing system according to an embodiment of the present disclosure; FIG. 3 is a block diagram illustrating a hardware configuration of a data processing device according to an embodiment of the present disclosure; FIG. 4 is a block diagram illustrating a functional configuration of a data processing device according to an embodiment of the present disclosure; FIG. 5 is a diagram illustrating an example of prompt data according to an embodiment of the present disclosure; FIG. 6 is a diagram illustrating an example of prompt data according to an embodiment of the present disclosure; FIG. 7 is a diagram illustrating an example of prompt data according to an embodiment of the present disclosure; FIG. 8 is a diagram illustrating an example of a flowchart of dialogue processing according to an embodiment of the present disclosure;

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In the following embodiments, a data processing device is disclosed that utilizes generative models to answer questions about data to be processed.

[0012] [Outline of the Present Disclosure] FIG. 1 is an example of a schematic diagram illustrating dialogue processing according to an embodiment of the present disclosure. In the embodiment described below, a data processing device 100 receives data to be processed and a question for a generative model regarding the data to be processed. The data processing device then causes the generative model 30 to select an expert agent suitable for answering the question and have the generative model 30 answer the question from the perspective of the selected expert agent. For example, as shown in FIG. 1 , upon receiving a video (3-minute video clip) and a question (Question & 5 options) regarding the contents of the video, the generative model 30 determines a sewing expert agent and an object detection expert agent as the expert agents to answer the question and generates prompts for each agent. The generative model 30 then generates a response to the question regarding the video from the perspective of the determined expert agent. The expert agent and coordinator agent are functions of the generative model 30 operating as an expert and a coordinator.

[0013] In the example shown in FIG. 1 , a video of a person knitting and a question about what the knitter is doing are provided. Upon receiving the video and the question, the generative model determines an expert agent suitable for generating an answer to the question about the video. Here, the generative model determines a sewing expert agent and an object detection expert agent, and generates a prompt for each expert agent to answer the question. For example, the generative model generates a prompt for the sewing expert agent saying, "You are a sewing expert. Please watch the video and answer the following questions." For the object detection expert agent, the generative model generates a prompt saying, "You are an object detection expert. Please recognize the objects in the video and answer the questions." The generative model then answers the question from the perspective and position of each expert agent. Each expert agent answers the question using multiple servers. For example, each expert agent uses a video analysis server, a speech recognition server, and a text analysis server. For example, in the case of a video of someone knitting, a crafts expert agent might respond, "I think the person filming is using knitting needles to knit a scarf," while an object detection expert agent might respond, "It's highly likely that the tool this woman is using is a knitting needle."

[0014] In this way, the data processing device 100 can cause an expert agent that is suited to the data to be processed and the question to generate an answer to the question, thereby enabling the expert agent to provide a more appropriate answer from its own perspective and standpoint.

[0015] [Information Processing System] Fig. 2 is an example of a schematic diagram showing an information processing system according to an embodiment of the present disclosure. In an embodiment described below, as shown in Fig. 2, the information processing system 10 includes a user device 20, a generative model 30, and a data processing device 100. The user device 20 uses the generative model 30 via the data processing device 100 to perform interactive processing with the generative model 30.

[0016] The user device 20 is, for example, an information processing device such as a personal computer, a smartphone, or a tablet, and is operated by a user who uses the generative model 30.

[0017] The generative model 30 is, for example, any machine learning model that realizes interactive processing with a user, such as ChatGPT, etc. Typically, such a generative model 30 is managed and operated by a model management server (not shown) or the like.

[0018] The data processing device 100 is an information processing device, such as a local server or a cloud server, that is communicatively connected to the user device 20 and a model management server, and mediates the user's use of the generative model 30. Note that the functions and processes of the data processing device 100 described below may be realized by, for example, an application installed on the user device 20.

[0019] In the dialogue processing according to this embodiment, in step S1, the user device 20 receives from the user data to be processed and a question regarding the generative model 30 related to the data to be processed, and then transmits the received data to be processed and the question to the data processing device 100. The data to be processed may be, for example, a video, and the question may be a question about what a person or the like is doing in the video.

[0020] In step S2, the data processing device 100 creates first prompt data requesting the determination of an expert agent suitable for answering a question regarding the data to be processed, and sends the created first prompt data to the generative model 30.

[0021] In step S3, the generative model 30 receives the first prompt data, determines an expert agent to answer the question, and sends the answer to the data processing device 100. FIG. 9 shows an example of a configuration diagram of the generative model 30. FIG. 9 shows an example of a configuration diagram of a system in which the generative model 30 receives prompt data from the data processing device 100 and answers the question. The generative model is connected to a video analysis server 210, a speech recognition server 220, and a sentence analysis server 230. The servers to which the generative model 30 is connected are not limited to those shown in FIG. 9. The generative model 30 may be connected to other servers.

[0022] For example, the generative model 30 analyzes a video received from the data processing device 100 using the video analysis server 210, the voice recognition server 220, and the text analysis server 230. For example, the generative model 30 analyzes video footage (moving image) using the video analysis server 210. Furthermore, the generative model 30 converts the audio of the video into text using the voice recognition server 220, and then analyzes the content of the text using the text analysis server 230. Based on these analyses, the generative model 30 can analyze the video received from the data processing device 100 as a "video of a person knitting." The generative model 30 determines an expert agent based on the analysis results.

[0023] The generative model 30 may receive first prompt data including candidate expert agents and select multiple expert agents to answer the query from the candidate expert agents. The data processing device 100 may determine candidate expert agents based on history. For example, the data processing device 100 may determine candidate expert agents based on the history of expert agents identified for similar videos or similar queries.

[0024] At this time, multiple expert agents may be determined, and a coordinator agent may be further provided to consolidate the answer statements from the multiple determined expert agents. The number of expert agents to be determined may be set in advance. If a large number of expert agents are determined, the cost of generating prompts to instruct the expert agents increases. Furthermore, if there are too many expert agents, expert agents with weak relationships may participate, resulting in a decrease in answer accuracy. For example, the number of expert agents to be determined is 2 to 5. The coordinator agent may consolidate the answer statements from the multiple expert agents, or may be configured to refer an answer statement from one expert agent to other expert agents, receive answer statements from the other expert agents, and converge the answer statements from the multiple expert agents into one.

[0025] The data processing device 100 generates second prompt data instructing each expert agent to generate a response sentence to the question sentence. When a coordinator agent is provided, the data processing device 100 may generate third prompt data instructing the coordinator agent to compile the response sentences from the expert agents.

[0026] In step S4, the data processing device 100 transmits a request for an answer to the question to each expert agent. For example, the data processing device 100 transmits second prompt data for each expert agent generated in step S3 to each expert agent. The data processing device 100 also receives answer sentences from each expert agent and transmits third prompt data to the coordinator agent instructing the expert agent to compile the answer sentences.

[0027] In step S5, the generative model 30 transmits the answer sentence compiled by the coordinator agent to the data processing device 100.

[0028] The video analysis may use the results of a previous analysis, or may involve analyzing a new video. For example, each expert agent may answer the question using the analysis results of the video analyzed when answering in step S3. For example, each expert agent may answer the question by analyzing the video again. For example, an expert agent may answer the question by receiving the analysis results of another expert agent.

[0029] For example, regarding "a video of a person knitting," a handicraft expert agent will respond, "I think the person filming is knitting a scarf using knitting needles," and an object detection expert agent will respond, "It is highly likely that the tools this woman is using are knitting needles for handicrafts." For example, the coordinator agent will transmit to the data processing device 100, "After considering the opinions of the experts, it is highly likely that this woman is knitting a scarf using knitting needles."

[0030] 10-12 show three detailed sequence diagrams of steps S4 and S5 when a coordinator agent is provided. Figures 10-12 show examples of querying two expert agents. When a coordinator agent is not provided, the data processing device 100 may receive response sentences from each expert agent and forward them to the user device 20.

[0031] 10, the data processing device 100 transmits second prompt data and video to each expert agent instructing them to send a reply sentence to the coordinator agent. Each expert agent transmits a reply sentence to the coordinator agent. The data processing device 100 transmits third prompt data to the coordinator agent instructing it to receive and summarize the responses from each expert agent. The coordinator agent receives and summarizes the reply sentences from each expert agent and transmits the summarized reply sentences to the data processing device 100.

[0032] 11, the data processing device 100 transmits second prompt data and video to each expert agent. Each expert agent transmits a response sentence to the data processing device 100. The data processing device transmits third prompt data including the response sentences received from each expert agent to the coordinator agent. The coordinator agent compiles the received response sentences and transmits them to the data processing device 100.

[0033] 12 shows the data processing device 100 transmitting second and third prompt data and video to the coordinator agent. The second and third prompt data transmitted by the data processing device to the coordinator agent include both an instruction to request a response from each expert agent and an instruction to the coordinator agent to compile the response from each expert agent. The coordinator agent transmits the second prompt data and video to each expert agent. Each expert agent transmits a response to the coordinator agent. The coordinator agent compiles the response received from each expert agent and transmits it to the data processing device 100.

[0034] The answers sent by each expert agent are not limited to answer sentences (in text format). For example, each expert agent may answer in a video style. For example, each expert agent may answer by including a video segment important for understanding the answer sentence along with the answer sentence.

[0035] In step S6 , the data processing device 100 transmits the answer sentences (summarized answer sentences) acquired from the generative model 30 to the user device 20 .

[0036] 2, steps S2 to S5 are executed sequentially between the data processing device 100 and the generative model 30, but the dialogue processing according to the present disclosure is not limited to this, and for example, steps S2 to S5 may be executed collectively within the generative model 30. That is, the data processing device 100 may request the generative model 30 to generate an answer to the question by selecting and using an expert agent, and the answer generated by selecting and using the expert agent within the generative model 30 may be returned to the data processing device 100.

[0037] 3 is a block diagram illustrating an example of the hardware configuration of a data processing device 100. Here, the data processing device 100 is realized by a computing device such as a server, a personal computer, a smartphone, or a tablet. The data processing device 100 includes a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0038] Programs or instructions for realizing various functions and processes described below in the data processing device 100 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory.

[0039] When the storage medium is set in the drive device 101, the program or instructions are installed from the storage medium to the storage device 102 or the memory device 103 via the drive device 101. However, the program or instructions do not necessarily have to be installed from the storage medium, but may be downloaded from any external device via a network or the like.

[0040] The storage device 102 is realized by a hard disk drive (HDD) or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

[0041] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or instruction is activated, it reads and stores the program, instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0042] The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processes of the data processing device 100 described below in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc. stored in the memory device 103.

[0043] The user interface device 105 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a touch panel, and realizes an interface between a user and the data processing device 100. For example, the user operates the data processing device 100 by operating a GUI (Graphical User Interface) displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0044] The communication device 106 is realized by various communication circuits that execute communication processes with external devices, the Internet, a communication network such as a LAN (Local Area Network), and the like.

[0045] However, the above-described hardware configuration is merely an example, and the data processing device 100 according to the present disclosure may be realized by any other suitable hardware configuration.

[0046] [Data Processing Device] Next, a data processing device 100 according to an embodiment of the present disclosure will be described. Fig. 4 is an example of a block diagram showing the functional configuration of the data processing device 100 according to an embodiment of the present disclosure. As shown in Fig. 4, the data processing device 100 includes an acquisition unit 110, a prompt creation unit 120, and a model interface unit 130. For example, one or more functional units of the acquisition unit 110, the prompt creation unit 120, and the model interface unit 130 may be realized by one or more processors 104 executing one or more programs or instructions stored in the memory device 103.

[0047] The acquisition unit 110 acquires data to be processed and a question related to the data to be processed. For example, the data to be processed may be a video, and the question may be related to the content of the video. Specifically, as shown in FIG. 1 , the video may be a video of a person C knitting, and the question may be to identify one of five options 0 to 4 related to the work of person C.

[0048] For example, option 0 might be "Person C's adjustments (work) are intended to create a more complex pattern," option 1 might be "Person C's work is intended to create a more vibrant and colorful pattern," option 2 might be "Person C's work is intended to maintain tension in the thread and prevent finger pain," option 3 might be "Person C's work is intended to create a more textured pattern," and option 4 might be "Person C's precision work is intended to create a more symmetrical pattern."

[0049] The data to be processed is not limited to video (moving image), but may be data in other formats such as audio, images (still images), text, or a mixture of two or more types of data. Furthermore, the question is not limited to a selection from options, but may request the generation of an answer in any format by the generative model 30. For example, the question may be a free-form question about the purpose of the work performed by person C in the video.

[0050] The prompt creation unit 120 determines an agent appropriate for the question regarding the data to be processed, and creates first prompt data for having the agent respond to the question regarding the data to be processed. For example, if the video is of a person C knitting, and the question is to identify one of options 0 to 4 regarding C's work, the prompt creation unit 120 may create first prompt data as shown in FIG. 5.

[0051] The example first prompt data shown in FIG. 5 instructs the generative model 30 to determine two expert agents and to create prompts for the determined expert agents.

[0052] The prompt data portion P1 is an instruction to the generative model 30 to identify what person C is doing in the video from among options 0 to 4. The prompt creation unit 120 can create the prompt data portion P1 based on the processing target data and the question received from the user device 20. For example, as in this example, when the prompt creation unit 120 receives from the user device 20 a video of person C knitting and options related to the work that person C is doing in the video, the prompt creation unit 120 may create the prompt data portion P1 shown in FIG.

[0053] Furthermore, the prompt creation unit 120 creates a prompt data portion P2 that instructs the generative model 30 to select multiple expert agents (two in FIG. 5 ) with different specialties to answer the question and generate prompt data for each expert agent to answer the question. The prompt data portion P2 may be a standard phrase. The prompt creation unit 120 may create the prompt data portion P2 by modifying the standard phrase. The number of expert agents identified by the prompt creation unit 120 is arbitrary. For example, the prompt creation unit 120 may change the number of expert agents depending on the received video and question. Alternatively, the prompt creation unit 120 may not specify the number of expert agents and may instruct the generative model 30 to determine the number. The prompt creation unit 120 may create a prompt data portion P2 that includes multiple candidate expert agents and select a specific expert agent from the candidate expert agents. The prompt creation unit 120 may also create a prompt data portion P2 that includes the history of the identified expert agents for similar videos or similar questions.

[0054] The prompt generator 120 also generates a prompt data portion P3 that shows examples of prompt data to be responded to by each expert agent. When the target data is a video and the question is a dialogue related to the video, the prompt generator 120 may generate the prompt data portion P3 as a fixed phrase, or may modify the fixed phrase in the prompt data portion P3 depending on the video and the question. The prompt data portion P3 may include multiple examples.

[0055] The prompt generator 120 may generate a prompt data portion P4 that indicates the format of the prompt data for each expert agent as an output format. For example, the prompt generator 120 generates a prompt data portion P4 that indicates the name of an expert agent and the output format of the prompt for that expert agent.

[0056] Note that the term "expert agent" here may be replaced with "expert" or the like, and may refer to an entity that is knowledgeable in a particular field or area. By specifying an expert agent in this manner, the generative model 30 can be given a "role" to function as the expert agent. The generative model 30 may determine the expert agent that generates the target data and an answer to the question using the video analysis server 210, the speech recognition server 220, the text analysis server 230, a database, etc. Then, each expert agent may respond to the prompt data using the video analysis server 210, the speech recognition server 220, the text analysis server 230, a database, etc.

[0057] After creating the first prompt data for the processing target and the question sentence in this way, the prompt creator 120 sends the created first prompt data to the model interface unit 130 .

[0058] The model interface unit 130 transmits first prompt data to the generative model 30 and acquires an answer sentence from the generative model 30. That is, upon acquiring first prompt data from the prompt creation unit 120, the model interface unit 130 transmits the acquired first prompt data and video to the generative model 30 and receives an answer sentence from the generative model 30 via the first prompt data. The video may be embedded in the first prompt data, may be transmitted separately to the generative model 30, or may be stored on a server or the like accessible from the generative model 30.

[0059] For example, when the model interface unit 130 provides the example of the first prompt data shown in FIG. 5 to the generative model 30, the model interface unit 130 obtains answer sentences shown as determined expert agents and prompt data for each expert agent from the generative model 30 as shown in FIG. 6.

[0060] FIG. 6 shows an example in which the generative model 30 identifies two expert agents, a “textile artist” and an “ergonomics specialist,” and creates prompt data for each expert agent in accordance with prompt data portion P4 in FIG. 5, such as “You are a textile artist. Watch the video from the perspective of a professional textile artist and answer the following questions based on your expertise. Please think about it step by step,” and “You are an ergonomics expert. Watch the video from the perspective of an ergonomics expert and answer the following questions based on your expertise. Please think about it step by step.”

[0061] The prompt generator 120 generates second prompt data based on the response sentence to the first prompt data. For example, when the response sentence shown in FIG. 6 is obtained from the generative model 30, the prompt generator 120 generates examples of second prompt data shown in FIGS. 7A and 7B and provides the generated second prompt data and video to the model interface unit 130. The video may be embedded in the second prompt data, may be separately transmitted to the generative model 30, or may be stored on a server or the like accessible from the generative model 30. Here, the second prompt data shown in FIG. 7A is an example requesting a response as a "textile artist," and the second prompt data shown in FIG. 7B is an example requesting a response as an "ergonomics specialist."

[0062] When the model interface unit 130 acquires the second prompt data from the prompt creation unit 120, it transmits the acquired second prompt data to the generative model 30 and acquires an answer sentence from the generative model 30. Each expert agent transmits the generated answer to the model interface unit 130.

[0063] 10, the model interface unit 130 may transmit second prompt data to each expert agent for transmitting the generated answer to the coordinator agent, and may transmit third prompt data to the coordinator agent for transmitting an instruction to aggregate the answers from the expert agents. Each expert agent transmits the generated answer to the coordinator agent.

[0064] In response to the second prompt data, each expert agent generates a response sentence and transmits it to the data processing device 100 or the coordinator agent. The model interface unit 130 may aggregate the response sentences from each expert agent. The model interface unit 130 may receive the response sentences aggregated by the coordinator agent. For example, the model interface unit 130 may combine a response sentence to the prompt data for "textile artist" and a response sentence to the prompt data for "ergonomics specialist" obtained from the generative model 30 (e.g., by concatenating the two response sentences), and transmit the combined response sentence to the user device 20.

[0065] The model interface unit 130 may transmit to the coordinator agent third prompt data including an instruction to aggregate the response sentences from the plurality of expert agents and the response sentences from the plurality of expert agents.

[0066] The response sentences from each expert agent are not necessarily the same. For this reason, the prompt creation unit 120 may set a coordinator agent that aggregates the response sentences from multiple expert agents and instruct the aggregation of the response sentences. The coordinator agent acquires response sentences from multiple expert agents, and if it determines that the acquired response sentences are significantly different, it aggregates these response sentences.

[0067] For example, the prompt generator 120 may generate third prompt data instructing the coordinator agent to aggregate response sentences from a plurality of expert agents, and provide the third prompt data to the model interface unit 130 together with second prompt data for each expert agent. The coordinator agent transmits second prompt data to each expert agent based on the third prompt data and the second prompt data. Upon receiving the response sentences from each expert agent, the coordinator agent aggregates these response sentences based on the third prompt data. The response sentences from each expert agent may be embedded in the third prompt data, or may be transmitted to each expert agent, or may be stored on a server or the like accessible to each expert agent.

[0068] As a specific method of aggregation, for example, the coordinator agent may aggregate answer statements from multiple expert agents by majority vote. For example, if three expert agents are selected, two of the expert agents select option 0, and the remaining expert agent selects option 2, the coordinator agent may respond to the user device 20 with option 0 by majority vote.

[0069] Furthermore, when the coordinator agent determines that it is not possible to aggregate the answer sentences from each expert agent, for example, when the majority rule of the answer sentences does not hold, it may provide each expert agent with a history of answer sentences from all expert agents and have each expert agent re-answer the question sentence for the data to be processed. For example, in order to aggregate the answer sentences from each expert agent, the coordinator agent may instruct each expert agent to refer to the answer sentences from other (or all) expert agents and generate an answer sentence taking the referred answer sentences into consideration. In this way, by referring to the answer sentences from other (or all) expert agents, each expert agent generates an answer sentence taking these into consideration, thereby enabling the aggregation of answer sentences.

[0070] 13-14 show examples of sequence diagrams in which the coordinator agent makes each expert agent resubmit a response.

[0071] When the coordinator agent (a data processing device in the case of FIG. 11 ) receives answer sentences from each expert agent but determines that the answer sentences cannot be aggregated, it requests each expert agent to submit a new answer. The coordinator agent may transmit the answer sentences from each expert and a request for a new answer to each expert agent. The answer sentences from each expert may be embedded in the request for a new answer, or may be transmitted to each expert agent, or may be stored on a server or the like accessible from each expert agent.

[0072] 13 shows an example of a sequence diagram in which the coordinator agent sends a request for a re-reply to each expert agent individually. The coordinator agent sends a request for a re-reply to each expert agent, and each expert agent sends a re-reply statement to the coordinator agent.

[0073] 14 shows an example of a sequence diagram in which a coordinator agent sends a request for a re-reply to each expert agent in turn. The coordinator agent sends a request for a re-reply to expert agent 1 (e.g., a textile artist), and expert agent 1 adds its own re-reply sentence along with the request for a re-reply and sends it to another expert agent, expert agent 2 (e.g., an ergonomic specialist). Expert agent 2 adds its own re-reply sentence to the re-reply sentence of expert agent 1 received from expert agent 1 and sends it to the coordinator agent.

[0074] If the coordinator agent is unable to aggregate the reply sentences even after receiving the reply sentences, the coordinator agent may repeat the request for a reply up to a predetermined number of times.

[0075] According to the above-described data processing device 100, an expert agent suited to the data to be processed and the question can be made to generate an answer to the question, thereby generating a more appropriate answer from an expert's perspective.

[0076] [Interaction Processing] Next, an interaction processing with the generative model 30 according to an embodiment of the present disclosure will be described. Fig. 8 is an example of a flowchart of an interaction processing with the generative model 30 according to an embodiment of the present disclosure. The interaction processing is performed by the above-described data processing device 100, and more specifically, may be realized by one or more processors 104 of the data processing device 100 executing one or more programs or instructions stored in one or more memory devices 103.

[0077] 8, in step S101, the data processing device 100 acquires data to be processed and a question. For example, the data processing device 100 may acquire a video and a question related to the video from the user device 20.

[0078] In step S102, the data processing device 100 creates first prompt data for determining an expert agent suitable for the question regarding the data to be processed. An "expert agent" is an expert in a certain field or area. Upon receiving the first prompt data, the generative model 30 determines one or more expert agents suitable for answering the question from the video and question included in the first prompt data, and provides the answer to the data processing device 100.

[0079] In step S103, the data processing device 100 creates second prompt data for causing the expert agents to answer the question about the data to be processed. That is, once the expert agents are determined in step S102, the data processing device 100 creates second prompt data for causing each expert agent to answer the question.

[0080] In step S104, the data processing device 100 transmits the second prompt data to the generative model 30 and acquires an answer sentence from the generative model 30. For example, when multiple expert agents are selected, the data processing device 100 acquires an answer sentence from each expert agent and transmits it to the user device 20. The data processing device 100 may aggregate (combine) the answer sentences acquired from each expert agent and transmit the aggregated answer sentences to the user device 20.

[0081] In order to aggregate response sentences from multiple expert agents, the data processing device 100 may set up a coordinator agent, have the coordinator agent aggregate the response sentences obtained from the multiple expert agents, and provide the aggregated response sentences to the user device 20.

[0082] According to the above-described interactive processing, an expert agent that is suited to the data to be processed and the question can be made to generate an answer to the question, thereby enabling a more appropriate answer to be generated from an expert's perspective.

[0083] (Supplementary Note 1) A data processing device comprising: an acquisition unit that acquires data to be processed and a question sentence related to the data to be processed; a prompt creation unit that determines an expert agent suitable for the question sentence and creates prompt data for the expert agent to answer the question sentence; and a model interface unit that transmits the prompt data to a generative model and acquires an answer sentence from the generative model. (Supplementary Note 2) The data processing device according to Supplementary Note 1, wherein the prompt creation unit creates prompt data for determining the expert agent. (Supplementary Note 3) The data processing device according to Supplementary Note 1, wherein the prompt creation unit determines a plurality of expert agents suitable for the question sentence for the data to be processed. (Supplementary Note 4) The data processing device according to Supplementary Note 3, wherein the prompt creation unit sets a coordinator agent that aggregates the answer sentences of the plurality of expert agents. (Supplementary Note 5) The data processing device according to Supplementary Note 4, wherein the coordinator agent aggregates the answer sentences from the plurality of expert agents by majority vote. (Supplementary Note 6) The data processing device according to Supplementary Note 4, wherein, when the coordinator agent determines that it is not possible to aggregate answer sentences from the plurality of expert agents, the coordinator agent provides the plurality of expert agents with a history of answers from the plurality of expert agents and causes the plurality of expert agents to re-answer the question sentence for the data to be processed. (Supplementary Note 7) A data processing method, executed by one or more computers, comprising: acquiring data to be processed and a question sentence related to the data to be processed; determining an expert agent suitable for the question sentence and creating prompt data for causing the expert agent to answer the question sentence; sending the prompt data to a generative model and acquiring an answer sentence from the generative model.(Supplementary Note 8) A program that causes one or more computers to execute the following steps: acquiring data to be processed and a question sentence related to the data to be processed; determining an expert agent suitable for the question sentence and creating prompt data for causing the expert agent to answer the question sentence; and sending the prompt data to a generative model and acquiring an answer sentence from the generative model.

[0084] Although examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims.

[0085] The disclosures of the specification, drawings and abstract contained in Japanese Patent Application No. 2024-106840, filed on July 2, 2024, are incorporated herein by reference in their entirety.

[0086] The present disclosure is useful for interaction processing using generative models.

[0087] REFERENCE SIGNS LIST 10 Information processing system 20 User device 30 Generative model 100 Data processing device 110 Acquisition unit 120 Prompt creation unit 130 Model interface unit

Claims

1. A data processing device comprising: an acquisition unit that acquires data to be processed and a question related to the data to be processed; a prompt creation unit that creates prompt data to determine an expert agent suitable for the question and have the expert agent answer the question; and a model interface unit that transmits the prompt data to a generative model and acquires an answer from the generative model.

2. The data processing device according to claim 1, wherein said prompt creation section creates prompt data for determining said agent.

3. The data processing device according to claim 1, wherein the prompt creation unit determines a plurality of agents suited to the question sentence for the processing target data.

4. The data processing device according to claim 3, wherein the prompt creation section sets a coordinator agent that aggregates the response sentences of the plurality of agents.

5. The data processing device according to claim 4, wherein the coordinator agent aggregates the response statements from the plurality of agents by majority vote.

6. The data processing device according to claim 4, wherein, when the coordinator agent determines that it is unable to aggregate the response statements from the plurality of agents, it provides the plurality of agents with a response history from the plurality of agents and has the plurality of agents re-reply to the question statement for the data to be processed.

7. A data processing method, performed by one or more computers, comprising: acquiring data to be processed and a question related to the data to be processed; determining an expert agent suitable for the question and creating prompt data for causing the expert agent to answer the question; and sending the prompt data to a generative model and acquiring an answer from the generative model.

8. A program that causes one or more computers to execute the following steps: acquiring data to be processed and a question related to the data to be processed; determining an expert agent suitable for the question and creating prompt data for causing the expert agent to answer the question; and sending the prompt data to a generative model and acquiring an answer from the generative model.

Images