Information processing method, program, and information processing system

Abstract

An information processing method according to one embodiment is an information processing method executed by an information processing device and includes an acquisition step for inputting predetermined input information to a first trained model and a second trained model that is different from the first trained model and acquiring output information from each as a result thereof, an evaluation step for comparing and evaluating each of the output information, and an adjustment step for changing the input information in a case in which a result of the evaluation becomes a predetermined result.

Description

Information processing method, program, and information processing system

[0001] The present disclosure relates to an information processing method, a program, and an information processing system.

[0002] Japanese Patent No. 6558188 discloses a processing system that uses trained models. In this processing system, the trained model used for processing is switched based on timestamp information of the data.

[0003] Incidentally, when performing information processing using a trained model, there are cases where it is desired to change the trained model being used due to the discontinuation of the service of the trained model itself, switching to a similar trained model, etc. However, changing the trained model may change the output information, and there is room for improvement in this regard.

[0004] Taking the above facts into consideration, the present disclosure aims to enable changes to a trained model while suppressing changes in output information.

[0005] According to one embodiment of an information processing method, the information processing method is executed by an information processing device, and includes an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation becomes a specified result.

[0006] According to one embodiment of the program, an information processing device is caused to execute an information processing method including an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation becomes a specified result.

[0007] According to one embodiment, the information processing system is executed by an information processing device and includes an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation is a specified result.

[0008] According to one embodiment, it is possible to change a trained model while suppressing changes in output information.

[0009] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment; FIG. 2 is a diagram showing an example of the hardware configuration of a server according to an embodiment; FIG. 3 is a diagram showing an example of the functional configuration of a server according to an embodiment; FIG. 4 is a diagram showing an example of the flow of processing in an information processing system according to an embodiment; FIG. 5 is a diagram showing an example of the flow of evaluation steps in an information processing system according to an embodiment; and FIG. 6 is a diagram showing an example of the flow of evaluation steps in a modified example of an information processing system according to an embodiment.

[0010] An embodiment of an information processing system according to the present disclosure will be described below with reference to Figures 1 to 6. Note that identical or equivalent components and parts in each figure are denoted by the same reference numerals. Also, the dimensional proportions in the drawings have been exaggerated for the sake of explanation and may differ from the actual proportions.

[0011] (System Overview) First, an overview of the information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is an information processing system for performing at least a part of the change (switch) process of the trained model to be used. This information processing system 10 is not limited to one that performs the change process of the trained model for another information processing system that performs information processing using the trained model, but may also be one that performs the change process for the trained model that the information processing system 10 itself has.

[0012] (System Configuration) Fig. 1 is a diagram showing an example of the configuration of an information processing system 10 according to this embodiment. As shown in Fig. 1, the information processing system 10 according to this embodiment includes a server 12 as an information processing device and a user terminal 14 as an information processing device, which are communicably connected to each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof.

[0013] The user terminal 14 is an example of an information processing device that is operated by a user U to input and display various information. The user terminal 14 may be a personal computer (PC), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination thereof.

[0014] The server 12 is an example of an information processing device that acquires information input from the user terminal 14, processes the information, and outputs the results. The server 12 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server 12 will be described later.

[0015] 2 is a block diagram showing the hardware configuration of the server 12. The server 12 includes a processor 120, a memory 122, a storage 124, a communication I / F 126, an input / output I / F 128, and a drive device 134, which are communicatively connected to each other via a bus B.

[0016] The processor 120 controls each component of the server 12 and realizes the functions of the server 12 by loading various programs stored in the storage 124 into the memory 122 and executing them. The programs executed by the processor 120 include, but are not limited to, an operating system (OS) and a program 220 described below. The execution of these programs by the processor 120 realizes part of the state visualization method according to this embodiment. The processor 120 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or a combination thereof.

[0017] The memory 122 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0018] The storage 124 stores the OS, various programs described below, and various data. The storage 124 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination thereof.

[0019] The communication I / F 126 is an interface for connecting the server 12 to external devices including the user terminal 14 via the network N and controlling communication. The communication I / F 126 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0020] The input / output I / F 128 is an interface for connecting an input device 132 and an output device 130 to the server 12. The input device 132 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, an operation button, or a combination thereof. The output device 130 is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination thereof.

[0021] The drive device 134 reads and writes data from and to the disk media 136. The drive device 134 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or a combination thereof. The disk media 136 is, for example, a compact disc (CD), a digital versatile disc (DVD), a floppy disc (FD), a magneto-optical disc (MO), a Blu-ray (registered trademark) disc (BD), or a combination thereof.

[0022] In this embodiment, the program may be written to memory 122 or storage 124 during the manufacturing stage of server 12, may be provided to server 12 via network N, or may be provided to server 12 via a non-transitory computer-readable recording medium such as disk media 136.

[0023] Furthermore, the hardware configuration of the user terminal 14 is substantially the same as the hardware configuration of the server 12 described above, and therefore a detailed description thereof will be omitted.

[0024] (Functional Configuration) Next, the functional configuration of the server 12 will be described. Fig. 3 is a diagram showing an example of the functional configuration of the server 12. When executing various programs, the server 12 uses the above-mentioned hardware resources to realize various functions. The server 12 has a communication unit 20, a storage unit 22, and a control unit 24 as the functional configuration realized by the server 12. Each functional configuration is realized when the processor 120 reads and executes a program 220 stored in the memory 122 or the storage 124.

[0025] The communication unit 20 is realized by the communication I / F 126. The communication unit 20 transmits and receives information to and from the user terminal 14 via the network N. The communication unit 20 receives information input from the user terminal 14. The communication unit 20 also transmits information to the user terminal 14 and receives requests from the user U from the user terminal 14.

[0026] The memory unit 22 is realized by the memory 122 and the storage 124. The memory unit 22 stores a program 220, a first trained model 222, a second trained model 224, input information 226, output information 228, evaluation information 230, and adjustment information 232.

[0027] The first trained model 222 and the second trained model 224 are configured by at least one trained machine learning model. As an example, the first trained model 222 and the second trained model 224 are large-scale language models that are trained with large amounts of text data from internet articles, books, websites, etc. When text data called a prompt is input as input information 226, information processing is performed in accordance with the prompt, and as a result, various data including the text data are generated and output as output information 228.

[0028] The first trained model 222 and the second trained model 224 are considered to be different trained models. As a specific example, the first trained model 222 is GPT-4 from OpenAI, and the second trained model 224 is Cloude from ANTHROPIC. Note that the first trained model 222 and the second trained model 224 are not limited to this example and may be language models including other large-scale language models. Furthermore, the first trained model 222 and the second trained model 224 may each be stored on an external server or the like rather than in the memory unit 22. Note that the above-mentioned phrase "(the first trained model 222 and the second trained model 224 are considered to be) different (trained models)" includes not only different trained models but also different versions of the same trained model as "different" models.

[0029] As described above, the input information 226 is a prompt that instructs processing on the first trained model 222 and the second trained model 224, and is text data in natural language. This input information 226 is input by the user U, but is not limited to this, and may be information automatically generated or modified by the system.

[0030] The output information 228 is information indicating the results of processing performed by at least one of the first trained model 222 and the second trained model 224 based on the prompt. The output information 228 is various information including text information and image information, and is stored in association with the prompt.

[0031] The evaluation information 230 is information on the results of an evaluation of the output information 228 of at least one of the first trained model 222 and the second trained model 224 and the prompt (input information 226) linked thereto. Specifically, the evaluation information 230 is information that determines the intention of the input information 226 and evaluates how well the output information 228 matches that intention. The evaluation information 230 also includes information that compares the output information 228 of the first trained model 222 and the second trained model 224 and determines the degree of similarity in content. Details will be described later.

[0032] The adjustment information 232 is text data that has been modified from the input information 226. This adjustment information 232 is information in which information that reflects the changes has been added to the original input information 226, or information in which the original input information 226 has been rewritten based on the changes. Details will be described later.

[0033] The control unit 24 is realized by the processor 120 reading and executing the program 220 from the memory 122 (see FIG. 2 ) and working in cooperation with other hardware components. The control unit 24 includes an information acquisition unit 242, an information processing unit 244, an output evaluation unit 246, an input information adjustment unit 248, and an output unit 250.

[0034] The information acquisition unit 242 acquires information about large-scale language models specified by the user U as the first trained model 222 and the second trained model 224, as well as information for using the large-scale language models. Specifically, the information acquisition unit 242 acquires the type and name of the currently used trained model as the first trained model 222 (GPT-4, a large-scale language model, in this embodiment) and information for using the trained model, such as an ID, password, and API key. Furthermore, the information acquisition unit 242 acquires the type and name of the trained model (Cloud, a large-scale language model, in this embodiment) and information for using the trained model, such as an ID, password, and API key, for the second trained model 224, whose use the user wishes to switch (change) from the first trained model 222 currently being used. This information about the first trained model 222 and the second trained model 224 may be acquired when a trained model switching instruction is received from the user U or the information processing device, or may be acquired in advance regardless of whether a switching instruction is issued.

[0035] Furthermore, the information acquisition unit 242 acquires various pieces of information required for processing in the information processing unit 244. Specifically, it acquires input information 226 that instructs processing and output information 228 that is the processing result of at least one of the first trained model 222 and the second trained model 224 based on the input information 226. This acquired information may be information acquired from the user terminal 14, information stored in the memory unit 22, or information acquired from an external data source or the like.

[0036] The information processing unit 244 performs information processing using the first trained model 222 and the second trained model 224 based on the input information 226. This processing based on the input information 226 executes, as necessary, control to perform processing based on the same input information 226 in each of the first trained model 222 and the second trained model 224, and control to perform processing based on the input information 226 in only one of the first trained model 222 and the second trained model 224.

[0037] The output evaluation unit 246 acquires output information 228, which is the processing result of the first trained model 222 and the second trained model 224, based on the same input information 226, and evaluates the difference in the output content for the input information 226. This evaluation is performed according to a predetermined processing flow that combines multiple processes including the determination of intention, which will be described later, using the output information of the first trained model 222 as a reference. The output evaluation unit 246 performs the evaluation of the output information 228 using the first trained model 222. That is, the evaluation is performed using the first trained model 222 by prompting the first trained model 222 to evaluate the two pieces of output information 228 for the input information 226 and instructing the first trained model 222 to perform the evaluation according to a predetermined processing flow.

[0038] The output evaluator 246 also determines the intent of the input information 226. This intent refers to the thought or purpose behind the prompt, which is the input information 226. Specifically, intents are broadly categorized into five categories: "brainstorming," "confirmation," "summary," "information request," and "other." "Brainstorming" is when the user is seeking ideas or suggestions. In most cases, the user does not have a set answer. "Confirmation" provides some guidelines or instructions (e.g., a prompt that indicates a role to a large-scale language model, such as "You are an intellectual property expert") without specific input for the user to process. In this case, the system typically responds with an affirmative or acknowledgment. "Summarization" is when the user is seeking high-level, broad information. Examples include writing code in a specific programming language, defining or explaining specific technical terms, summarizing a text, or answering general knowledge questions that do not require detailed specificity. "Information request" is when the user is seeking low-level information or performing a very detailed, specific task. Examples include requesting specific information in a specific scenario, requesting that a sentence be translated or rephrased, formatting a response in a specific style, or responding in a specific table format. "Other" is an intention that does not fall under the four above-mentioned intentions. Note that in this embodiment, the intentions are broadly classified into the five above-mentioned intentions, but this is not limiting, and classification by other intentions or classification by a different number of categories may also be used. Furthermore, when the output evaluation unit 246 classifies the intention of the input information 226 as "other," it further individually and specifically determines the intention of the input information 226.

[0039] The input information adjustment unit 248 adjusts the input information 226 when the evaluation result in the output evaluation unit 246 indicates that "the respective output information of the first trained model 222 and the second trained model 224 are dissimilar (hereinafter simply referred to as "dissimilar")," as a predetermined result. That is, the input information adjustment unit 248 adjusts the input information 226 using the output information of the first trained model 222 as a reference so that the output information of the second trained model 224 is similar to the output result of the first trained model 222. Specifically, the input information (prompt) that instructs the first trained model 222 to add supplemental information, etc. to the input information 226 or to newly construct the input information 226 by incorporating the supplemental information, etc., is input to the first trained model 222, and the corresponding output information is adjusted as the input information 226, i.e., the adjustment information 232. This supplemental information includes the output style, output format, subject information, background information, detailed information, context-specific instruction information, etc. Specific examples of output style include academic output, explanatory or educational output, narrative output, polite output, persuasive sales output, explicit output, or implicit output. In other words, output style is information that describes how the output is presented. Specific examples of output format include text format, table format, and list format. The output format may also include level information, the number of hierarchies, and the number of layers. In other words, output format is information that describes the structure and format for displaying the output results. Specific examples of target audience information include general users, developers such as programmers, business professionals, and experts. In other words, target audience information is information that describes to whom the output results are intended. Specific examples of background information include emails to clients and job postings for recruitment. In other words, background information is information that describes the context for how the output will be used. Specific examples of detailed information include simple, concise, comprehensive, detailed, and explanatory. In other words, detailed information is information that describes the sophistication of the information. The adjustment information 232 obtained by the input information adjustment unit 248 is linked to the input information 226 and the evaluation information 230 and stored in the storage unit 22 .Furthermore, the context-specific instruction information is instruction information for comparing output information 228 obtained by inputting input information 226 to a reference trained model (in this embodiment, the first trained model 222) with output information 228 obtained by inputting the same input information 226 as the above-mentioned input information 226 to a target trained model (in this embodiment, the second trained model 224), and reducing the difference between the two. As a specific example, if the input information 226 is "I would like to exchange Japanese yen for US dollars," the first trained model 222 will obtain output information 228 stating "Based on the TTS (Telegraphic Transfer Selling) rate, the current exchange rate is 148 yen to the dollar," while the second trained model 224 will obtain output information 228 stating "Depending on the rate, the current exchange rate is likely to be in the 140 yen range to the dollar." In this case, since the output of the second trained model 224 is abstract compared to the output of the reference first trained model 222, context-specific instruction information such as "exchange shall be made based on the TTS rate" is generated to reduce this difference. In other words, if the output information 228 of the second trained model 224 is abstract compared to the output information 228 of the first trained model 222, the context-specific instruction information is specific information that is in line with the context and added by narrowing down some perspective. Note that the context-specific instruction information includes not only the above-mentioned level of abstraction, but also information that reduces the output difference between the first trained model 222 and the second trained model 224 in other perspectives such as reliability, completeness, availability, timeliness, and relevance.

[0040] When the input information adjustment unit 248 generates the adjustment information 232, the information processing unit 244 acquires the adjustment information 232 and controls the second trained model 224 to perform processing based on the adjustment information 232. The output evaluation unit 246 evaluates the output information 228 of the second trained model 224 based on the adjustment information 232 and the output information 228 of the first trained model 222 based on the input information 226, using the first trained model 222, according to a predetermined processing flow. When the evaluation result is "dissimilar," the input information adjustment unit 248 adjusts the input information 226 again, and when adjustment information 232 is generated as a result of this adjustment, the information processing unit 244 acquires the adjustment information 232 and controls the second trained model 224 to perform processing based on the adjustment information 232, and repeats the processing up to the evaluation by the output evaluation unit 246. In other words, the above series of processes is repeated until it is determined that "the output information of the first trained model 222 and the output information of the second trained model 224 are similar" (corresponding to "when a predetermined result is not obtained" in claim 5). Note that when adjusting the input information 226 again, the input information 226 to be adjusted may be the input information 226 evaluated by the output evaluation unit 246 immediately before, the input information 226 evaluated by the output evaluation unit 246 for the first time, or other input information 226.

[0041] If the evaluation result by the output evaluation unit 246 is that "the output information of the first trained model 222 and the second trained model 224 are similar (hereinafter simply referred to as "similar")," the output unit 250 controls the user terminal 14 to notify it that it is possible to switch from the first trained model 222 to the second trained model 224.

[0042] (Overall flow of processing executed by information processing system 10) Next, the operation of the information processing system 10 will be described. FIG. 4 is a flowchart showing an example of the flow of processing executed by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby carrying out processing. Although not shown, when the processor 120 receives operation information to terminate the operation of the information processing system 10, or operation termination information from the user terminal 14 during the ongoing determination processing (these will be simply referred to as "termination operations"), the processor 120 terminates the processing based on the program 220 that is currently being processed.

[0043] The processor 120 determines whether or not there is an instruction to switch the trained model used by the system to another trained model (step S100). This instruction may be issued by the user U through the user terminal 14, or may be issued automatically by the information processing system 10, the user terminal 14, or another information processing device based on a predetermined determination result. If there is no switching instruction (step S100: NO), the processor 120 terminates processing based on the program 220. On the other hand, if there is a switching instruction (step S100: YES), the processor 120 acquires various information related to the first trained model 222 and the second trained model 224 (step S102).

[0044] The processor 120 acquires input information 226 (step S102) and performs processing using the first trained model 222 based on the input information 226 (step S106). The processor 120 also performs processing using the second trained model 224 based on the same input information 226 (step S108).

[0045] The processor 120 acquires output information 228, which is the processing result of the first trained model 222 and the second trained model 224 (step S110). Note that the processing of step S110 corresponds to the "acquisition step" in claim 1. Then, the processor 120 evaluates the difference between the output content and the input information 226 (step S112). Thereafter, the processor 120 determines whether the evaluation results are similar (step S114). If the evaluation results are determined to be similar (step S114: YES), the processor 120 proceeds to step S126, which will be described later. On the other hand, if the evaluation results are determined to be not similar (i.e., dissimilar) (step S114: NO), the processor 120 adjusts the input information 226 (step S116). Note that steps S112 and S114 correspond to the "evaluation step" in claim 1, and step S116 corresponds to the "adjustment step" in claim 1. In other words, in the adjustment step, the input information 226 is changed so that the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 are similar.

[0046] The processor 120 performs processing on the second trained model 224 based on the adjustment information 232 generated as a result of adjusting the input information 226 (step S118). The processor 120 then acquires the output information of the second trained model 224 based on the adjustment information 232 and the output information 228 of the first trained model 222 based on the input information 226 (step S120), and evaluates the difference between the output contents of the acquired output information 228 (step S122). The processor 120 then determines whether the evaluation results are similar (step S124). If the processor 120 determines that the models are similar (step S124: YES), the processor 120 notifies the user terminal 14 that switching from the first trained model 222 to the second trained model 224 is possible (step S126), and terminates processing. On the other hand, if the processor 120 determines that the models are not similar (i.e., dissimilar) (step S124: NO), the processor 120 proceeds to step S116. The processes from step S120 to step S124 correspond to the "reevaluation step" of claim 3, and step S126 corresponds to the "output step" of claim 5.

[0047] (Processing Executed by Information Processing System 10 - Flow of Evaluation Step) Next, a predetermined processing flow that combines multiple processes performed in the evaluation step implemented by the information processing system 10 will be described. Fig. 5 is a flowchart showing an example of the processing flow of the evaluation step by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby performing the processing. This processing flow corresponds to the processing performed in steps S114 and S124 in the flowchart in Fig. 4 described above.

[0048] The processor 120 acquires the input information 226 and the output information 228, which are the processing results of the first trained model 222 and the second trained model 224 (step S200). Then, the processor 120 determines the intention of the input information 226 (step S202).

[0049] The processor 120 determines whether the intention of the input information 226 is clear (step S204). If the intention is not clear (step S204: NO), the processor 120 proceeds to step S230, which will be described later. On the other hand, if the intention is clear (step S204: YES), the processor 120 determines whether the intention is intended for brainstorming (step S206). If the intention is not intended for brainstorming (step S206: NO), the processor 120 proceeds to step S208, which will be described later. On the other hand, if the intention is intended for brainstorming (step S206: YES), the processor 120 determines whether each output of the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 (hereinafter simply referred to as "each output") matches the previously determined intention (step S222). If each output matches the previously determined intention (step S222: YES), the processor 120 determines whether the intentions of the outputs are of the same type (step S224). On the other hand, if each output does not match the previously determined intention (step S222: NO), the processor 120 performs a dissimilarity determination (step S232) and ends this processing flow.

[0050] If it is determined in step S224 that the intentions of the respective outputs are of the same type (step S224: YES), the processor 120 performs a similarity determination (step S230) and ends this processing flow. On the other hand, if the intentions of the respective outputs are not of the same type (step S224: NO), the processor 120 proceeds to step S232.

[0051] The processor 120 determines whether the intention of the input information 226 is confirmation (step S208). If the intention is confirmation (step S208: YES), the processor 120 proceeds to step S222. On the other hand, if the intention is not confirmation (step S208: NO), the processor 120 proceeds to step S210, which will be described later.

[0052] The processor 120 determines whether the intention of the input information 226 is to summarize (step S210). If the intention is to summarize (step S210: YES), the processor 120 determines whether each output retains important information from the original information before summarization (step S226). If the original information is retained (step S226: YES), the processor 120 proceeds to step S230. On the other hand, if the original information is not retained (step S226: NO), the processor 120 proceeds to step S232. Note that if the intention is not to summarize (step S210: NO), the processor 120 proceeds to step S212, which will be described later.

[0053] The processor 120 determines whether the intention of the input information 226 is an information request (step S212). If the intention is an information request (step S212: YES), the processor 120 determines whether each output satisfies the request level for the information request of the input information 226 (step S228). If the request level is satisfied (step S228: YES), the processor 120 proceeds to step S230. On the other hand, if the request level is not satisfied (step S228: NO), the processor 120 proceeds to step S232. Note that if the intention is not an information request (step S212: NO), the processor 120 determines the intention of the input information 226 as other (step S214).

[0054] The processor 120 determines whether each output matches the intention estimated from the input information 226 (step S216). Here, "intention" refers to the specific intention of the input information 226 (see the bottom of paragraph

[0038] ). If the output does not match the intention (step S216: NO), the processor 120 proceeds to step S232. On the other hand, if the output matches the intention (step S216: YES), the processor 120 determines whether there is similar information for each output (step S218). This determination regarding similar information is made based on the content, even if the expressions are different. If there is no similar information (step S218: NO), the processor 120 proceeds to step S232. On the other hand, if there is similar information (step S218: YES), the processor 120 determines whether each output corresponds to the information in the input information 226 (step S220). Regarding the correspondence of this information, for example, it may be determined whether or not the output style, output format, subject information, background information, and detailed information listed as the supplemental information described above are compatible, or it may be determined whether or not other information is compatible. If there is no compatibility (step S220: NO), the processor 120 proceeds to step S232. On the other hand, if there is compatibility (step S220: YES), the processor 120 proceeds to step S230.

[0055] (Effects of One Embodiment) According to the information processing system 10 of this embodiment, an acquisition step is performed to acquire output information 228 that is the result of inputting predetermined input information 226 to a first trained model 222 and a second trained model 224 that is different from the first trained model 222, an evaluation step is performed to compare and evaluate the output information 228, and an adjustment step is performed to change the input information 226 when the evaluation result is a predetermined result, i.e., dissimilar. This makes it possible to change the output information 228 based on the changed input information 226. In other words, it is possible to make the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 similar. This makes it possible to change the trained models while suppressing changes in the output information 228.

[0056] Furthermore, the evaluation step evaluates the output information of the second trained model 224 based on the output information of the first trained model 222. Therefore, by using this evaluation result, it becomes easy to select the second trained model 224, which has a small change in output compared to the first trained model 222, which is the trained model currently being used for information processing. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0057] Furthermore, a re-evaluation step is performed in which output information 228, which is the processing result when adjustment information 232, which is the input information 226 changed in the adjustment step, is input to the second trained model 224, is obtained, and the output information 228 from the first trained model 222, which is the processing result when the input information 226 before the change, is input, is compared and evaluated, thereby making it possible to generate adjustment information 232 with even less change in the output information 228 and to select the second trained model 224. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0058] Furthermore, since the adjustment step is executed again when the evaluation result in the re-evaluation step is a predetermined result, i.e., dissimilar, it is possible to generate adjustment information 232 and select the second trained model 224 with even less change in the output information 228. This makes it possible to change the trained model while further suppressing change in the output information 228.

[0059] Furthermore, since the method includes an output step of outputting a notification that it is possible to change from the first trained model 222 to the second trained model 224 when the evaluation result does not become a predetermined result, i.e., when the evaluation result becomes similar, the user can change the trained model based on the notification. This enables a smooth change of the trained model.

[0060] Furthermore, because the first trained model 222 and the second trained model 224 are language models, adjusting the input information 226 is equivalent to tuning the prompt. In other words, since the input information 226 can be easily adjusted, it is possible to change the trained model while further suppressing changes in the output information 228.

[0061] Furthermore, in the evaluation step, the output information 228 is evaluated according to a predetermined processing flow that combines multiple processes, thereby enabling stable evaluation. Therefore, the generation of adjustment information 232 and the selection of the second trained model 224, which further reduce changes in the output information 228, can be performed with higher accuracy through stable evaluation. This makes it possible to change the trained model while further suppressing changes in the output information 228. From another perspective, by performing processing according to a predetermined processing flow, processing instruction prompts can be compiled in advance when performing evaluation using a large-scale language model. In other words, compared to sequential processing in which a processing instruction prompt is input and then another processing instruction prompt is input based on the processing results, the number of tokens can be reduced, thereby enabling cost reduction.

[0062] Furthermore, in the evaluation step, the output information 228 is evaluated in accordance with the intention of the input information 226, so that it is possible to more accurately generate adjustment information 232 that further reduces changes in the output information 228 and select the second trained model 224. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0063] In the above-described embodiment, the first trained model 222 and the second trained model 224 are large-scale language models, but the present invention is not limited to this and may be any other language model, or a trained model other than a large-scale language model, such as an image classification model, an image generation model, an object detection model, a speech recognition model, or an inference model. In the image classification model, the outputs of the two image classification models may be evaluated using a comparison of accuracy for each class, a comparison of confusion matrices, an ROC curve and an AUC (Area Under the Curve), Cohen's Kappa coefficient, or the like, and the input information 226 may be adjusted by adjusting hyperparameters, adjusting a learning rate, augmenting data, or the like. In the image generation model, the outputs of the two image generation models may be evaluated using Inception Score, Frechet Inception Distance, Perceptual Path Length, Kernel Inception Distance, visual comparison, etc., and the input information 226 may be adjusted by adjusting hyperparameters, balancing the generator and the classifier, etc. In the object detection model, the outputs of the two object detection models may be evaluated using Intersection over Union, Precision and Recall, Mean Average Precision, etc., and the input information 226 may be adjusted by adjusting the size and aspect ratio of the anchor box, adjusting hyperparameters, adjusting the threshold for Non-Maximum Suppression, etc. In the speech recognition model, the outputs of the two speech recognition models may be evaluated using the Word Error Rate, Sentence Error Rate, Character Error Rate, etc., and the input information 226 may be adjusted by adjusting hyperparameters, removing noise from training data, or augmenting the data.In the inference model, the outputs of the two inference models may be evaluated using the Mean Absolute Error, Root Mean Squared Error, R-squared (coefficient of determination), Mean Absolute Percentage Error, etc., and the input information 226 may be adjusted by adjusting hyperparameters, selecting and adjusting features, regularizing to prevent overlearning, etc.

[0064] Moreover, although the evaluation is configured to be performed using the first trained model 222, the configuration is not limited to this, and the second trained model 224 or another trained model may be used for the evaluation. Furthermore, although the evaluation is configured to be performed using the output information 228 of the first trained model 222 as a criterion, the configuration is not limited to this, and the evaluation may be performed using the output information 228 of the second trained model 224, output information of other trained models or algorithms, information specified by the user U, or the like as a criterion.

[0065] Furthermore, in the above-described embodiment, the re-evaluation step is executed, but the present invention is not limited to this, and the re-evaluation may not be executed.

[0066] In addition, in the output step, a notification that a change from the first trained model 222 to the second trained model 224 is possible is output, but this is not limited to this, and a configuration may be adopted in which no notification is issued. Furthermore, regardless of whether the above-mentioned notification is issued or not, an instruction to switch from the first trained model 222 to the second trained model 224 may be output, or the switching may be automatically executed.

[0067] Furthermore, although the configuration is such that the adjustment information 232, which is the input information 226 changed in the adjustment step, is input only to the second trained model 224, the configuration is not limited thereto, and the adjustment information 232 may be input only to the first trained model 222, or to both the first trained model 222 and the second trained model 224. When the adjustment information 232 is input only to the first trained model 222, the input information 226 is changed based on the output information 228 of the second trained model 224, the configuration is not limited thereto. Furthermore, when the adjustment information 232 is input to both the first trained model 222 and the second trained model 224, the configuration is such that either the output information 228 of the first trained model 222 or the output information 228 of the second trained model 224 is dynamically set as the reference depending on the content of the output information 228, the configuration is not limited thereto.

[0068] (Variation 1) In the above-described embodiment, as shown in FIG. 5, the evaluation is performed in accordance with a predetermined flow in the evaluation step. However, in this evaluation, the similarity may be evaluated in multiple stages of three or more, such as "high similarity," "moderate similarity," and "dissimilar."

[0069] As an example, Fig. 6 is a flowchart showing an example of the flow of processing of the evaluation step by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby performing the processing. This processing flow corresponds to the processing performed in steps S114 and S124 in the flowchart in Fig. 4 described above. Note that the same processes as those in the above-described embodiment are assigned the same reference numerals, and their description will be omitted.

[0070] After step S202, the processor 120 determines whether the intention of the input information 226 is clear (step S300). If the intention is not clear (step S300: NO), the processor 120 proceeds to step S312, which will be described later. On the other hand, if the intention is clear (step S300: YES), the processor 120 proceeds to step S206.

[0071] In the processing of step S222, if each output matches the intention determined in the previous determination (step S222: YES), the processor 120 determines whether the intentions of each output are of the same type (step S302). On the other hand, if each output does not match the intention determined in the previous determination (step S222: NO), the processor 120 proceeds to processing of step S232. On the other hand, if the intentions of each output are of the same type (step S302: YES), the processor 120 performs a high similarity determination (step S312) and ends this processing flow. On the other hand, if the intentions of each output are not of the same type (step S302: NO), the processor 120 proceeds to processing of step S232.

[0072] In the process of step S210, if the intention is summarization (step S210: YES), the processor 120 determines whether each output retains important information from the original information before summarization (step S304). If the original information is retained (step S304: YES), the processor 120 proceeds to step S308, which will be described later. On the other hand, if the original information is not retained (step S304: NO), the processor 120 proceeds to step S232.

[0073] In the process of step S212, if the intention is an information request (step S212: YES), the processor 120 determines whether each output satisfies the request level for the information request of the input information 226 (step S306). If the request level is satisfied (step S306: YES), the processor 120 proceeds to step S308. On the other hand, if the request level is not satisfied (step S306: NO), the processor 120 proceeds to step S232.

[0074] If the processing in step S220 shows that the two images correspond (step S220: YES), the processor 120 determines whether the similarity is high (step S308). This similarity determination involves calculating the similarity using a predetermined similarity determination process, and determining that the similarity is high if the similarity is equal to or greater than the predetermined level, or medium if the similarity is equal to or less than the predetermined level. If the similarity is equal to or greater than the predetermined level (step S308: YES), the processor 120 determines that the similarity is high (step S312) and ends the processing. On the other hand, if the similarity is equal to or less than the predetermined level (step S308: NO), the processor 120 determines that the similarity is medium (step S310) and ends the processing.

[0075] According to the above-described modified example, by configuring evaluation in three or more stages, it is possible to generate adjustment information 232 that further reduces changes in the output information 228 and select the second trained model 224 with higher accuracy through stable evaluation. This makes it possible to change the trained model while further suppressing changes in the output information 228. Note that, although the above-described modified example uses multiple stages, the present invention is not limited to this, and a configuration in which evaluation is performed continuously according to the similarity score may also be used.

[0076] (Variant 2) Furthermore, when the above-described information processing system is viewed from a different perspective, the problem (purpose) that the information processing system according to this embodiment aims to solve can also be viewed as "outputting a trained model in accordance with desired specified information."

[0077] If the problem is understood as described above, an invention as a means for solving the problem could be, for example, as follows: "An information processing method executed by an information processing device, comprising: an acquisition step of acquiring designated predetermined information and output information that is a result of inputting the predetermined input information to a trained model; an evaluation step of comparing and evaluating the predetermined information and the output information; and an adjustment step of changing the input information when the evaluation results in dissimilarities."

[0078] According to the above configuration, the output information of the trained model is compared with predetermined information for evaluation. Specifically, the similarity is evaluated. If the evaluation results in dissimilarity, the input information is changed, thereby making the output information of the trained model similar to the predetermined information. This predetermined information is at least one of output information from another trained model, output information from the same trained model under different versions or conditions, output information via a system using a rule base, etc., and information created or specified by a person such as a user, and is specified as the desired information to be obtained. This makes it possible to obtain output from the trained model that conforms to the specified desired predetermined information.

[0079] In this modification, the output information of the trained model may be evaluated based on the predetermined information. This makes it possible to obtain the output of the trained model that is more in line with the specified desired predetermined information.

[0080] Furthermore, this modified example may include a re-evaluation step of acquiring output information obtained by inputting the input information changed in the adjustment step to the trained model, and comparing and evaluating the output information with the predetermined information. This makes it possible to obtain an output of the trained model that is more in line with the specified desired predetermined information.

[0081] Furthermore, in this modified example, the adjustment step may be executed again if the result of the evaluation in the re-evaluation step is dissimilar. This makes it possible to obtain an output of a trained model that is more in line with the specified desired information.

[0082] Furthermore, this modified example may be configured to "evaluate the output information according to a predetermined processing flow that combines multiple processes." This allows for stable evaluation. Therefore, it is possible to obtain the output of a trained model that is more in line with the specified desired information. From another perspective, by performing processing according to a predetermined processing flow, it is possible to summarize processing instruction prompts in advance when performing evaluation using a large-scale language model. In other words, compared to sequential processing in which a processing instruction prompt is input and then another processing instruction prompt is input based on the processing result, the number of tokens can be reduced, thereby enabling cost reduction.

[0083] Furthermore, in this variation, the evaluation step may be configured to evaluate the output information in accordance with the intent of the input information. This makes it possible to obtain an output that is less likely to deviate from the intent of the desired predetermined information, thereby obtaining an output of a trained model that is more in line with the specified desired predetermined information.

[0084] <Additional Notes> This embodiment includes the following disclosure.

[0085] (Supplementary Note 1) An information processing method executed by an information processing device, comprising: an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; and an adjustment step of changing the input information when the result of the evaluation is a specified result.

[0086] (Supplementary Note 2) The information processing method according to Supplementary Note 1, wherein the evaluation step evaluates the output information of the second trained model based on the output information of the first trained model.

[0087] (Supplementary Note 3) The information processing method according to Supplementary Note 1 or Supplementary Note 2, further comprising a re-evaluation step of obtaining output information obtained by inputting the input information changed in the adjustment step into the second trained model, and comparing and evaluating the output information from the first trained model to which the input information before the change has been input.

[0088] (Supplementary Note 4) The information processing method according to Supplementary Note 3, wherein the adjustment step is executed again when a result of the evaluation by the re-evaluation step becomes a predetermined result.

[0089] (Supplementary Note 5) The information processing method according to any one of Supplementary Notes 1 to 4, further comprising an output step of outputting, when a result of the evaluation does not become the predetermined result, a message indicating that the first trained model can be changed to the second trained model.

[0090] (Supplementary Note 6) The information processing method according to any one of Supplementary Notes 1 to 5, wherein the first trained model and the second trained model are language models.

[0091] (Supplementary Note 7) The information processing method according to any one of Supplementary Notes 1 to 6, wherein in the evaluation step, the output information is evaluated according to a predetermined processing flow that combines a plurality of processes.

[0092] (Supplementary Note 8) The information processing method according to any one of Supplementary Notes 1 to 7, wherein in the evaluation step, the output information is evaluated in accordance with an intention of the input information.

[0093] (Appendix 9) A program for causing an information processing device to execute an information processing method, including: an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating each of the output information; and an adjustment step of changing the input information when the result of the evaluation is a specified result.

[0094] (Supplementary Note 10) An information processing system executed by an information processing device, which performs an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation is a specified result.

[0095] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present disclosure is defined by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure.

[0096] The disclosure of Japanese Patent Application No. 2024-18222, filed on February 8, 2024, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An information processing method executed by an information processing device, comprising: an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; and an adjustment step of changing the input information when the result of the evaluation is a specified result.

2. The information processing method according to claim 1, wherein the evaluation step evaluates the output information of the second trained model based on the output information of the first trained model.

3. The information processing method of claim 1, further comprising a re-evaluation step of obtaining output information by inputting the input information changed in the adjustment step into the second trained model, and comparing and evaluating the output information from the first trained model to which the input information before the change has been input.

4. The information processing method according to claim 3, wherein the adjustment step is executed again when the result of the evaluation in the re-evaluation step is a predetermined result.

5. The information processing method according to claim 1, further comprising an output step of outputting information indicating that the first trained model can be changed to the second trained model if the result of the evaluation does not become the predetermined result.

6. The information processing method according to claim 1, wherein the first trained model and the second trained model are language models.

7. The information processing method according to claim 1, wherein in the evaluation step, the output information is evaluated according to a predetermined processing flow that combines a plurality of processes.

8. The information processing method according to claim 1, wherein in the evaluation step, the output information is evaluated in accordance with the intention of the input information.

9. A program for causing an information processing device to execute an information processing method including: an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating each of the output information; and an adjustment step of changing the input information when the result of the evaluation is a specified result.

10. An information processing system executed by an information processing device, which performs an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating each of the output information, and an adjustment step of changing the input information when the result of the evaluation becomes a specified result.

Images