Information processing method, program, and information processing system

Abstract

To enable modification of trained models while suppressing changes in output information.SOLUTION: In an information processing system comprising a server as an information processing device and a user terminal as an information processing device, which are communicatively connected to each other via a network, an information processing method executed by the server as an information processing device includes an acquisition step of acquiring pieces of output information that are results of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the pieces of output information, and an adjustment step of changing the input information when a result of the evaluation becomes a specified result.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses a processing system that uses trained models. In this processing system, the trained model used for processing is switched based on timestamp information of the data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6558188 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when performing information processing using a trained model, there are cases where it is desired to change the trained model being used due to the discontinuation of the service of the trained model itself, switching to a similar trained model, etc. However, changing the trained model may change the output information, and there is room for improvement in this regard.

[0005] Taking the above facts into consideration, the present invention aims to enable changes to a trained model while suppressing changes in output information. [Means for solving the problem]

[0006] According to one embodiment of an information processing method, the information processing method is executed by an information processing device, and includes an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation becomes a specified result.

[0007] According to one embodiment of the program, an information processing device is caused to execute an information processing method including an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the first trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation becomes a specified result.

[0008] According to one embodiment, the information processing system is executed by an information processing device and includes an acquisition step of acquiring output information that is the result of inputting specified input information to a first trained model and a second trained model that is different from the trained model, an evaluation step of comparing and evaluating the output information, and an adjustment step of changing the input information when the result of the evaluation is a specified result. [Effects of the Invention]

[0009] According to one embodiment, it is possible to change a trained model while suppressing changes in output information. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a processing flow of an information processing system according to an embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a flow of evaluation steps in an information processing system according to an embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a flow of evaluation steps in a modified example of an information processing system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 6. In each drawing, the same or equivalent components and parts are denoted by the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] (System Overview) First, an overview of an information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is an information processing system for performing at least a part of the change (switch) process of a trained model to be used. This information processing system 10 is not limited to a system that performs change processing of a trained model for another information processing system that performs information processing using the trained model, but may also be a system that performs change processing on a trained model that the information processing system 10 itself owns.

[0013] (System Configuration) Fig. 1 is a diagram showing an example of the configuration of an information processing system 10 according to this embodiment. As shown in Fig. 1, the information processing system 10 according to this embodiment includes a server 12 as an information processing device and a user terminal 14 as an information processing device, which are communicably connected to each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof.

[0014] The user terminal 14 is an example of an information processing device that is operated by a user U to input and display various information. The user terminal 14 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these.

[0015] The server 12 is an example of an information processing device that acquires information input from the user terminal 14, processes the information, and outputs the results. The server 12 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server 12 will be described later.

[0016] (Hardware configuration) 2 is a block diagram showing the hardware configuration of the server 12. The server 12 includes a processor 120, a memory 122, a storage 124, a communication I / F 126, an input / output I / F 128, and a drive device 134, which are communicatively connected to each other via a bus B.

[0017] The processor 120 controls each component of the server 12 and realizes the functions of the server 12 by loading various programs stored in the storage 124 into the memory 122 and executing them. The programs executed by the processor 120 include, but are not limited to, an operating system (OS) and a program 220 described below. Execution of these programs by the processor 120 realizes part of the state visualization method according to this embodiment. The processor 120 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or a combination thereof.

[0018] The memory 122 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0019] The storage 124 stores the OS, various programs described below, and various data. The storage 124 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0020] The communication I / F 126 is an interface for connecting the server 12 to external devices including the user terminal 14 via the network N and controlling communication. The communication I / F 126 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0021] The input / output I / F 128 is an interface for connecting an input device 132 and an output device 130 to the server 12. The input device 132 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, an operation button, or a combination thereof. The output device 130 is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination thereof.

[0022] The drive device 134 reads and writes data from and to the disk media 136. The drive device 134 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or a combination thereof. The disk media 136 is, for example, a compact disc (CD), a digital versatile disc (DVD), a floppy disk (FD), a magneto-optical disk (MO), a Blu-ray (registered trademark) disc (BD), or a combination thereof.

[0023] In this embodiment, the program may be written into memory 122 or storage 124 during the manufacturing stage of server 12, may be provided to server 12 via network N, or may be provided to server 12 via a non-transitory computer-readable recording medium such as disk media 136.

[0024] Furthermore, the hardware configuration of the user terminal 14 is substantially the same as the hardware configuration of the server 12 described above, and therefore a detailed description thereof will be omitted.

[0025] (Functional configuration) Next, the functional configuration of the server 12 will be described. Fig. 3 is a diagram showing an example of the functional configuration of the server 12. When executing various programs, the server 12 uses the above-mentioned hardware resources to realize various functions. The server 12 has a communication unit 20, a storage unit 22, and a control unit 24 as the functional configuration realized by the server 12. Each functional configuration is realized by the processor 120 reading and executing a program 220 stored in the memory 122 or the storage 124.

[0026] The communication unit 20 is realized by the communication I / F 126. The communication unit 20 transmits and receives information to and from the user terminal 14 via the network N. The communication unit 20 receives information input from the user terminal 14. The communication unit 20 also transmits information to the user terminal 14 and receives requests from the user U from the user terminal 14.

[0027] The memory unit 22 is realized by a memory 122 and a storage 124. The memory unit 22 stores a program 220, a first trained model 222, a second trained model 224, input information 226, output information 228, evaluation information 230, and adjustment information 232.

[0028] The first trained model 222 and the second trained model 224 are configured by at least one trained machine learning model. As an example, the first trained model 222 and the second trained model 224 are large-scale language models that are trained with large amounts of text data from internet articles, books, websites, etc. When text data called a prompt is input as input information 226, information processing is performed according to the prompt, and as a result, various data including the text data are generated and output as output information 228.

[0029] The first trained model 222 and the second trained model 224 are considered to be different trained models. As a specific example, the first trained model 222 is GPT-4 from OpenAI, and the second trained model 224 is Claude from ANTHROPIC. Note that the first trained model 222 and the second trained model 224 are not limited to this example and may be language models including other large-scale language models. Furthermore, the first trained model 222 and the second trained model 224 may be stored not in the memory unit 22 but on an external server or the like. Note that the above-mentioned "(the first trained model 222 and the second trained model 224 are considered to be) different (trained models)" includes not only different trained models but also different versions of the same trained model as "different" models.

[0030] As described above, the input information 226 is a prompt that instructs processing on the first trained model 222 and the second trained model 224, and is text data in natural language. This input information 226 is input by the user U, but is not limited to this, and may be information automatically generated or modified by the system.

[0031] The output information 228 is information indicating the results of processing performed by at least one of the first trained model 222 and the second trained model 224 based on the prompt. The output information 228 is various information including text information and image information, and is stored in association with the prompt.

[0032] The evaluation information 230 is information on the results of an evaluation of the output information 228 of at least one of the first trained model 222 and the second trained model 224 and the prompt (input information 226) linked thereto. Specifically, the evaluation information 230 is information that determines the intention of the input information 226 and evaluates how well the output information 228 matches that intention. The evaluation information 230 also includes information that compares the output information 228 of the first trained model 222 and the second trained model 224 and determines the degree of similarity in content. Details will be described later.

[0033] The adjustment information 232 is text data that has been modified from the input information 226. This adjustment information 232 is information in which information that reflects the changes has been added to the original input information 226, or information in which the original input information 226 has been rewritten based on the changes. Details will be described later.

[0034] The control unit 24 is realized by the processor 120 reading and executing the program 220 from the memory 122 (see FIG. 2) and working in cooperation with other hardware components. The control unit 24 includes an information acquisition unit 242, an information processing unit 244, an output evaluation unit 246, an input information adjustment unit 248, and an output unit 250.

[0035] The information acquisition unit 242 acquires information about large-scale language models specified by the user U as the first trained model 222 and the second trained model 224, as well as information for using the large-scale language models. Specifically, the information acquisition unit 242 acquires the type and name of the currently used trained model as the first trained model 222 (GPT-4, a large-scale language model, in this embodiment) and information for using the trained model, such as an ID, password, and API key. Furthermore, the information acquisition unit 242 acquires the type and name of the trained model (Claude, a large-scale language model, in this embodiment) and information for using the trained model, such as an ID, password, and API key, for the trained model whose usage is to be switched (changed) from the currently used first trained model 222, as the second trained model 224. The information about the first trained model 222 and the second trained model 224 may be acquired when a trained model switching instruction is received from the user U or an information processing device, or may be acquired in advance regardless of whether a switching instruction is issued.

[0036] Furthermore, the information acquisition unit 242 acquires various pieces of information required for processing in the information processing unit 244. Specifically, it acquires input information 226 that instructs processing and output information 228 that is the processing result of at least one of the first trained model 222 and the second trained model 224 based on the input information 226. This acquired information may be information acquired from the user terminal 14, information stored in the memory unit 22, or information acquired from an external data source or the like.

[0037] The information processing unit 244 performs information processing using the first trained model 222 and the second trained model 224 based on the input information 226. The processing based on this input information 226 executes, as necessary, control to perform processing based on the same input information 226 in each of the first trained model 222 and the second trained model 224, and control to perform processing based on the input information 226 in only one of the first trained model 222 and the second trained model 224.

[0038] The output evaluation unit 246 acquires output information 228, which is the processing result of the first trained model 222 and the second trained model 224, based on the same input information 226, and evaluates the difference in the output content for the input information 226. This evaluation is performed according to a predetermined processing flow that combines multiple processes including the determination of intention, which will be described later, using the output information of the first trained model 222 as a reference. The output evaluation unit 246 performs the evaluation of the output information 228 using the first trained model 222. That is, the evaluation is performed using the first trained model 222 by prompting the first trained model 222 to evaluate the two pieces of output information 228 for the input information 226 and instructing the first trained model 222 to perform the evaluation according to a predetermined processing flow.

[0039] The output evaluator 246 also determines the intent of the input information 226. This intent refers to the thought or purpose behind the prompt, which is the input information 226. Specifically, intents are broadly categorized into five categories: "brainstorming," "confirmation," "summary," "information request," and "other." "Brainstorming" is when the user is seeking ideas or suggestions. In most cases, the user does not have a set answer. "Confirmation" provides some guidelines or instructions (e.g., a prompt that instructs a large-scale language model on the role of "You are an intellectual property expert") without specific input for the user to process. In this case, the system typically responds with an affirmative or acknowledgment. "Summarization" is when the user is seeking high-level, broad information. Examples include writing code in a specific programming language, defining or explaining specific technical terms, summarizing a text, or answering general knowledge questions that do not require detailed specificity. "Information request" is when the user is seeking low-level information or performing a very detailed, specific task. Examples include requesting specific information in a specific scenario, requesting that a sentence be translated or rephrased, formatting a response in a specific style, or responding in a specific tabular format. "Other" is an intention that does not fall under the four above-mentioned intentions. Note that in this embodiment, the intentions are broadly classified into the five above-mentioned intentions, but this is not limiting, and classification by other intentions or classification by a different number of categories may also be used. Furthermore, when the output evaluation unit 246 classifies the intention of the input information 226 as "other," it further individually and specifically determines the intention of the input information 226.

[0040] The input information adjustment unit 248 adjusts the input information 226 when the evaluation result in the output evaluation unit 246 indicates that "the respective output information of the first trained model 222 and the second trained model 224 are dissimilar (hereinafter simply referred to as "dissimilar")," as a predetermined result. That is, the input information adjustment unit 248 adjusts the input information 226 using the output information of the first trained model 222 as a reference so that the output information of the second trained model 224 is similar to the output result of the first trained model 222. Specifically, the input information (prompt) that instructs the first trained model 222 to add supplementary information or the like to the input information 226 or to newly construct the input information 226 by incorporating the supplementary information or the like is input to the first trained model 222, and the corresponding output information is adjusted as the input information 226, i.e., the adjustment information 232. This supplementary information includes the output style, output format, subject information, background information, detailed information, context-specific instruction information, etc. Specific examples of output style include academic output, explanatory or educational output, narrative output, polite output, persuasive sales output, explicit output, or implicit output. In other words, output style is information that describes how the output is presented. Specific examples of output format include text format, table format, and list format. The output format may also include level information, the number of hierarchies, and the number of layers. In other words, output format is information that describes the structure and format for displaying the output results. Specific examples of target audience information include general users, developers such as programmers, business professionals, and experts. In other words, target audience information is information that describes to whom the output results are intended. Specific examples of background information include emails to clients and job postings for recruitment. In other words, background information is information that describes the context for how the output will be used. Specific examples of detailed information include simple, concise, comprehensive, detailed, and explanatory. In other words, detailed information is information that describes the sophistication of the information. The adjustment information 232 obtained by the input information adjustment unit 248 is linked to the input information 226 and the evaluation information 230 and stored in the storage unit 22.Furthermore, the context-specific instruction information is instruction information for comparing output information 228 obtained by inputting input information 226 to a reference trained model (in this embodiment, the first trained model 222) with output information 228 obtained by inputting the same input information 226 as the above-mentioned input information 226 to a target trained model (in this embodiment, the second trained model 224), and reducing the difference between the two. As a specific example, if the input information 226 is "I would like to exchange Japanese yen for US dollars," the first trained model 222 will obtain output information 228 saying, "Based on the TTS (Telegraphic Transfer Selling) rate, the current exchange rate is 148 yen to the dollar," while the second trained model 224 will obtain output information 228 saying, "It depends on the exchange rate, but the current exchange rate is likely to be in the 140 yen range to the dollar." In this case, since the output of the second trained model 224 is abstract compared to the output of the reference first trained model 222, context-specific instruction information such as "exchange shall be made based on the TTS rate" is generated to reduce this difference. In other words, if the output information 228 of the second trained model 224 is abstract compared to the output information 228 of the first trained model 222, the context-specific instruction information is specific information that is in line with the context and added by narrowing down some perspective. Note that the context-specific instruction information includes not only the above-mentioned level of abstraction, but also information that reduces the output difference between the first trained model 222 and the second trained model 224 in other perspectives such as reliability, completeness, availability, timeliness, and relevance.

[0041] When the input information adjustment unit 248 generates the adjustment information 232, the information processing unit 244 acquires the adjustment information 232 and controls the second trained model 224 to perform processing based on the adjustment information 232. The output evaluation unit 246 evaluates the output information 228 of the second trained model 224 based on the adjustment information 232 and the output information 228 of the first trained model 222 based on the input information 226, using the first trained model 222, according to a predetermined processing flow. When the evaluation result is "dissimilar," the input information adjustment unit 248 adjusts the input information 226 again, and when the adjustment information 232 resulting from this adjustment is generated, the information processing unit 244 acquires the adjustment information 232 and controls the second trained model 224 to perform processing based on the adjustment information 232, and repeats the processing up to the evaluation by the output evaluation unit 246. In other words, the above series of processes are repeated until it is determined that "the output information of the first trained model 222 and the second trained model 224 are similar" (corresponding to "when a predetermined result is not obtained" in claim 5). Note that when adjusting the input information 226 again, the input information 226 to be adjusted may be the input information 226 evaluated by the output evaluation unit 246 immediately before, the input information 226 evaluated by the output evaluation unit 246 for the first time, or other input information 226.

[0042] When the evaluation result by the output evaluation unit 246 indicates that "the output information of the first trained model 222 and the second trained model 224 are similar (hereinafter simply referred to as "similar")," the output unit 250 controls to notify the user terminal 14 that it is possible to switch from the first trained model 222 to the second trained model 224.

[0043] (Overall flow of processing executed by information processing system 10) Next, the operation of the information processing system 10 will be described. Fig. 4 is a flowchart showing an example of the flow of processing by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby performing processing. Although not shown, when the processor 120 receives operation information to terminate the operation of the information processing system 10, or operation termination information from the user terminal 14 during the determination processing being executed (these will be simply referred to as "termination operations"), the processor 120 terminates the processing based on the program 220 being processed.

[0044] The processor 120 determines whether or not there is an instruction to switch the trained model used by the system to another trained model (step S100). This instruction may be issued by the user U through the user terminal 14, or may be issued automatically by the information processing system 10, the user terminal 14, or another information processing device based on a predetermined determination result. If there is no switching instruction (step S100: NO), the processor 120 terminates processing based on the program 220. On the other hand, if there is a switching instruction received (step S100: YES), the processor 120 acquires various information related to the first trained model 222 and the second trained model 224 (step S102).

[0045] The processor 120 acquires input information 226 (step S102) and performs processing using the first trained model 222 based on the input information 226 (step S106). The processor 120 also performs processing using the second trained model 224 based on the same input information 226 (step S108).

[0046] The processor 120 acquires output information 228, which is the processing result of the first trained model 222 and the second trained model 224 (step S110). The processing of step S110 corresponds to the "acquisition step" of claim 1. Then, the processor 120 evaluates the difference between the output content and the input information 226 (step S112). Thereafter, the processor 120 determines whether the evaluation results are similar or not (step S114). If the evaluation results are determined to be similar (step S114: YES), the processor 120 proceeds to step S126, which will be described later. On the other hand, if the evaluation results are determined to be not similar (i.e., dissimilar) (step S114: NO), the processor 120 adjusts the input information 226 (step S116). Note that steps S112 and S114 correspond to the "evaluation step" of claim 1, and step S116 corresponds to the "adjustment step" of claim 1. In other words, in the adjustment step, the input information 226 is changed so that the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 are similar to each other.

[0047] The processor 120 performs processing on the second trained model 224 based on the adjustment information 232 generated as a result of adjusting the input information 226 (step S118). Then, the processor 120 acquires the output information of the second trained model 224 based on the adjustment information 232 and the output information 228 of the first trained model 222 based on the input information 226 (step S120), and evaluates the difference between the output contents of the acquired output information 228 (step S122). The processor 120 then determines whether the evaluation results are similar (step S124). If the processor 120 determines that the models are similar (step S124: YES), the processor 120 notifies the user terminal 14 that it is possible to switch from the first trained model 222 to the second trained model 224 (step S126), and terminates the processing. On the other hand, if the processor 120 determines that the models are not similar (i.e., dissimilar) (step S124: NO), the processor 120 proceeds to step S116. The processes from step S120 to step S124 described above correspond to the "reevaluation step" of claim 3, and step S126 corresponds to the "output step" of claim 5.

[0048] (Flow of the process-evaluation step executed by the information processing system 10) Next, a description will be given of a predetermined processing flow that combines multiple processes performed in the evaluation step executed by the information processing system 10. Fig. 5 is a flowchart showing an example of the processing flow of the evaluation step by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby performing the processing. This processing flow corresponds to the processing performed in steps S114 and S124 in the flowchart in Fig. 4 described above.

[0049] The processor 120 acquires the input information 226 and the output information 228, which are the processing results of the first trained model 222 and the second trained model 224 (step S200). Then, the processor 120 determines the intention of the input information 226 (step S202).

[0050] The processor 120 determines whether the intention of the input information 226 is clear (step S204). If the intention is not clear (step S204: NO), the processor 120 proceeds to step S230, which will be described later. On the other hand, if the intention is clear (step S204: YES), the processor 120 determines whether the intention is intended for brainstorming (step S206). If the intention is not intended for brainstorming (step S206: NO), the processor 120 proceeds to step S208, which will be described later. On the other hand, if the intention is intended for brainstorming (step S206: YES), the processor 120 determines whether each output of the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 (hereinafter simply referred to as "each output") matches the previously determined intention (step S222). If each output matches the intention of the previous determination (step S222: YES), processor 120 determines whether the intentions of each output are of the same type (step S224). On the other hand, if each output does not match the intention of the previous determination (step S222: NO), processor 120 performs a dissimilarity determination (step S232) and ends this processing flow.

[0051] If it is determined in step S224 that the intentions of the respective outputs are of the same type (step S224: YES), processor 120 performs a similarity determination (step S230) and ends this processing flow. On the other hand, if the intentions of the respective outputs are not of the same type (step S224: NO), processor 120 proceeds to step S232.

[0052] Processor 120 determines whether the intention of input information 226 is confirmation (step S208). If the intention is confirmation (step S208: YES), processor 120 proceeds to step S222. On the other hand, if the intention is not confirmation (step S208: NO), processor 120 proceeds to step S210, which will be described later.

[0053] The processor 120 determines whether the intention of the input information 226 is to summarize (step S210). If the intention is to summarize (step S210: YES), the processor 120 determines whether each output retains important information in the original information before summarization (step S226). If the original information is retained (step S226: YES), the processor 120 proceeds to step S230. On the other hand, if the original information is not retained (step S226: NO), the processor 120 proceeds to step S232. Note that if the intention is not to summarize (step S210: NO), the processor 120 proceeds to step S212, which will be described later.

[0054] The processor 120 determines whether the intention of the input information 226 is an information request (step S212). If the intention is an information request (step S212: YES), the processor 120 determines whether each output satisfies the request level for the information request of the input information 226 (step S228). If the request level is satisfied (step S228: YES), the processor 120 proceeds to step S230. On the other hand, if the request level is not satisfied (step S228: NO), the processor 120 proceeds to step S232. Note that if the intention is not an information request (step S212: NO), the processor 120 determines the intention of the input information 226 to be other (step S214).

[0055] The processor 120 determines whether each output matches the intention estimated from the input information 226 (step S216). The "intention" here corresponds to the individual specific intention of the input information 226 (see paragraph

[0056] (See the lower part.) If it does not match the intention (step S216: NO), processor 120 proceeds to step S232. On the other hand, if it matches the intention (step S216: YES), processor 120 determines whether or not there is similar information in each output (step S218). This determination regarding similar information is made based on the content even if the expressions are different. If there is no similar information (step S218: NO), processor 120 proceeds to step S232. On the other hand, if there is similar information (step S218: YES), processor 120 proceeds to step S232. Regarding the correspondence of this information, for example, it may be determined whether or not there is correspondence in the output style, output format, target person information, background information, and detailed information listed as the supplemental information described above, or it may be determined whether or not there is correspondence in other information. If there is no correspondence (step S220: NO), processor 120 proceeds to step S232. On the other hand, if it is supported (step S220: YES), processor 120 proceeds to step S230.

[0057] (Effects of one embodiment) According to the information processing system 10 of this embodiment, an acquisition step is performed to acquire output information 228, which is the result of inputting predetermined input information 226 to a first trained model 222 and a second trained model 224 that is different from the first trained model 222, an evaluation step is performed to compare and evaluate the output information 228, and an adjustment step is performed to change the input information 226 when the evaluation result is a predetermined result, i.e., dissimilar. This makes it possible to change the output information 228 based on the changed input information 226. In other words, it is possible to make the output information 228 of the first trained model 222 and the output information 228 of the second trained model 224 similar. This makes it possible to change the trained model while suppressing changes in the output information 228.

[0058] Furthermore, the evaluation step evaluates the output information of the second trained model 224 based on the output information of the first trained model 222. Therefore, by using this evaluation result, it becomes easy to select the second trained model 224, which has a small change in output compared to the first trained model 222, which is the trained model currently being used for information processing. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0059] Furthermore, a re-evaluation step is performed in which output information 228, which is the processing result when adjustment information 232, which is the input information 226 changed in the adjustment step, is input to the second trained model 224, is obtained, and the output information 228 from the first trained model 222, which is the processing result when the input information 226 before the change, is input, is compared and evaluated, thereby making it possible to generate adjustment information 232 with even less change in the output information 228 and select the second trained model 224. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0060] Furthermore, since the adjustment step is executed again when the evaluation result in the re-evaluation step is a predetermined result, i.e., dissimilar, it is possible to generate adjustment information 232 and select the second trained model 224 with even less change in the output information 228. This makes it possible to change the trained model while further suppressing change in the output information 228.

[0061] Furthermore, if the evaluation result does not become a predetermined result, i.e., if the evaluation result is similar, an output step is included that outputs a notification that it is possible to change from the first trained model 222 to the second trained model 224, so that the user can change the trained model based on the notification. This enables a smooth change of the trained model.

[0062] Furthermore, since the first trained model 222 and the second trained model 224 are language models, adjusting the input information 226 is what is called prompt tuning. In other words, since the input information 226 can be easily adjusted, it is possible to change the trained model while further suppressing changes in the output information 228.

[0063] Furthermore, in the evaluation step, the output information 228 is evaluated according to a predetermined processing flow that combines multiple processes, thereby enabling stable evaluation. Therefore, the generation of adjustment information 232 that further reduces changes in the output information 228 and the selection of the second trained model 224 can be performed with higher accuracy through stable evaluation. This makes it possible to change the trained model while further suppressing changes in the output information 228. From another perspective, by performing processing according to a predetermined processing flow, it is possible to summarize processing instruction prompts in advance when performing evaluation using a large-scale language model. In other words, compared to sequential processing in which a processing instruction prompt is input and then another processing instruction prompt is input based on the processing result, the number of tokens can be reduced, thereby enabling cost reduction.

[0064] Furthermore, in the evaluation step, the output information 228 is evaluated in accordance with the intention of the input information 226, so that it is possible to more accurately generate adjustment information 232 that further reduces changes in the output information 228 and select the second trained model 224. This makes it possible to change the trained model while further suppressing changes in the output information 228.

[0065] In the above-described embodiment, the first trained model 222 and the second trained model 224 are large-scale language models. However, the present invention is not limited to this. Trained models other than large-scale language models, such as other language models, image classification models, image generation models, object detection models, speech recognition models, and inference models, may also be used. For image classification models, the outputs of the two image classification models may be evaluated using a comparison of accuracy for each class, a comparison of confusion matrices, an ROC curve and an AUC (Area Under the Curve), Cohen's Kappa coefficient, etc., and the input information 226 may be adjusted by adjusting hyperparameters, adjusting a learning rate, data augmentation, etc. For image generation models, the outputs of the two image generation models may be evaluated using an Inception Score, a Frechet Inception Distance, a Perceptual Path Length, a Kernel Inception Distance, visual comparison, etc., and the input information 226 may be adjusted by adjusting hyperparameters, adjusting the balance between the generator and the classifier, etc. For the object detection model, the outputs of the two object detection models may be evaluated using intersection over union, precision and recall, mean average precision, etc., and the input information 226 may be adjusted by adjusting the anchor box size and aspect ratio, adjusting hyperparameters, adjusting the non-maximum suppression threshold, etc. For the speech recognition model, the outputs of the two speech recognition models may be evaluated using word error rate, sentence error rate, character error rate, etc., and the input information 226 may be adjusted by adjusting hyperparameters, removing noise from training data, data augmentation, etc.In the inference model, the outputs of the two inference models may be evaluated using the Mean Absolute Error, Root Mean Squared Error, R-squared (coefficient of determination), Mean Absolute Percentage Error, etc., and the input information 226 may be adjusted by adjusting hyperparameters, selecting and adjusting features, regularizing to prevent overlearning, etc.

[0066] In addition, although the evaluation is configured to be performed using the first trained model 222, it is not limited to this, and it may be configured to use the second trained model 224 or another trained model for evaluation. Furthermore, although the evaluation is configured to be performed using the output information 228 of the first trained model 222 as a standard, it is not limited to this, and it may be configured to perform the evaluation using the output information 228 of the second trained model 224, output information of other trained models or algorithms, information specified by the user U, or the like as a standard.

[0067] Furthermore, in the above-described embodiment, the re-evaluation step is executed, but the present invention is not limited to this, and the re-evaluation may not be executed.

[0068] In addition, in the output step, a notification that it is possible to change from the first trained model 222 to the second trained model 224 is output, but this is not limiting and the configuration may be such that no notification is sent. Furthermore, regardless of whether the above-mentioned notification is sent or not, an instruction to switch from the first trained model 222 to the second trained model 224 may be output, or the switching may be performed automatically.

[0069] Furthermore, the adjustment information 232, which is the input information 226 changed in the adjustment step, is configured to be input only to the second trained model 224, but this is not a limitation, and the input may be input only to the first trained model 222, or to both the first trained model 222 and the second trained model 224. When the adjustment information 232 is input only to the first trained model 222, the input information 226 is changed based on the output information 228 of the second trained model 224, but this is not a limitation. When the adjustment information 232 is input to both the first trained model 222 and the second trained model 224, either the output information 228 of the first trained model 222 or the output information 228 of the second trained model 224 is dynamically set as a reference depending on the content of the output information 228, but this is not a limitation.

[0070] (Variation 1) In the above-described embodiment, as shown in FIG. 5, the evaluation is performed in accordance with a predetermined flow in the evaluation step. However, in this evaluation, the similarity may be evaluated in multiple stages of three or more, such as "high similarity," "moderate similarity," and "dissimilar."

[0071] As an example, Fig. 6 is a flowchart showing an example of the processing flow of the evaluation step by the information processing system 10. The processor 120 reads out the program 220 stored in the storage 124, expands it in the memory 122, and executes it, thereby performing the processing. This processing flow corresponds to the processing performed in steps S114 and S124 in the flowchart in Fig. 4 described above. Note that the same processing as in the above-described embodiment is assigned the same reference numerals, and the description thereof will be omitted.

[0072] After processing step S202, processor 120 determines whether the intention of input information 226 is clear (step S300). If the intention is not clear (step S300: NO), processor 120 proceeds to step S312, which will be described later. On the other hand, if the intention is clear (step S300: YES), processor 120 proceeds to step S206.

[0073] In the process of step S222, if each output matches the intention of the previous determination (step S222: YES), processor 120 determines whether the intentions of each output are of the same type (step S302). On the other hand, if each output does not match the intention of the previous determination (step S222: NO), processor 120 shifts the process to step S232. On the other hand, if the intentions of each output are of the same type (step S302: YES), processor 120 performs a high similarity determination (step S312) and ends this process flow. On the other hand, if the intentions of each output are not of the same type (step S302: NO), processor 120 shifts the process to step S232.

[0074] In the process of step S210, if the intention is to summarize (step S210: YES), processor 120 determines whether each output retains important information in the original information before summarization (step S304). If the original information is retained (step S304: YES), processor 120 proceeds to step S308, which will be described later. On the other hand, if the original information is not retained (step S304: NO), processor 120 proceeds to step S232.

[0075] In the process of step S212, if the intention is an information request (step S212: YES), processor 120 determines whether each output satisfies the request level for the information request of input information 226 (step S306). If the request level is satisfied (step S306: YES), processor 120 proceeds to the process of step S308. On the other hand, if the request level is not satisfied (step S306: NO), processor 120 proceeds to the process of step S232.

[0076] If the processing in step S220 shows that the two images correspond (step S220: YES), processor 120 determines whether the similarity is high (step S308). This similarity determination involves calculating the similarity using a predetermined similarity determination process, and determining that the similarity is high if the similarity is equal to or greater than the predetermined similarity, or medium if the similarity is equal to or less than the predetermined similarity. If the similarity is equal to or greater than the predetermined similarity (step S308: YES), processor 120 determines that the similarity is high (step S312) and ends the processing. On the other hand, if the similarity is equal to or less than the predetermined similarity (step S308: NO), processor 120 determines that the similarity is medium (step S310) and ends the processing.

[0077] According to the above-described modified example, by configuring evaluation in three or more stages, it is possible to generate adjustment information 232 that further reduces changes in the output information 228 and select the second trained model 224 with higher accuracy through stable evaluation. This makes it possible to change the trained model while further suppressing changes in the output information 228. Note that, although the above-described modified example uses multiple stages, the present invention is not limited to this, and a configuration in which evaluation is performed continuously according to the similarity score may also be used.

[0078] (Variation 2) Furthermore, if we look at the above-mentioned information processing system from a different perspective, the problem (purpose) that the information processing system of this embodiment aims to solve can also be seen as "outputting a trained model in accordance with desired specified information."

[0079] If the problem is understood as above, the invention as a means to solve the problem would be, for example, as follows: "An information processing method executed by an information processing device, An acquisition step of acquiring specified predetermined information and output information that is a result of inputting the specified input information into the trained model; an evaluation step of comparing and evaluating the predetermined information and the output information; an adjustment step of changing the input information when the evaluation result indicates dissimilarity; "An information processing method including the following."

[0080] According to the above configuration, the output information of the trained model is compared with predetermined information for evaluation. Specifically, the similarity is evaluated. If the evaluation results in dissimilarity, the input information is changed, thereby making the output information of the trained model similar to the predetermined information. This predetermined information is at least one of output information from another trained model, output information from the same trained model under different versions or conditions, output information via a system using a rule base, etc., and information created or specified by a person such as a user, and is specified as the desired information to be obtained. This makes it possible to obtain output from the trained model that conforms to the specified desired predetermined information.

[0081] In this modification, the output information of the trained model may be evaluated based on the predetermined information. This makes it possible to obtain the output of the trained model that is more in line with the specified desired predetermined information.

[0082] Furthermore, this modified example may include a re-evaluation step of acquiring output information obtained by inputting the input information changed in the adjustment step to the trained model, and comparing and evaluating the output information with the predetermined information. This makes it possible to obtain an output of the trained model that is more in line with the specified desired predetermined information.

[0083] Furthermore, in this modified example, the adjustment step may be executed again if the result of the evaluation in the re-evaluation step is dissimilar. This makes it possible to obtain an output of a trained model that is more in line with the specified desired information.

[0084] Furthermore, this modified example may be configured to "evaluate the output information according to a predetermined processing flow that combines multiple processes." This allows for stable evaluation. Therefore, it is possible to obtain the output of a trained model that is more in line with the specified desired information. From another perspective, by performing processing according to a predetermined processing flow, it is possible to summarize processing instruction prompts in advance when performing evaluation using a large-scale language model. In other words, compared to sequential processing in which a processing instruction prompt is input and then another processing instruction prompt is input based on the processing result, the number of tokens can be reduced, thereby enabling cost reduction.

[0085] Furthermore, in this variation, the evaluation step may be configured to evaluate the output information in accordance with the intent of the input information. This makes it possible to obtain an output that is less likely to deviate from the intent of the desired predetermined information, thereby obtaining an output of a trained model that is more in line with the specified desired predetermined information.

[0086] <Additional Notes> The present embodiment includes the following disclosure.

[0087] (Appendix 1) An information processing method executed by an information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; An information processing method including:

[0088] (Appendix 2) The evaluation step evaluates output information of the second trained model based on output information of the first trained model. 1. The information processing method described in Appendix 1.

[0089] (Appendix 3) a re-evaluation step of acquiring output information obtained by inputting the input information changed in the adjustment step into the second trained model, and comparing and evaluating the output information from the first trained model to which the input information before the change has been input; 1. The information processing method described in Appendix 1.

[0090] (Appendix 4) and executing the adjustment step again when the result of the evaluation by the re-evaluation step is a predetermined result. 1. The information processing method described in Appendix 3.

[0091] (Appendix 5) an output step of outputting, when a result of the evaluation does not become the predetermined result, a message indicating that the first trained model can be changed to the second trained model; 1. The information processing method described in Appendix 1.

[0092] (Appendix 6) the first trained model and the second trained model are language models. 1. The information processing method described in Appendix 1.

[0093] (Appendix 7) In the evaluation step, the output information is evaluated according to a predetermined processing flow that combines a plurality of processes. 1. The information processing method described in Appendix 1.

[0094] (Appendix 8) In the evaluation step, the output information is evaluated in accordance with the intention of the input information. 1. The information processing method described in Appendix 1.

[0095] (Appendix 9) In the information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; A program for executing an information processing method including the steps of:

[0096] (Appendix 10) An information processing system executed by an information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; An information processing system that performs the following:

[0097] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. Furthermore, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0098] 10 Information Processing Systems 12 Server (information processing device) 14 User terminal (information processing device) 220 Programs 222 First trained model 224 Second trained model 226 Input Information 228 Output Information

Claims

1. An information processing method executed by an information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; An information processing method including:

2. The evaluation step evaluates output information of the second trained model based on output information of the first trained model. The information processing method according to claim 1 .

3. a re-evaluation step of acquiring output information obtained by inputting the input information changed in the adjustment step into the second trained model, and comparing and evaluating the output information from the first trained model to which the input information before the change has been input; The information processing method according to claim 1 .

4. and executing the adjustment step again when the result of the evaluation by the re-evaluation step is a predetermined result.

4. The information processing method according to claim 3.

5. an output step of outputting, when a result of the evaluation does not become the predetermined result, a message indicating that the first trained model can be changed to the second trained model; The information processing method according to claim 1 .

6. the first trained model and the second trained model are language models. The information processing method according to claim 1 .

7. In the evaluation step, the output information is evaluated according to a predetermined processing flow that combines a plurality of processes. The information processing method according to claim 1 .

8. In the evaluation step, the output information is evaluated in accordance with the intention of the input information. The information processing method according to claim 1 .

9. In the information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; A program for executing an information processing method including the steps of:

10. An information processing system executed by an information processing device, an acquisition step of acquiring output information that is a result of inputting predetermined input information into a first trained model and a second trained model that is different from the first trained model; an evaluation step of comparing and evaluating the output information; an adjustment step of changing the input information when the result of the evaluation is a predetermined result; An information processing system that performs the following:

Images