EVALUATION PROGRAM, INFORMATION PROCESSING APPARATUS, AND EVALUATION METHOD

Abstract

A technique is provided for evaluating the accuracy of classification information assigned to document data using a large-scale language model. [Solution] The evaluation program causes a computer to perform the following processes: a process of acquiring document data and classification information associated with the document data; a process of acquiring explanatory information that explains the classification information; a process of generating a first instruction sentence that includes an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and a process of outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on the result obtained by inputting the first instruction sentence into a large-scale language model.

Description

[Technical field]

[0001] The present disclosure relates to an evaluation program, an information processing device, and an evaluation method. [Background technology]

[0002] In recent years, various large-scale language models have been developed. A large-scale language model is a language model trained on a huge amount of text data, and is trained to be able to process various natural languages.

[0003] Non-Patent Document 1 discloses a method for improving the output accuracy of ChatGPT, which is an example of a large-scale language model. Specifically, Non-Patent Document 1 discloses that the accuracy of the output is improved by inputting the instruction "Let's think step by step" into ChatGPT. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "The spell to make ChatGPT smarter", Nikkei Newspaper, [online], March 24, 2023, [Retrieved June 11, 2024], Internet<URL:https: / / www.nikkei.com / article / DGXZQOUC22BVO0S3A320C2000000 / > Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, classification information may be added to document data. The added classification information is used, for example, for analyzing document data. When the amount of document data is huge or when many types of classification information are added, it is difficult to determine whether correct classification information has been added.

[0006] The present disclosure has been made to solve the problems described above, and an object of one aspect is to provide a technique for evaluating the accuracy of classification information assigned to document data using a large-scale language model. [Means for solving the problem]

[0007] In one example of the present disclosure, an evaluation program is provided, which causes a computer to execute the following processes: acquiring document data and classification information associated with the document data, acquiring explanatory information indicating an explanation of the classification information, generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information, and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

[0008] In one example of the present disclosure, the first instruction sentence further includes an instruction to output an evaluation reason when evaluating whether the classification information associated with the document data is correct or not. The evaluation result includes the evaluation reason.

[0009] In one example of the present disclosure, the first instruction further includes an instruction to cause the large-scale language model to output classification information of a correct answer candidate when classification information associated with the document data is incorrect. The evaluation result includes the classification information of the correct answer candidate.

[0010] In one example of the present disclosure, the process of acquiring the explanatory information includes a process of generating a second instruction sentence for generating an explanation of the classification information, and a process of acquiring a result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model as the explanatory information.

[0011] In one example of the present disclosure, the description information includes a sentence that provides a definition of the classification information.

[0012] In an example of the present disclosure, the explanation information includes information associating other document data different from the document data with correct classification information corresponding to the other document data.

[0013] In one example of the present disclosure, the document data is patent information, and classification information associated with the document data includes labels indicating components of the invention.

[0014] In one example of the present disclosure, the classification information includes a label indicating whether the document data is related to other document data.

[0015] In one example of the present disclosure, the classification information associated with the document data includes classification information that has been assigned in advance by a user operation.

[0016] In one example of the present disclosure, the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

[0017] In one example of the present disclosure, the classification information associated with the document data includes classification information assigned by a classifier that is pre-trained to receive document data and output classification information related to the document data.

[0018] In another example of the present disclosure, an evaluation program is provided, which causes a computer to execute the following processes: acquiring document data, generating a first instruction sentence for summarizing the document data, acquiring summary data of the document data based on a result obtained by inputting the first instruction sentence into a first large-scale language model, generating a second instruction sentence including an instruction for evaluating whether the summary data of the document data is correct, and outputting an evaluation result regarding accuracy of the summary of the document data based on a result obtained by inputting the second instruction sentence into a second large-scale language model.

[0019] In another example of the present disclosure, an information processing device is provided. The information processing device includes a control unit for controlling the information processing device. The control unit executes a process of acquiring document data and classification information associated with the document data, a process of acquiring explanatory information indicating an explanation of the classification information, a process of generating a first instruction sentence including an instruction for evaluating whether or not the classification information associated with the document data is correct based on the explanatory information, and a process of outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

[0020] In another example of the present disclosure, a computer-executed evaluation method is provided, which includes the steps of acquiring document data and classification information associated with the document data, acquiring explanatory information indicating an explanation of the classification information, generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information, and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

[0021] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 illustrates an example of a device configuration of an information processing system. [Diagram 2] FIG. 1 is a diagram illustrating a classification evaluation process. [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a user terminal. [Diagram 5]It is a diagram showing an example of the data flow among an information processing apparatus, a user terminal, and a server. [Figure 6] It is a diagram showing an example of the setting screen displayed in step S110. [Figure 7] It is a diagram showing an example of the setting screen displayed in step S112. [Figure 8] It is a diagram showing an example of the instruction text generated in step S122. [Figure 9] It is a diagram showing an example of the evaluation result screen displayed in step S150. [Figure 10] It is a diagram showing a setting screen for accepting a plurality of patent information as evaluation targets. [Figure 11] It is a diagram showing an example of the evaluation result screen when a plurality of patent information is specified as an evaluation target. [Figure 12] It is a diagram for explaining an example of generating explanatory information using a large language model. [Figure 13] It is a diagram for explaining another example of the classification evaluation process. [Figure 14] It is a diagram for explaining an example of assigning classification labels using a large language model. [Figure 15] It is a diagram for explaining an example where the evaluation target is the abstract of patent information.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, each embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Note that each embodiment and each modification example described below may be selectively combined as appropriate.

[0024] <A. Information Processing System 10> First, with reference to FIG. 1, the device configuration of the information processing system 10 will be described. FIG. 1 is a diagram showing an example of the device configuration of the information processing system 10.

[0025] 1, the information processing system 10 includes an information processing device 100, a user terminal 200, and a server 300. The information processing device 100, the user terminal 200, and the server 300 are configured to be able to communicate with each other via a network NW (for example, the Internet).

[0026] The information processing device 100 is, for example, a notebook or desktop PC (Personal Computer), a tablet terminal, a smartphone, or other computer equipped with a communication function. The number of information processing devices 100 constituting the information processing system 10 may be one, or may be two or more. The information processing device 100 is operated, for example, by a company "A."

[0027] The user terminal 200 is, for example, a notebook or desktop PC, a tablet terminal, a smartphone, or other computer with a communication function. The number of user terminals 200 constituting the information processing system 10 may be one, or may be two or more. The user terminal 200 is owned, for example, by a user "A" who is a general user.

[0028] The server 300 is, for example, a notebook or desktop PC (Personal Computer), a tablet terminal, a smartphone, or other computer equipped with a communication function. The number of servers 300 constituting the information processing system 10 may be one, or two or more. The server 300 is operated, for example, by a company "B."

[0029] The server 300 stores a large-scale language model 324. The large-scale language model 324 is a language model trained with a huge amount of text data of more than several billions, and is trained to be able to process various natural languages. The large-scale language model 324 is also called LLM (Large Language Models). The large-scale language model 324 is trained to generate an output according to an input of a prompt when the input of the prompt is received.

[0030] Examples of the large-scale language model 324 include the GPT series, such as GPT-3 (Generative Pretrained Transformer) and GPT-4, PaLM (Scaling Language Modeling with Pathways), LLaMA (Large Language Model Meta AI), and known LLM. In addition to the GPT series, the large-scale language model may be a Transformer-based large-scale language model, such as BERT (Bidirectional Encoder Representations from Transformers), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), or any other large-scale language model, regardless of the specific scale.

[0031] For example, company "B" publishes an API (Application Programming Interface) for utilizing the functions of the large-scale language model 324. This allows designers and general users of company "A" to utilize the functions of the large-scale language model 324 through the API.

[0032] The various processes described in this specification may be implemented in the information processing device 100, the user terminal 200, the server 300, or other computers.

[0033] Also, in the above description, an example in which the information processing system 10 includes the server 300 has been described. However, the information processing system 10 may not include the server 300. In this case, the information processing system 10 is composed of one or more information processing devices 100 and one or more user terminals 200.

[0034] Furthermore, in the above description, an example in which the large language model 324 is stored in the server 300 has been described. However, the large language model 324 may be stored in the information processing device 100 or may be stored in the user terminal 200.

[0035] <B. Overview> The information processing device 100 provides various functions to the user terminal 200. As an example, the information processing device 100 provides the user terminal 200 with a function for evaluating whether the classification information attached to the document data is accurate.

[0036] "Document data" means data including text. Examples of document data include technical information such as patent information and papers, and business documents such as securities reports.

[0037] Technical information is, for example, data including a technical idea. A technical idea is a technical means for solving a technical problem. Technical information is, for example, something in which technical means are described as a character string. Technical information includes, for example, invention information in which invention-specific matters are described.

[0038] The technical information is, for example, patent information. The patent information includes, for example, patent documents such as published patent gazettes, patent publications, published patent gazettes, republished patents, and utility model publications. As an example, the patent information is composed of bibliographic items, a description, claims, drawings, and an abstract. In addition to text, the patent information may include drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, and the like. The bibliographic items include, for example, an application number, a publication number, a patent registration number, an application date, a publication date, a registration date, an applicant, a patent owner, a name of the invention, an agent, and an application country. The technical information may be a non-patent document such as a technical publication.

[0039] The above-mentioned "classification information" is information for distinguishing document data. Examples of classification information given to document data as patent information include labels for classifying technical fields, labels indicating components of an invention such as invention-specific matters, and labels for classifying applicants or inventors. Examples of classification information given to document data other than patent information include labels for classifying company names, labels for classifying business fields, and labels for classifying authors. Classification information is not limited to information for distinguishing document data based on text, but may also be information for distinguishing document data based on drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, bibliographic items, and the like.

[0040] As another example, the classification information may be some evaluation result. When a user performs a prior art search or an invalidity search, the user may evaluate whether the patent information is related to other documents. The classification information may be an evaluation label indicating whether such patent information is related to other documents.

[0041] The classification information may be defined by a character string, a numeric string, or a symbol string, or may be defined by at least one combination of letters, numbers, and symbols. The number of classification information items assigned to one document data item may be one or more.

[0042] The classification information may be used in any way. As one example, a user analyzes document data based on the classification information provided. As another example, a user searches for document data based on the classification information provided. As yet another example, a user uses data in which document data and classification information are associated as learning data. This makes it possible to generate a prediction model that receives document data and outputs classification information corresponding to the document data.

[0043] In the following, patent information is used as an example of "document data", but "document data" is not limited to patent information. In the following, classification labels are used as an example of "classification information", but "classification information" is not limited to classification labels.

[0044] A function for evaluating whether or not the classification label 123B assigned to the patent information 123A is accurate will be described with reference to Fig. 2. Fig. 2 is a diagram showing an outline of the evaluation process (hereinafter, also referred to as "classification evaluation process").

[0045] First, the information processing device 100 acquires the evaluation target data 123. The evaluation target data 123 may be acquired from any source. As an example, the evaluation target data 123 is acquired from a patent database 124 (see FIG. 3) stored in a storage device in the information processing device 100, which will be described later.

[0046] In the evaluation target data 123, patent information 123A is associated with a classification label 123B. The classification label 123B is, for example, given in advance to the patent information 123A by a user operation.

[0047] Next, the information processing device 100 acquires explanation information 126 related to the classification label 123B. In the explanation information 126, for example, the definition of the classification label 123B is specified in an explanatory sentence.

[0048] Next, the information processing device 100 generates a directive 128 to be input to the large-scale language model 324. The directive 128 is, for example, registered in advance as a template in the information processing device 100. The directive 128 includes argument portions 130, 132, and 136.

[0049] The acquired explanatory information 126 is specified in the argument section 130. An instruction for evaluating whether the classification information associated with the patent information is correct or not is specified in the argument section 132. The instruction may be predefined or may be input by the user. The acquired combination of patent information 123A and classification label 123B is specified in the argument section 136.

[0050] As a result, the information processing device 100 generates an instruction sentence 128 for having the user evaluate, based on the explanation information 126, whether or not the classification label 123B associated with the patent information 123A is correct.

[0051] The generated instruction sentence 128 is input to the large-scale language model 324. Upon receiving the instruction sentence 128, the large-scale language model 324 generates a response according to the instruction sentence 128. The generated response is output to the information processing device 100.

[0052] The information processing device 100 outputs an evaluation result 230 regarding the accuracy of the classification label 123B associated with the patent information 123A based on the answer result obtained from the large-scale language model 324. The accuracy may be indicated by two values, correct and incorrect, or may be indicated by a score indicating the degree of accuracy, or may be indicated by a sentence.

[0053] The evaluation result 230 output by the information processing device 100 may be output to any destination. As an example, the output destination is the user terminal 200. The evaluation result 230 output to the user terminal 200 is displayed on the display of the user terminal 200, for example.

[0054] As described above, the information processing apparatus 100 evaluates the accuracy of the classification label 123B associated with the patent information 123A by using the large language model 324. It is novel to use the large language model 324 for such evaluation of the classification label 123B. Also, by providing the explanatory information 126 of the classification label 123B to the large language model 324, the accuracy of the classification evaluation process is improved.

[0055] <C. Hardware Configuration> Next, with reference to FIGS. 3 and 4, the hardware configurations of the information processing apparatus 100 and the user terminal 200 shown in FIG. 1 above will be described in order.

[0056] Note that the hardware configuration of the server 300 shown in FIG. 1 is the same as that of the information processing apparatus 100, so the description thereof will not be repeated.

[0057] (C1. Information Processing Apparatus 100) First, with reference to FIG. 3, the hardware configuration of the information processing apparatus 100 shown in FIG. 1 will be described. FIG. 3 is a schematic diagram showing an example of the hardware configuration of the information processing apparatus 100.

[0058] The information processing apparatus 100 includes a control device 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a communication interface 104, a display interface 105, an input interface 107, and an auxiliary storage device 120. These components are connected to a bus 110.

[0059] The control device 101 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or a combination thereof.

[0060] The control device 101 controls the operation of the information processing device 100 by executing various programs such as an evaluation program 122 and an operating system. Based on receiving an execution command for various programs, the control device 101 reads the programs from the auxiliary storage device 120 or the ROM 102 to the RAM 103. The RAM 103 functions as a working memory and temporarily stores various data required for executing the various programs.

[0061] A LAN (Local Area Network), an antenna, and the like are connected to the communication interface 104. The information processing device 100 exchanges data with external devices via the communication interface 104. The external devices include, for example, a user terminal 200, a server 300, and other communication devices.

[0062] A display 106 is connected to the display interface 105. The display interface 105 sends an image signal for displaying an image to the display 106 in accordance with a command from the control device 101 or the like. The display 106 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or other display device. The display 106 may be configured integrally with the information processing device 100, or may be configured separately from the information processing device 100.

[0063] An input device 108 is connected to the input interface 107. The input device 108 is, for example, a mouse, a keyboard, a touch panel, or other device capable of accepting a user's operation. Note that the input device 108 may be configured integrally with the information processing device 100, or may be configured separately from the information processing device 100.

[0064] The auxiliary storage device 120 is, for example, a hard disk, a flash memory, an SSD (Solid State Drive), or other storage medium. The auxiliary storage device 120 stores an evaluation program 122, a patent database 124, the above-mentioned explanatory information 126, the above-mentioned instruction statement 128, and the like. The storage location of these is not limited to the auxiliary storage device 120, and may be stored in a storage area (for example, a cache memory, etc.) of the control device 101, the ROM 102, the RAM 103, an external device, and the like.

[0065] The evaluation program 122 may be provided not as a standalone program but as part of an arbitrary program. In this case, various processes defined in the evaluation program 122 are realized in cooperation with an arbitrary program such as the evaluation program 222 described later. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 122 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 122 may be realized by dedicated hardware. Furthermore, the information processing device 100 may be configured in a form similar to a so-called cloud service in which at least one server executes part of the processing of the evaluation program 122.

[0066] (C2. User terminal 200) Next, the hardware configuration of the user terminal 200 shown in Fig. 1 will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of the hardware configuration of the user terminal 200.

[0067] The user terminal 200 includes a control device 201, a ROM 202, a RAM 203, a communication interface 204, a display interface 205, an input interface 207, and an auxiliary storage device 220. These components are connected to a bus 210.

[0068] The control device 201 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one CPU, at least one GPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0069] The control device 201 controls the operation of the user terminal 200 by executing various programs such as an evaluation program 222 and an operating system. Based on receiving an execution command for each program, the control device 201 reads the program from the auxiliary storage device 220 or the ROM 202 to the RAM 203. The RAM 203 functions as a working memory and temporarily stores various data required for executing the program.

[0070] A LAN, an antenna, and the like are connected to the communication interface 204. The user terminal 200 exchanges data with external devices via the communication interface 204. The external devices include, for example, the information processing device 100, the server 300, and other communication devices. The user terminal 200 may be configured to be able to download the evaluation program 222 from the information processing device 100.

[0071] A display 206 is connected to the display interface 205. The display interface 205 sends an image signal for displaying an image to the display 206 in accordance with a command from the control device 201 or the like. The display 206 is, for example, a liquid crystal display, an organic EL display, or other display device. The display 206 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0072] An input device 208 is connected to the input interface 207. The input device 208 is, for example, a mouse, a keyboard, a touch panel, or other device capable of accepting a user's operation. The input device 208 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0073] The auxiliary storage device 220 is, for example, a hard disk, a flash memory, an SSD, or other storage media. The auxiliary storage device 220 stores the evaluation program 222 and the like. The storage location of the evaluation program 222 is not limited to the auxiliary storage device 220, and may be stored in a storage area (for example, a cache memory, etc.) of the control device 201, the ROM 202, the RAM 203, an external device (for example, a server), or the like.

[0074] The evaluation program 222 may be provided not as a standalone program but as part of an arbitrary program. In this case, various processes defined in the evaluation program 222 are realized in cooperation with an arbitrary program such as the above-mentioned evaluation program 122. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 222 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 222 may be realized by dedicated hardware. Furthermore, the user terminal 200 may be configured in a form similar to a so-called cloud service in which at least one server executes part of the processing of the evaluation program 222.

[0075] <D.データフロー> Next, the operation of the information processing system 10 will be described with reference to Fig. 5 to Fig. 9. Fig. 5 is a diagram showing an example of a data flow between the information processing device 100, the user terminal 200, and the server 300.

[0076] (D1. Step S110) First, the process of step S110 shown in Fig. 5 will be described with reference to Fig. 6. Fig. 6 shows a setting screen 400A as an example displayed in step S110.

[0077] The setting screen 400A receives various settings necessary for executing the classification evaluation process. The setting screen 400A includes, for example, input fields 412, 414, and 416.

[0078] The input field 412 receives input related to the above-mentioned explanatory information 126 (see FIG. 2). In the input field 412, for example, a definition related to the classification label 123B to be evaluated is input as the explanatory information 126. The definition is input by, for example, a user operation via the above-mentioned input device 208.

[0079] In the example of FIG. 6, a definition relating to the classification label "patent analysis", a definition relating to the classification label "trademark analysis", and a definition relating to the classification label "other information analysis" are input in the input field 412.

[0080] The classification label "patent analysis" is a classification label assigned to patent information related to patent analysis technology. The classification label "trademark analysis" is a classification label assigned to patent information related to trademark analysis technology. The classification label "other information analysis" is a classification label assigned to patent information related to analysis technology of intellectual property rights and technical information other than patents and trademarks.

[0081] The input field 414 accepts input of a rule regarding a method of evaluating the correctness of the classification label 123B. The method of inputting the rule is arbitrary. The rule may be input by a text input operation by the user, or may be input by selecting a specific template from predefined templates.

[0082] The input field 416 accepts input of an output format related to the evaluation result of the classification label 123B. The input method of the output format is arbitrary. The output format may be input by a text input operation by the user, or may be input by a selection operation of selecting a specific template from predefined templates.

[0083] (D2. Step S112) Next, the process of step S112 shown in Fig. 5 will be described with reference to Fig. 7. Fig. 7 is a diagram showing a setting screen 400B as an example displayed in step S112.

[0084] The setting screen 400B is displayed, for example, by scrolling the above-mentioned setting screen 400A (see FIG. 6). The setting screen 400B accepts settings of search conditions for searching the evaluation target data 123 from the above-mentioned patent database 124 (see FIG. 3), etc. The setting screen 400B includes, for example, an input field 420, an acquisition button 422, an input field 424, an expansion button 426, a display field 428, a selection field 430, and an evaluation start button 432.

[0085] The input field 420 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input in the input field 420 include the number of the patent information (e.g., application number, publication number, or patent number) and the country in which the application was published.

[0086] The input field 424 accepts input of classification evaluation conditions. Examples of classification evaluation conditions that can be input include a method of assigning the classification label 123B. The classification label 123B is assigned in advance by various methods, as described below. The input field 424 accepts settings regarding the method by which the classification label assigned is to be evaluated.

[0087] In addition, an input field 424 accepts settings for evaluation points in the patent information 123A. Examples of evaluation points that can be set include the abstract, claims, title of the invention, technical field, background art, problems that the invention aims to solve, effects of the invention, means for solving the problems, and the description. The evaluation points set in the patent information 123A are used in the classification evaluation process of the classification label.

[0088] In addition, the information processing device 100 may perform the classification evaluation process using the sentences themselves at the evaluation points set in the patent information 123A, or may perform the classification evaluation process using features extracted from the evaluation points set in the patent information 123A.

[0089] The method of inputting the classification evaluation conditions into the input field 424 is arbitrary. As one example, a list of settable classification evaluation conditions is displayed by pressing the expand button 426. The user can reflect the classification evaluation conditions in the input field 424 by performing a selection operation on the list.

[0090] When the acquire button 422 is pressed, the data of the evaluation target is reflected as the target patent in a display field 428 based on the information set in the input field 420 and the input field 424. The user can edit the content reflected in the display field 428 as desired.

[0091] The selection field 430 accepts the selection of the large-scale language model 324 to be used. The large-scale language model 324 selected in the selection field 430 is used in the classification evaluation process.

[0092] When the user presses the evaluation start button 432, the user terminal 200 transmits the information entered on the setting screens 400A and 400B to the information processing device 100.

[0093] (D3. Step S122) Next, in step S122, the information processing device 100 generates a directive 128 to be input to the large-scale language model 324.

[0094] 8 is a diagram showing an example of a directive 128 generated in step S122. The directive 128 shown in FIG. 8 is a specific example of the directive 128 shown in FIG.

[0095] The instruction statement 128 includes argument sections 130, 132, 134, and 136. The information processing device 100 specifies the explanatory information 126 input in the above-mentioned input field 412 (see FIG. 6) in the argument section 130. The information processing device 100 also specifies the judgment rule input in the above-mentioned input field 414 (see FIG. 6) in the argument section 132. Furthermore, the information processing device 100 specifies the output format input in the above-mentioned input field 416 (see FIG. 6) in the argument section 134. Furthermore, the information processing device 100 specifies the data to be evaluated input in the above-mentioned display field 428 (see FIG. 7) in the argument section 136.

[0096] Preferably, the instruction sentence 128 includes an instruction 132A. The instruction 132A is an instruction to output an evaluation reason when evaluating whether the classification label 123B associated with the patent information 123A to be evaluated is correct or not. By checking the evaluation reason that is output, the user can more accurately determine whether the classification label 123B associated with the patent information 123A is correct or not.

[0097] The instruction sentence 128 also includes an instruction 132B. The instruction 132B is an instruction for causing the large-scale language model 324 to output a classification label of a correct answer candidate when the classification label 123B associated with the patent information 123A to be evaluated is incorrect. The user can more accurately determine whether the classification label 123B is correct by checking the classification label 123B of the correct answer candidate output from the large-scale language model 324. Also, the task of correcting the classification label 123B becomes easier.

[0098] The information processing device 100 transmits the instruction statement 128 generated in step S122 to the server 300.

[0099] (D4. Step S130) Next, in step S130, the server 300 inputs the instruction text 128 received from the information processing apparatus 100 into the large language model 324. As a result, the large language model 324 generates response information corresponding to the instruction text 128. The server 300 transmits the generated response information to the information processing apparatus 100.

[0100] (D5. Steps S140, S150) Next, in step S140, the information processing apparatus 100 generates an evaluation result screen based on the response information received from the server 300. The evaluation result screen is described in a language such as HTML (HyperText Markup Language), for example.

[0101] Next, in step S150, the user terminal 200 displays the evaluation result screen generated by the information processing apparatus 100. FIG. 9 is a diagram showing an example of the evaluation result screen 400C displayed in step S150. The evaluation result screen 400C is displayed on the display 206 of the user terminal 200, for example.

[0102] The evaluation result screen 400C is output in the output format specified by the above-described argument section 134 (see FIG. 8). In the example of FIG. 9, the evaluation result screen 400C includes a classification label 442 assigned to the patent information to be evaluated, an evaluation result 444 indicating whether the assigned classification label 442 is correct, an evaluation reason 446 serving as the basis for the evaluation result 444, and a correct answer candidate classification label 448.

[0103] <E1. Others> Next, other examples of the above embodiment will be described.

[0104] In the setting screen 400B shown in FIG. 7 described above, only one piece of patent information 123A to be evaluated was specified, but a plurality of pieces of patent information 123A may be specified as evaluation targets.

[0105] FIG. 10 is a diagram showing a setting screen 400D that accepts a plurality of pieces of patent information 123A as evaluation targets.

[0106] The setting screen 400D accepts settings of search conditions for searching the patent information 123A to be evaluated from the above-mentioned patent database 124 (see FIG. 3). The user can specify the patent information 123A to be used in the classification evaluation process by inputting the search conditions. The setting screen 400D includes, for example, a selection field 452, an input field 454, and a button 456.

[0107] Selection field 452 accepts a selection input of a search type. For example, search types that can be specified in selection field 452 include "full-text search," "field search," "command search," "semantic search," and "number search."

[0108] The input field 454 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input in the input field 454 include the number of the patent information 123A (e.g., application number, publication number, etc.), the publication country of the application, etc. In the input field 454, one or more pieces of patent information 123A can be specified.

[0109] When the button 456 is pressed, a classification evaluation process is executed for the patent information 123A entered in the input field 454.

[0110] 11 is a diagram showing an evaluation result screen 400E as an example when a plurality of patent information 123A are designated as evaluation targets. The evaluation result screen 400E is displayed on the display 206 of the user terminal 200, for example.

[0111] The evaluation result screen 400E includes a display field 460 for displaying information related to the classification evaluation conditions, and a display field 462 for the evaluation results.

[0112] The display column 460 displays information such as that input into the above-described setting screens 400A and 400B (see FIGS. 6 and 7). As an example, the display column 460 displays "ID" (Identification), "Audit Setting Name", "Model", "Number of Acquired Items", "Status", "Acquisition Time", "Audit Target", and "Classification Definition".

[0113] "ID" is an identifier for uniquely identifying the audit evaluation result. The "Audit Setting Name" displays the name of the process to be executed. In the example of FIG. 11, the process name indicating "Classification Evaluation" is displayed in the "Audit Setting Name". "Model" indicates the type of the large language model 324 used during the classification evaluation process and corresponds to the information set in the above-described selection column 430 (see FIG. 7). "Number of Acquired Items" corresponds to the number of patent information 123A to be evaluated. "Status" indicates whether the classification evaluation process has ended normally. "Acquisition Time" indicates the time required from the start to the end of the execution of the classification evaluation process. "Audit Target" corresponds to the information selected in the above-described input column 424 (see FIG. 7). The "Classification Definition" displays the explanatory information 126 set in the above-described input column 412 (see FIG. 6).

[0114] The display column 462 for the evaluation result is provided according to the number of patent information 123A to be evaluated. In the example of FIG. 11, the display column 462A for the evaluation result regarding the first patent information and the display column 462B for the evaluation result regarding the second patent information are shown. Since the content displayed in each of the display columns 462A and 462B is as described in FIG. 9, the description thereof will not be repeated.

[0115] <E2. Others> Next, still other examples of the above-described embodiment will be described.

[0116] In the example of FIG. 6 described above, the explanatory information 126 was generated by the user entering the definition of the classification label 123B in text into the input column 412 of the setting screen 400A. However, the method for generating the explanatory information 126 is not limited to this.

[0117] Next, with reference to FIG. 12, another example of the method for generating the explanatory information 126 will be described. FIG. 12 is a diagram for explaining an example of generating the explanatory information 126 using the large language model 324.

[0118] The information processing apparatus 100 generates an instruction sentence 128A (second instruction sentence) for generating an explanation related to the classification label 123B. In the example of FIG. 12, the instruction sentence 128A is defined to generate definitions for the classification label "Patent Analysis", the classification label "Trademark Analysis", and the classification label "Other Information Analysis".

[0119] The types of classification labels included in the instruction sentence 128A may be input by the user or may be input by the information processing apparatus 100. As an example, the information processing apparatus 100 includes various classification labels associated with the patent information 123A to be evaluated in the instruction sentence 128A.

[0120] Thereafter, the information processing apparatus 100 acquires, as the explanatory information 126, the result obtained by inputting the instruction sentence 128A into the large language model 324. The acquired explanatory information 126 is sent to the user terminal 200 and reflected in the input field 412 (see FIG. 6) of the above-described setting screen 400A. Thereby, the user can easily generate the explanatory information 126.

[0121] Note that, in the above description, an example of generating the explanatory information 126 using the same large language model 324 as in the classification evaluation process has been described. However, the explanatory information 126 may be generated using another large language model different from the large language model 324.

[0122] <E3. Others> Next, still another example of the above-described embodiment will be described.

[0123] In the example such as FIG. 2 above, the explanatory information 126 was directly defined by the explanatory text indicating the definition of the classification label 123B. However, the method of defining the explanatory information 126 is not limited to this. As an example, the explanatory information 126 may be indirectly indicated by correct data showing the correspondence between patent information and classification labels, etc.

[0124] Referring to FIG. 13, more specific processing will be described. FIG. 13 is a diagram for explaining the classification evaluation processing in this example.

[0125] In this example, the information processing apparatus 100 acquires correct data 123X as the explanatory information 126. The correct data 123X includes patent information 123XA and the correct classification label 123XB associated with the patent information 123XA.

[0126] Next, the information processing apparatus 100 acquires an instruction sentence 128B to be input to the large language model 324. The instruction sentence 128B may be predefined or input by the user. The instruction sentence 128B is defined to evaluate the evaluation target data 123 based on the correct data 123X.

[0127] As described above, the information processing apparatus 100 evaluates the accuracy of the classification label 123B associated with the patent information 123A to be evaluated based on the past classification tendency. As a result, the user does not need to define the definition of the classification label 123B.

[0128] <E4. Others> Next, still other examples of the above embodiment will be described.

[0129] In the example of FIG. 2 above, the description was made on the premise that the classification label 123B associated with the patent information 123A was given by a user operation. However, the method of assigning the classification label 123B to the patent information 123A is not limited to this.

[0130] Hereinafter, with reference to FIG. 14, another example of a method for assigning classification label 123B will be described. FIG. 14 is a diagram for explaining an example of assigning classification label 123B using large language model 324.

[0131] The information processing apparatus 100 acquires an instruction sentence 128C for instructing to assign classification label 123B to patent information 123A. Next, the information processing apparatus 100 inputs the instruction sentence 128C into the large language model 324 and acquires response information from the large language model 324. Thereby, the relevant classification label 123B is assigned to each of the patent information 123A.

[0132] Note that, in the above description, an example of assigning classification label 123B using the same large language model 324 as in the classification evaluation process has been described. However, classification label 123B may be assigned using another large language model different from large language model 324.

[0133] As another example, classification label 123B may be assigned by a classifier. The classifier is pre-generated by a learning process using a learning dataset. In the learning dataset, a correct classification label is associated with each of the learning patent information. The internal parameters of the classifier are pre-optimized by a machine learning process using such a learning dataset. Thereby, when the classifier receives the input of the patent information 123A, it outputs the classification label 123B related to the patent information 123A as an estimation result.

[0134] Various machine learning algorithms can be adopted as the learning method for generating the classifier. As an example, deep learning, support vector machine, etc. are adopted as the machine learning algorithm.

[0135] <E5. Others> Next, still another example of the above embodiment will be described.

[0136] In the above-mentioned example of Fig. 2, the information processing device 100 evaluates whether the classification label 123B associated with the patent information 123A is correct or not by using the large-scale language model 324. However, the target of the accuracy evaluation is not limited to the classification label 123B.

[0137] Another example of the evaluation target will be described below with reference to Fig. 15. Fig. 15 is a diagram for explaining an example in which the evaluation target is the abstract of the patent information 123A.

[0138] First, the information processing device 100 acquires the patent information 123A to be evaluated. Next, the information processing device 100 specifies the acquired patent information 123A for the instruction statement 128D for generating a summary. As a result, the information processing device 100 generates the instruction statement 128D for summarizing the patent information 123A.

[0139] A command statement 128D specifying patent information 123A is input to large-scale language model 324A. Upon receiving command statement 128D, large-scale language model 324A outputs summary data 127 corresponding to command statement 128D. The output summary data 127 is output to information processing device 100.

[0140] Next, the information processing device 100 specifies the abstract data 127 for the instruction statement 128E for evaluating whether the abstract is correct or not. As a result, the information processing device 100 generates the instruction statement 128E including an instruction for evaluating whether the abstract data 127 of the patent information 123A is correct or not.

[0141] The instruction sentence 128E in which the summary data 127 is specified is input to a large-scale language model 324B different from the large-scale language model 324A. When the large-scale language model 324B receives the instruction sentence 128E, it outputs an evaluation result according to the instruction sentence 128E. The output evaluation result is output to the information processing device 100. The evaluation result may be indicated by two values, correct and incorrect, or may be indicated by a score indicating the degree of accuracy, or may be indicated by a sentence.

[0142] In this manner, the information processing device 100 evaluates the accuracy of the summary data 127 created by the large-scale language model 324A using the other large-scale language model 324B.

[0143] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0144] 10 Information processing system, 100 Information processing device, 101 Control device, 102 ROM, 103 RAM, 104 Communication interface, 105 Display interface, 106 Display, 107 Input interface, 108 Input device, 110 Bus, 120 Auxiliary storage device, 122 Evaluation program, 123 Evaluation target data, 123A Patent information, 123B Classification label, 123X Correct answer data, 123XA Patent information, 123XB Classification label, 124 Patent database, 126 Explanation information, 127 Summary data, 128 Instruction, 128A Instruction, 128B Instruction, 128C Instruction, 128D Instruction, 128E Instruction, 130 Argument part, 132 Argument part, 132A Instruction, 132B Instruction, 134 Argument part, 136 Argument section, 200 user terminal, 201 control device, 202 ROM, 203 RAM, 204 communication interface, 205 display interface, 206 display, 207 input interface, 208 input device, 210 bus, 220 auxiliary storage device, 222 evaluation program, 230 evaluation result, 300 server, 324 large-scale language model, 324A large-scale language model, 324B large-scale language model, 400A setting screen, 400B setting screen, 400C evaluation result screen, 400D setting screen, 400E evaluation result screen, 412 input field, 414 input field, 416 input field, 420 input field, 422 obtain button, 424 input field, 426 expand button, 428 display field, 430 selection field, 432 evaluation start button, 442 classification label, 444 evaluation result, 446 Evaluation reason, 448 classification label, 452 selection field, 454 input field, 456 button, 460 display field, 462 display field, 462A display field, 462B display field.

Claims

1. 1. An evaluation program comprising: The evaluation program is configured to: A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; a process of generating a first instruction sentence including a combination of the document data and the classification information, and an instruction for evaluating whether or not the classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

2. the first instruction statement further includes an instruction to output a reason for evaluation of whether or not the classification information associated with the document data is correct, The evaluation program according to claim 1 , wherein the evaluation result includes a reason for the evaluation.

3. The first instruction sentence further includes an instruction to output classification information of a correct answer candidate to the large-scale language model when classification information associated with the document data is incorrect, The evaluation program according to claim 1 , wherein the evaluation result includes classification information of the correct answer candidates.

4. The process of acquiring the explanatory information includes: generating a second instruction for generating a description of the classification information; The evaluation program according to claim 1 , further comprising a process of acquiring, as the explanatory information, a result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model.

5. The evaluation program according to claim 1 , wherein the explanatory information includes a sentence that specifies a definition of the classification information.

6. The document data is patent information, 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes a label indicating an element of the invention.

7. 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes classification information that is assigned in advance by a user operation.

8. 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

9. the classification information associated with the document data includes classification information assigned by a classifier; 3. The evaluation program according to claim 1, wherein the classifier is trained in advance to receive document data and output classification information related to the document data.

10. An evaluation program comprising: The evaluation program is configured to: A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; A process of generating a first instruction sentence including an instruction for evaluating whether classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model. The explanation information includes information associating other document data different from the document data with correct classification information corresponding to the other document data.

11. An evaluation program comprising: The evaluation program is configured to: A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; A process of generating a first instruction sentence including an instruction for evaluating whether classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model. The classification information includes a label indicating whether the document data is related to other document data.

12. 1. An evaluation program comprising: The evaluation program is configured to: A process of acquiring document data; A process of generating a first instruction for summarizing the document data; obtaining summary data of the document data based on a result obtained by inputting the first instruction sentence into a first large-scale language model; generating a second instruction sentence including an instruction for evaluating whether or not the summary data of the document data is correct; and outputting an evaluation result regarding the accuracy of a summary of the document data based on a result obtained by inputting the second instruction sentence into a second large-scale language model.

13. An information processing device, A control unit for controlling the information processing device, The control unit is A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; a process of generating a first instruction sentence including a combination of the document data and the classification information, and an instruction for evaluating whether or not the classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data, based on a result obtained by inputting the first instruction sentence into a large-scale language model.

14. 1. A computer-implemented evaluation method comprising: obtaining document data and classification information associated with the document data; obtaining explanation information indicating an explanation of the classification information; generating a first instruction sentence including a combination of the document data and the classification information, and an instruction for evaluating whether the classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

Images