Evaluation program, information processing device, and evaluation method

Abstract

This evaluation program causes a computer to execute: processing for acquiring document data and classification information associated with the document data; processing for acquiring explanatory information describing an explanation of the classification information; processing for generating a first instruction sentence including an instruction to evaluate, on the basis of the explanatory information, whether the classification information associated with the document data is correct; and processing for outputting an evaluation result regarding accuracy of the classification information associated with the document data, on the basis of a result of inputting the first instruction sentence into a large language model.

Description

Evaluation program, information processing device, and evaluation method

[0001] The present disclosure relates to an evaluation program, an information processing device, and an evaluation method.

[0002] In recent years, various large-scale language models have been developed. Large-scale language models are language models trained using a huge amount of text data, and are trained to be able to process a variety of natural languages.

[0003] Non-Patent Document 1 discloses a method for improving the output accuracy of ChatGPT, an example of a large-scale language model. Specifically, Non-Patent Document 1 discloses that the accuracy of the output is improved by inputting the instruction "Let's think step by step" into ChatGPT.

[0004] "The Spell to Make ChatGPT Smarter," Nikkei Newspaper, [online], March 24, 2023, [Retrieved June 11, 2024], Internet <URL: https: / / www.nikkei.com / article / DGXZQOUC22BVO0S3A320C2000000 / >

[0005] Incidentally, document data may be assigned classification information. The assigned classification information is used, for example, for analyzing the document data. When the number of document data is enormous or when many types of classification information are assigned, it becomes difficult to determine whether the correct classification information has been assigned.

[0006] The present disclosure has been made to solve the above-mentioned problems, and in one aspect, its purpose is to provide a technology for evaluating the accuracy of classification information assigned to document data using a large-scale language model.

[0007] In one example of the present disclosure, there is provided an evaluation program that causes a computer to execute the following processes: acquire document data and classification information associated with the document data; acquire explanatory information that explains the classification information; generate a first instruction statement including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and output an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction statement into a large-scale language model.

[0008] In one example of the present disclosure, the first instruction sentence further includes an instruction to output a reason for evaluation of whether the classification information associated with the document data is correct. The evaluation result includes the reason for evaluation.

[0009] In one example of the present disclosure, the first instruction sentence further includes an instruction to cause the large-scale language model to output classification information of a correct answer candidate when classification information associated with the document data is incorrect. The evaluation result includes the classification information of the correct answer candidate.

[0010] In one example of the present disclosure, the process of obtaining the description information includes a process of generating a second instruction sentence for generating a description of the classification information, and a process of obtaining the result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model as the description information.

[0011] In one example of the present disclosure, the description information includes a sentence that defines a definition of the classification information.

[0012] In one example of the present disclosure, the explanation information includes information that associates other document data different from the document data with correct classification information corresponding to the other document data.

[0013] In one example of the present disclosure, the document data is patent information, and the classification information associated with the document data includes labels indicating elements of the invention.

[0014] In one example of the present disclosure, the classification information includes a label indicating whether the document data is related to other document data.

[0015] In one example of the present disclosure, the classification information associated with the document data includes classification information that has been assigned in advance by a user operation.

[0016] In one example of the present disclosure, the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

[0017] In one example of the present disclosure, the classification information associated with the document data includes classification information assigned by a classifier that is pre-trained to receive document data and output classification information related to the document data.

[0018] In another example of the present disclosure, there is provided an evaluation program that causes a computer to execute the following processes: acquiring document data, generating a first instruction statement for summarizing the document data, acquiring summary data of the document data based on a result obtained by inputting the first instruction statement into a first large-scale language model, generating a second instruction statement including an instruction for evaluating whether the summary data of the document data is correct, and outputting an evaluation result regarding the accuracy of the summary of the document data based on a result obtained by inputting the second instruction statement into a second large-scale language model.

[0019] In another example of the present disclosure, an information processing device is provided, the information processing device including a control unit for controlling the information processing device, wherein the control unit executes a process of acquiring document data and classification information associated with the document data, a process of acquiring explanatory information indicating an explanation of the classification information, a process of generating a first instruction statement including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information, and a process of outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction statement into a large-scale language model.

[0020] Another example of the present disclosure provides a computer-implemented evaluation method, comprising the steps of: acquiring document data and classification information associated with the document data; acquiring explanatory information indicating an explanation of the classification information; generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

[0021] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings.

[0022] 1 is a diagram showing an example of the device configuration of an information processing system. FIG. 1 is a diagram showing an example of the hardware configuration of an information processing device, which is a diagram showing an overview of classification evaluation processing. FIG. 1 is a diagram showing an example of the hardware configuration of a user terminal. FIG. 2 is a diagram showing an example of data flow between an information processing device, a user terminal, and a server. FIG. 3 is a diagram showing an example of a setting screen displayed in step S110. FIG. 4 is a diagram showing an example of a setting screen displayed in step S112. FIG. 5 is a diagram showing an example of an instruction statement generated in step S122. FIG. 6 is a diagram showing an example of an evaluation result screen displayed in step S150. FIG. 7 is a diagram showing a setting screen for accepting multiple patent information as evaluation targets. FIG. 8 is a diagram showing an example of an evaluation result screen when multiple patent information are designated as evaluation targets. FIG. 9 is a diagram explaining an example of generating explanatory information using a large-scale language model. FIG. 10 is a diagram explaining another example of classification evaluation processing. FIG. 11 is a diagram explaining an example of assigning classification labels using a large-scale language model. FIG. 12 is a diagram explaining an example where the evaluation target is a summary of patent information.

[0023] Hereinafter, each embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated. Note that each embodiment and each modified example described below may be selectively combined as appropriate.

[0024] <A. Information Processing System 10> First, the device configuration of the information processing system 10 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the device configuration of the information processing system 10.

[0025] 1, the information processing system 10 includes an information processing device 100, a user terminal 200, and a server 300. The information processing device 100, the user terminal 200, and the server 300 are configured to be able to communicate with each other via a network NW (for example, the Internet).

[0026] The information processing device 100 is, for example, a notebook or desktop personal computer (PC), a tablet terminal, a smartphone, or any other computer equipped with a communication function. The number of information processing devices 100 constituting the information processing system 10 may be one, or two or more. The information processing device 100 is operated, for example, by a company "A."

[0027] The user terminal 200 is, for example, a notebook or desktop PC, a tablet terminal, a smartphone, or any other computer with a communication function. The number of user terminals 200 constituting the information processing system 10 may be one, or may be two or more. The user terminal 200 is owned, for example, by user "A," who is a general user.

[0028] The server 300 is, for example, a notebook or desktop personal computer (PC), a tablet terminal, a smartphone, or any other computer with a communication function. The number of servers 300 constituting the information processing system 10 may be one, or two or more. The server 300 is operated, for example, by a company "B."

[0029] The server 300 stores a large-scale language model 324. The large-scale language model 324 is a language model trained on a huge amount of text data, exceeding billions of pieces, and is trained to be able to process various natural languages. The large-scale language model 324 is also called an LLM (Large Language Model). When a prompt is input, the large-scale language model 324 is trained to generate an output corresponding to the prompt.

[0030] Examples of the large-scale language model 324 include the GPT series such as GPT-3 (Generative Pretrained Transformer) and GPT-4, PaLM (Scaling Language Modeling with Pathways), LLaMA (Large Language Model Meta AI), and the well-known LLM. In addition to the GPT series, various large-scale language models may be used regardless of their specific scale, such as a Transformer-based large-scale language model such as BERT (Bidirectional Encoder Representations from Transformers), a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or an LSTM (Long Short Term Memory).

[0031] For example, company "B" publishes an API (Application Programming Interface) for using the functions of the large-scale language model 324. This allows designers and general users of company "A" to use the functions of the large-scale language model 324 through the API.

[0032] The various processes described in this specification may be implemented in the information processing device 100, the user terminal 200, the server 300, or other computers.

[0033] Furthermore, although the above description has been given of an example in which the information processing system 10 includes the server 300, the information processing system 10 does not have to include the server 300. In this case, the information processing system 10 is configured with one or more information processing devices 100 and one or more user terminals 200.

[0034] Furthermore, although the above describes an example in which the large-scale language model 324 is stored in the server 300, the large-scale language model 324 may be stored in the information processing device 100 or the user terminal 200.

[0035] <B. Overview> The information processing device 100 provides various functions to the user terminal 200. As an example, the information processing device 100 provides the user terminal 200 with a function for evaluating whether classification information assigned to document data is accurate.

[0036] "Document data" refers to data that includes text. Examples of document data include technical information such as patent information and papers, and business documents such as securities reports.

[0037] Technical information is, for example, data containing technical ideas. Technical ideas are technical means for solving technical problems. Technical information is, for example, a description of technical means in the form of text. Technical information includes, for example, invention information that describes invention-specific matters.

[0038] The technical information is, for example, patent information. The patent information includes, for example, patent documents such as published patent gazettes, patent publications, published patent gazettes, republished patents, and utility model publications. As an example, patent information is composed of bibliographic information, a description, claims, drawings, and an abstract. In addition to text, patent information may also include drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, and the like. The bibliographic information may include, for example, an application number, a publication number, a patent registration number, an application date, a publication date, a registration date, an applicant, a patent owner, a title of the invention, an agent, and an application country. Note that the technical information may also be non-patent literature such as a technical journal.

[0039] The "classification information" mentioned above is information for distinguishing document data. Examples of classification information assigned to document data as patent information include labels for classifying technical fields, labels indicating components of inventions such as invention-specific matters, and labels for classifying applicants or inventors. Examples of classification information assigned to document data other than patent information include labels for classifying company names, labels for classifying business fields, and labels for classifying authors. Classification information is not limited to information that distinguishes document data based on text, but may also be information that distinguishes document data based on drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, bibliographic items, etc.

[0040] As another example, the classification information may be some evaluation result. When a user conducts a prior art search or an invalidity search, the user may evaluate whether or not the patent information is related to other documents. The classification information may be an evaluation label indicating whether or not the patent information is related to other documents.

[0041] The classification information may be defined by a character string, a numeric string, or a symbol string, or by a combination of at least one of letters, numbers, and symbols. The number of classification information assigned to one document data may be one or more.

[0042] The classification information may be used in any way. For example, a user may analyze document data based on the classification information provided. For another example, a user may search for document data based on the classification information provided. For yet another example, a user may use data in which document data and classification information are associated as training data. This allows a prediction model to be generated that receives input document data and outputs classification information corresponding to the document data.

[0043] In the following, patent information will be used as an example of "document data," but "document data" is not limited to patent information. Also, in the following, classification labels will be used as an example of "classification information," but "classification information" is not limited to classification labels.

[0044] The function for evaluating whether the classification label 123B assigned to the patent information 123A is accurate will be described with reference to Fig. 2. Fig. 2 is a diagram showing an outline of the evaluation process (hereinafter also referred to as "classification evaluation process").

[0045] First, the information processing device 100 acquires the evaluation target data 123. The source from which the evaluation target data 123 is acquired is arbitrary. As an example, the evaluation target data 123 is acquired from a patent database 124 (see FIG. 3 ) described below that is stored in a storage device within the information processing device 100.

[0046] In the evaluation target data 123, patent information 123A and a classification label 123B are associated with each other. The classification label 123B is assigned in advance to the patent information 123A by, for example, a user operation.

[0047] Next, the information processing apparatus 100 acquires explanation information 126 relating to the classification label 123B. In the explanation information 126, for example, the definition of the classification label 123B is specified in an explanatory sentence.

[0048] Next, the information processing device 100 generates a directive 128 to be input to the large-scale language model 324. The directive 128 is, for example, registered in advance as a template in the information processing device 100. The directive 128 includes argument sections 130, 132, and 136.

[0049] The argument section 130 specifies the acquired explanatory information 126. The argument section 132 specifies an instruction for evaluating whether the classification information associated with the patent information is correct. The instruction may be predefined or may be input by the user. The argument section 136 specifies a combination of the acquired patent information 123A and classification label 123B.

[0050] As a result, the information processing device 100 generates an instruction statement 128 for causing an evaluation based on the explanation information 126 as to whether or not the classification label 123B associated with the patent information 123A is correct.

[0051] The generated instruction statement 128 is input to the large-scale language model 324. Upon receiving the instruction statement 128, the large-scale language model 324 generates a response according to the instruction statement 128. The generated response is output to the information processing device 100.

[0052] The information processing device 100 outputs an evaluation result 230 regarding the accuracy of the classification label 123B associated with the patent information 123A based on the response result obtained from the large-scale language model 324. The accuracy may be indicated by two values, correct and incorrect, or may be indicated by a score indicating the degree of accuracy, or may be indicated by a sentence.

[0053] The evaluation result 230 from the information processing device 100 may be output to any destination. As an example, the output destination is the user terminal 200. The evaluation result 230 output to the user terminal 200 is displayed on the display of the user terminal 200, for example.

[0054] As described above, the information processing device 100 evaluates the accuracy of the classification label 123B associated with the patent information 123A by using the large-scale language model 324. The use of the large-scale language model 324 for such evaluation of the classification label 123B is novel. Furthermore, providing the explanation information 126 of the classification label 123B to the large-scale language model 324 improves the accuracy of the classification evaluation process.

[0055] <C. Hardware Configuration> Next, the hardware configurations of the information processing device 100 and the user terminal 200 shown in FIG. 1 will be described in order with reference to FIGS. 3 and 4. FIG.

[0056] The hardware configuration of server 300 shown in FIG. 1 is similar to that of information processing device 100, and therefore description thereof will not be repeated.

[0057] (C1. Information Processing Apparatus 100) First, the hardware configuration of the information processing apparatus 100 shown in Fig. 1 will be described with reference to Fig. 3. Fig. 3 is a schematic diagram showing an example of the hardware configuration of the information processing apparatus 100.

[0058] The information processing device 100 includes a control device 101, a read only memory (ROM) 102, a random access memory (RAM) 103, a communication interface 104, a display interface 105, an input interface 107, and an auxiliary storage device 120. These components are connected to a bus 110.

[0059] The control device 101 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or a combination thereof.

[0060] The control device 101 controls the operation of the information processing device 100 by executing various programs such as an evaluation program 122 and an operating system. Upon receiving an execution command for one of the various programs, the control device 101 reads the program from the auxiliary storage device 120 or the ROM 102 into the RAM 103. The RAM 103 functions as a working memory and temporarily stores various data required for executing the various programs.

[0061] A LAN (Local Area Network), an antenna, etc. are connected to the communication interface 104. The information processing device 100 exchanges data with external devices via the communication interface 104. The external devices include, for example, a user terminal 200, a server 300, and other communication devices.

[0062] A display 106 is connected to the display interface 105. The display interface 105 sends an image signal for displaying an image to the display 106 in accordance with a command from the control device 101 or the like. The display 106 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or other display device. The display 106 may be configured integrally with the information processing device 100 or may be configured separately from the information processing device 100.

[0063] An input device 108 is connected to the input interface 107. The input device 108 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 108 may be configured integrally with the information processing device 100 or may be configured separately from the information processing device 100.

[0064] The auxiliary storage device 120 is, for example, a hard disk, a flash memory, an SSD (Solid State Drive), or other storage medium. The auxiliary storage device 120 stores the evaluation program 122, the patent database 124, the above-mentioned explanatory information 126, the above-mentioned directive 128, etc. The storage location of these is not limited to the auxiliary storage device 120, and may be stored in a storage area (e.g., cache memory, etc.) of the control device 101, the ROM 102, the RAM 103, an external device, etc.

[0065] The evaluation program 122 may be provided not as a standalone program but as part of an arbitrary program. In this case, various processes defined in the evaluation program 122 are realized in cooperation with an arbitrary program such as the evaluation program 222 described below. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 122 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 122 may be realized by dedicated hardware. Furthermore, the information processing device 100 may be configured in the form of a so-called cloud service in which at least one server executes part of the processing of the evaluation program 122.

[0066] (C2. User Terminal 200) Next, the hardware configuration of the user terminal 200 shown in Fig. 1 will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of the hardware configuration of the user terminal 200.

[0067] The user terminal 200 includes a control device 201, a ROM 202, a RAM 203, a communication interface 204, a display interface 205, an input interface 207, and an auxiliary storage device 220. These components are connected to a bus 210.

[0068] The control device 201 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one CPU, at least one GPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0069] The control device 201 controls the operation of the user terminal 200 by executing various programs such as an evaluation program 222 and an operating system. Upon receiving an execution command for one of the programs, the control device 201 reads the program from the auxiliary storage device 220 or the ROM 202 into the RAM 203. The RAM 203 functions as a working memory and temporarily stores various data required for executing the program.

[0070] A LAN, an antenna, and the like are connected to the communication interface 204. The user terminal 200 exchanges data with external devices via the communication interface 204. The external devices include, for example, the information processing device 100, the server 300, and other communication devices. The user terminal 200 may be configured to be able to download an evaluation program 222 from the information processing device 100.

[0071] A display 206 is connected to the display interface 205. The display interface 205 sends an image signal for displaying an image to the display 206 in accordance with a command from the control device 201 or the like. The display 206 is, for example, a liquid crystal display, an organic EL display, or other display device. The display 206 may be configured integrally with the user terminal 200 or may be configured separately from the user terminal 200.

[0072] An input device 208 is connected to the input interface 207. The input device 208 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 208 may be configured integrally with the user terminal 200 or may be configured separately from the user terminal 200.

[0073] The auxiliary storage device 220 is, for example, a hard disk, a flash memory, an SSD, or other storage medium. The auxiliary storage device 220 stores the evaluation program 222 and the like. The storage location of the evaluation program 222 is not limited to the auxiliary storage device 220, and may be stored in a storage area of ​​the control device 201 (for example, a cache memory), the ROM 202, the RAM 203, an external device (for example, a server), or the like.

[0074] The evaluation program 222 may be provided not as a standalone program but as part of an arbitrary program. In this case, the various processes defined in the evaluation program 222 are realized in cooperation with an arbitrary program such as the above-mentioned evaluation program 122. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 222 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 222 may be realized by dedicated hardware. Furthermore, the user terminal 200 may be configured in the form of a so-called cloud service in which at least one server executes part of the processing of the evaluation program 222.

[0075] <D. Data Flow> Next, the operation of the information processing system 10 will be described with reference to Figures 5 to 9. Figure 5 is a diagram showing an example of data flow between the information processing device 100, the user terminal 200, and the server 300.

[0076] (D1. Step S110) First, the process of step S110 shown in Fig. 5 will be described with reference to Fig. 6. Fig. 6 is a diagram showing a setting screen 400A as an example displayed in step S110.

[0077] The setting screen 400A receives various settings required for executing the classification evaluation process. The setting screen 400A includes, for example, input fields 412, 414, and 416.

[0078] The input field 412 receives input related to the above-described explanatory information 126 (see FIG. 2 ). In the input field 412, for example, a definition related to the classification label 123B to be evaluated is input as the explanatory information 126. The definition is input by, for example, a user operation via the above-described input device 208.

[0079] In the example of FIG. 6, a definition relating to the classification label "patent analysis," a definition relating to the classification label "trademark analysis," and a definition relating to the classification label "other information analysis" are entered in the input field 412.

[0080] The classification label "Patent Analysis" is a classification label assigned to patent information related to patent analysis techniques. The classification label "Trademark Analysis" is a classification label assigned to patent information related to trademark analysis techniques. The classification label "Other Information Analysis" is a classification label assigned to patent information related to analytical techniques for intellectual property rights and technical information other than patents and trademarks.

[0081] The input field 414 accepts input of a rule regarding the method of evaluating the correctness of the classification label 123B. The method of inputting the rule is arbitrary. The rule may be input by a text input operation by the user, or may be input by selecting a specific template from predefined templates.

[0082] The input field 416 accepts input of an output format for the evaluation result of the classification label 123B. The output format may be input by any method. The output format may be input by a text input operation by the user, or by a selection operation in which a specific template is selected from predefined templates.

[0083] (D2. Step S112) Next, the process of step S112 shown in Fig. 5 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a setting screen 400B displayed in step S112.

[0084] The setting screen 400B is displayed, for example, by scrolling the setting screen 400A (see FIG. 6 ). The setting screen 400B accepts settings of search conditions for searching the evaluation target data 123 from the patent database 124 (see FIG. 3 ). The setting screen 400B includes, for example, an input field 420, an acquire button 422, an input field 424, an expand button 426, a display field 428, a selection field 430, and an evaluation start button 432.

[0085] The input field 420 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input into the input field 420 include the patent information number (e.g., application number, publication number, or patent number) and the country in which the application was published.

[0086] The input field 424 accepts input of classification evaluation conditions. One example of a classification evaluation condition that can be input is the method of assigning the classification label 123B. The classification label 123B is assigned in advance using various methods, as will be described later. The input field 424 accepts settings regarding the method by which the classification label assigned will be evaluated.

[0087] In addition, an input field 424 accepts settings for evaluation points in the patent information 123A. Examples of evaluation points that can be set include the abstract, claims, title of the invention, technical field, background art, problems that the invention aims to solve, effects of the invention, means for solving the problems, and description. The evaluation points set in the patent information 123A are used in the classification evaluation process for the classification label.

[0088] In addition, the information processing device 100 may perform the classification evaluation process using the sentences themselves at the evaluation points set in the patent information 123A, or may perform the classification evaluation process using feature quantities extracted from the evaluation points set in the patent information 123A.

[0089] There is no particular method for inputting the classification evaluation conditions into the input field 424. For example, pressing the expand button 426 displays a list of configurable classification evaluation conditions. The user can reflect the classification evaluation conditions in the input field 424 by performing a selection operation on the list.

[0090] When the acquire button 422 is pressed, the data of the evaluation target is reflected as the target patent in a display field 428 based on the information set in the input fields 420 and 424. The user can edit the content reflected in the display field 428 as desired.

[0091] The selection field 430 accepts the selection of the large-scale language model 324 to be used. The large-scale language model 324 selected in the selection field 430 is used in the classification evaluation process.

[0092] When the user presses the evaluation start button 432, the user terminal 200 transmits the information entered on the setting screens 400A and 400B to the information processing device 100.

[0093] (D3. Step S122) Next, in step S122, the information processing apparatus 100 generates a directive 128 to be input to the large-scale language model 324.

[0094] 8 is a diagram showing an example of a directive 128 generated in step S122. The directive 128 shown in FIG. 8 is a specific example of the directive 128 shown in FIG.

[0095] The directive 128 includes argument sections 130, 132, 134, and 136. The information processing device 100 specifies the explanation information 126 input in the above-mentioned input field 412 (see FIG. 6 ) in the argument section 130. The information processing device 100 also specifies the judgment rule input in the above-mentioned input field 414 (see FIG. 6 ) in the argument section 132. Furthermore, the information processing device 100 specifies the output format input in the above-mentioned input field 416 (see FIG. 6 ) in the argument section 134. Furthermore, the information processing device 100 specifies the data to be evaluated input in the above-mentioned display field 428 (see FIG. 7 ) in the argument section 136.

[0096] Preferably, the instruction 128 includes an instruction 132A. The instruction 132A is an instruction for outputting an evaluation reason when evaluating whether the classification label 123B associated with the patent information 123A to be evaluated is correct. By checking the evaluation reason that is output, the user can more accurately determine whether the classification label 123B associated with the patent information 123A is correct.

[0097] The instruction sentence 128 also includes an instruction 132B. The instruction 132B is an instruction for causing the large-scale language model 324 to output a classification label of a correct answer candidate when the classification label 123B associated with the patent information 123A to be evaluated is incorrect. By checking the classification label 123B of the correct answer candidate output from the large-scale language model 324, the user can more accurately determine whether the classification label 123B is correct. This also facilitates the task of correcting the classification label 123B.

[0098] The information processing device 100 transmits the instruction statement 128 generated in step S122 to the server 300.

[0099] (D4. Step S130) Next, in step S130, the server 300 inputs the instruction statement 128 received from the information processing device 100 to the large-scale language model 324. As a result, the large-scale language model 324 generates answer information according to the instruction statement 128. The server 300 transmits the generated answer information to the information processing device 100.

[0100] (D5. Steps S140 and S150) Next, in step S140, information processing device 100 generates an evaluation result screen based on the response information received from server 300. The evaluation result screen is written in a language such as HTML (HyperText Markup Language).

[0101] Next, in step S150, the user terminal 200 displays the evaluation result screen generated by the information processing device 100. Fig. 9 is a diagram showing an example of the evaluation result screen 400C displayed in step S150. The evaluation result screen 400C is displayed on the display 206 of the user terminal 200, for example.

[0102] The evaluation result screen 400C is output in the output format specified in the above-mentioned argument section 134 (see Fig. 8 ). In the example of Fig. 9 , the evaluation result screen 400C includes a classification label 442 assigned to the patent information to be evaluated, an evaluation result 444 indicating whether the assigned classification label 442 is correct or not, an evaluation reason 446 that is the basis for the evaluation result 444, and a classification label 448 of a correct answer candidate.

[0103] <E1. Others> Next, other examples of the above embodiment will be described.

[0104] In the setting screen 400B shown in FIG. 7 described above, only one piece of patent information 123A to be evaluated is designated, but multiple pieces of patent information 123A may be designated as the subject of evaluation.

[0105] FIG. 10 is a diagram showing a setting screen 400D that accepts a plurality of patent information 123A as evaluation targets.

[0106] The setting screen 400D accepts settings of search conditions for searching the patent information 123A to be evaluated from the patent database 124 (see FIG. 3). By inputting the search conditions, the user can specify the patent information 123A to be used in the classification evaluation process. The setting screen 400D includes, for example, a selection field 452, an input field 454, and a button 456.

[0107] The selection field 452 accepts a selection input of a search type. For example, search types that can be specified in the selection field 452 include "full-text search," "field search," "command search," "semantic search," and "number search."

[0108] The input field 454 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input in the input field 454 include the number of the patent information 123A (e.g., application number, publication number, etc.), the country in which the application was published, etc. One or more pieces of patent information 123A can be specified in the input field 454.

[0109] When the button 456 is pressed, a classification evaluation process is executed for the patent information 123A entered in the input field 454.

[0110] 11 is a diagram showing an example of an evaluation result screen 400E when a plurality of patent information 123A are designated as evaluation targets. The evaluation result screen 400E is displayed on the display 206 of the user terminal 200, for example.

[0111] The evaluation result screen 400E includes a display field 460 for displaying information related to the classification evaluation conditions, and a display field 462 for the evaluation results.

[0112] The display field 460 displays information entered in the setting screens 400A and 400B (see FIGS. 6 and 7) described above. As an example, the display field 460 displays an "ID" (Identification), a "Review setting name", a "Model", the "Number of acquisitions", a "Status", an "Acquisition time", a "Review target", and a "Classification definition".

[0113] "ID" is an identifier for uniquely identifying the peer review evaluation results. "Peer review setting name" displays the name of the process to be executed. In the example of Figure 11, the process name indicating "classification evaluation" is displayed in "Peer review setting name." "Model" indicates the type of large-scale language model 324 used during the classification evaluation process, and corresponds to the information set in the selection field 430 (see Figure 7) described above. "Number of acquisitions" corresponds to the number of patent information 123A to be evaluated. "Status" indicates whether the classification evaluation process was completed successfully. "Acquisition time" indicates the time required from the start to the end of the classification evaluation process. "Peer review target" corresponds to the information selected in the input field 424 (see Figure 7) described above. "Classification definition" displays the explanatory information 126 set in the input field 412 (see Figure 6) described above.

[0114] The evaluation result display column 462 is provided according to the number of patent information 123A to be evaluated. In the example of Fig. 11, a display column 462A for the evaluation result regarding the first patent information and a display column 462B for the evaluation result regarding the second patent information are shown. The contents displayed in each of the display columns 462A and 462B are the same as those described in Fig. 9, and therefore the description thereof will not be repeated.

[0115] <E2. Others> Next, still another example of the above embodiment will be described.

[0116] 6, the user inputs the definition of the classification label 123B in text form into the input field 412 of the setting screen 400A, thereby generating the description information 126. However, the method for generating the description information 126 is not limited to this.

[0117] Another example of a method for generating the description information 126 will be described below with reference to Fig. 12. Fig. 12 is a diagram for explaining an example of generating the description information 126 using the large-scale language model 324.

[0118] The information processing device 100 generates an instruction 128A (second instruction) for generating an explanation related to the classification label 123B. In the example of Fig. 12, the instruction 128A is specified to generate definitions for the classification label "patent analysis," the classification label "trademark analysis," and the classification label "other information analysis."

[0119] The types of classification labels to be included in the instruction statement 128A may be input by the user or may be input by the information processing device 100. As an example, the information processing device 100 includes, in the instruction statement 128A, various classification labels associated with the patent information 123A to be evaluated.

[0120] Thereafter, the information processing device 100 acquires the result obtained by inputting the instruction statement 128A into the large-scale language model 324 as the explanatory information 126. The acquired explanatory information 126 is sent to the user terminal 200 and reflected in the input field 412 (see FIG. 6 ) of the setting screen 400A described above. This allows the user to easily generate the explanatory information 126.

[0121] In the above, an example has been described in which the explanatory information 126 is generated using the same large-scale language model 324 as used during the classification evaluation process, but the explanatory information 126 may also be generated using a large-scale language model other than the large-scale language model 324.

[0122] <E3. Others> Next, still another example of the above embodiment will be described.

[0123] 2 and other examples, the explanatory information 126 is directly defined by an explanatory sentence that defines the classification label 123B. However, the method of defining the explanatory information 126 is not limited to this. As an example, the explanatory information 126 may be indirectly indicated by correct answer data that indicates the correspondence between the patent information and the classification label.

[0124] A more specific process will be described with reference to Fig. 13. Fig. 13 is a diagram for explaining the classification evaluation process in this example.

[0125] In this example, the information processing device 100 acquires correct answer data 123X as the explanation information 126. The correct answer data 123X includes patent information 123XA and a correct classification label 123XB associated with the patent information 123XA.

[0126] Next, the information processing device 100 acquires an instruction statement 128B to be input to the large-scale language model 324. The instruction statement 128B may be defined in advance or may be input by a user. The instruction statement 128B specifies that the evaluation target data 123 should be evaluated based on the correct answer data 123X.

[0127] As a result, the information processing device 100 evaluates the accuracy of the classification label 123B associated with the patent information 123A to be evaluated based on past classification trends. This eliminates the need for the user to specify the definition of the classification label 123B.

[0128] <E4. Others> Next, still another example of the above embodiment will be described.

[0129] 2, the explanation has been given on the assumption that the classification label 123B associated with the patent information 123A is assigned by a user operation. However, the method of assigning the classification label 123B to the patent information 123A is not limited to this.

[0130] Another example of a method for assigning classification labels 123B will be described below with reference to Fig. 14. Fig. 14 is a diagram for explaining an example of assigning classification labels 123B using a large-scale language model 324.

[0131] The information processing device 100 acquires an instruction statement 128C for instructing that a classification label 123B be assigned to the patent information 123A. Next, the information processing device 100 inputs the instruction statement 128C into the large-scale language model 324 and acquires response information from the large-scale language model 324. As a result, the related classification label 123B is assigned to each piece of patent information 123A.

[0132] In the above, an example has been described in which the classification label 123B is assigned using the same large-scale language model 324 as used in the classification evaluation process, but the classification label 123B may also be assigned using a large-scale language model other than the large-scale language model 324.

[0133] As another example, the classification label 123B may be assigned by a classifier. The classifier is generated in advance by a learning process using a training dataset. In the training dataset, a correct classification label is associated with each piece of training patent information. The internal parameters of the classifier are optimized in advance by a machine learning process using such a training dataset. As a result, when the classifier receives patent information 123A as input, it outputs the classification label 123B associated with the patent information 123A as an estimated result.

[0134] Various machine learning algorithms can be used as a learning method for generating a classifier, such as deep learning and support vector machines.

[0135] <E5. Others> Next, still another example of the above embodiment will be described.

[0136] 2 and the like, the information processing device 100 evaluates whether the classification label 123B associated with the patent information 123A is correct or not by using the large-scale language model 324. However, the target of the accuracy evaluation is not limited to the classification label 123B.

[0137] Another example of the evaluation target will be described below with reference to Fig. 15. Fig. 15 is a diagram for explaining an example in which the evaluation target is an abstract of the patent information 123A.

[0138] First, the information processing device 100 acquires the patent information 123A to be evaluated. Next, the information processing device 100 specifies the acquired patent information 123A in a directive 128D for generating a summary. As a result, the information processing device 100 generates a directive 128D for summarizing the patent information 123A.

[0139] The directive 128D specifying the patent information 123A is input to the large-scale language model 324A. Upon receiving the input of the directive 128D, the large-scale language model 324A outputs summary data 127 corresponding to the directive 128D. The output summary data 127 is output to the information processing device 100.

[0140] Next, the information processing device 100 specifies the abstract data 127 for the instruction statement 128E for evaluating whether the abstract is correct or not, thereby generating the instruction statement 128E including an instruction for evaluating whether the abstract data 127 of the patent information 123A is correct or not.

[0141] The instruction statement 128E specifying the summary data 127 is input to a large-scale language model 324B that is different from the large-scale language model 324A. When the large-scale language model 324B receives the instruction statement 128E, it outputs an evaluation result corresponding to the instruction statement 128E. The output evaluation result is output to the information processing device 100. The evaluation result may be represented by two values, correct or incorrect, or may be represented by a score indicating the degree of accuracy, or may be represented as a sentence.

[0142] As described above, the information processing device 100 evaluates the accuracy of the summary data 127 created by the large-scale language model 324A using the other large-scale language model 324B.

[0143] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0144] 10 Information processing system, 100 Information processing device, 101 Control device, 102 ROM, 103 RAM, 104 Communication interface, 105 Display interface, 106 Display, 107 Input interface, 108 Input device, 110 Bus, 120 Auxiliary storage device, 122 Evaluation program, 123 Evaluation target data, 123A Patent information, 123B Classification label, 123X Correct answer data, 123XA Patent information, 123XB Classification label, 124 Patent database, 126 Explanation information, 127 Summary data, 128 Instruction statement, 128A Instruction statement, 128B Instruction statement, 128C Instruction statement, 128D Instruction statement, 128E Instruction statement, 130 Argument section, 132 Argument section, 132A Instruction, 132B Instruction, 134 Argument section, 136 Argument section, 200 User terminal, 201 Control device, 202 ROM, 203 RAM, 204 Communication interface, 205 Display interface, 206 Display, 207 Input interface, 208 Input device, 210 Bus, 220 Auxiliary storage device, 222 Evaluation program, 230 Evaluation result, 300 Server, 324 Large-scale language model, 324A Large-scale language model, 324B Large-scale language model, 400A Setting screen, 400B Setting screen, 400C Evaluation result screen, 400D Setting screen, 400E Evaluation result screen, 412 Input field, 414 Input field, 416 Input field, 420 Input field, 422 Acquire button, 424 Input field, 426 Expand button, 428 Display field, 430 Selection field, 432 Evaluation start button, 442 Classification label, 444 Evaluation result, 446 Reason for evaluation, 448 classification label, 452 selection field, 454 input field, 456 button, 460 display field, 462 display field, 462A display field, 462B display field.

Claims

1. An evaluation program that causes a computer to execute the following processes: a process of acquiring document data and classification information associated with the document data; a process of acquiring explanatory information that explains the classification information; a process of generating a first instruction sentence that includes an instruction to evaluate whether the classification information associated with the document data is correct based on the explanatory information; and a process of outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on the result obtained by inputting the first instruction sentence into a large-scale language model.

2. The evaluation program of claim 1, wherein the first instruction sentence further includes an instruction to output a reason for evaluation when evaluating whether the classification information associated with the document data is correct, and the evaluation result includes the reason for evaluation.

3. The evaluation program described in claim 1 or 2, wherein the first instruction sentence further includes an instruction to cause the large-scale language model to output classification information of a correct answer candidate when the classification information associated with the document data is incorrect, and the evaluation result includes the classification information of the correct answer candidate.

4. An evaluation program as described in claim 1 or 2, wherein the process of acquiring the explanatory information includes a process of generating a second instruction sentence for generating an explanation of the classification information, and a process of acquiring the result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model as the explanatory information.

5. The evaluation program according to claim 1 or 2, wherein the explanatory information includes a sentence that defines the classification information.

6. An evaluation program as described in claim 1 or 2, wherein the explanatory information includes information associating other document data different from the document data with correct classification information corresponding to the other document data.

7. The evaluation program according to claim 1 or 2, wherein the document data is patent information, and the classification information associated with the document data includes labels indicating elements of the invention.

8. The evaluation program according to claim 1 or 2, wherein the classification information includes a label indicating whether the document data is related to other document data.

9. The evaluation program according to claim 1 or 2, wherein the classification information associated with the document data includes classification information previously assigned by a user operation.

10. The evaluation program according to claim 1 or 2, wherein the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

11. An evaluation program as described in claim 1 or 2, wherein the classification information associated with the document data includes classification information assigned by a classifier, and the classifier is pre-trained to receive document data as input and output classification information related to the document data.

12. An evaluation program that causes a computer to execute the following processes: a process of acquiring document data; a process of generating a first instruction statement for summarizing the document data; a process of acquiring summary data of the document data based on the result obtained by inputting the first instruction statement into a first large-scale language model; a process of generating a second instruction statement including an instruction for evaluating whether the summary data of the document data is correct; and a process of outputting an evaluation result regarding the accuracy of the summary of the document data based on the result obtained by inputting the second instruction statement into a second large-scale language model.

13. An information processing device comprising a control unit for controlling the information processing device, wherein the control unit executes the following processes: a process of acquiring document data and classification information associated with the document data; a process of acquiring explanatory information indicating an explanation of the classification information; a process of generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and a process of outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on the result obtained by inputting the first instruction sentence into a large-scale language model.

14. An evaluation method executed by a computer, comprising the steps of: acquiring document data and classification information associated with the document data; acquiring explanatory information indicating an explanation of the classification information; generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on the result obtained by inputting the first instruction sentence into a large-scale language model.

Images