Evaluation program, information processing apparatus, and evaluation method

Abstract

To provide an evaluation program, an information processor and an evaluation method for evaluating the accuracy of classification information imparted to document data by using a large-scale language model.SOLUTION: In the information processing system, the information processing apparatus executes a process of acquiring document data and classification information associated with the document data, a process of acquiring explanation information indicating an explanation of the classification information, a process of generating a first directive sentence including a directive for evaluating whether or not the classification information associated with the document data is correct based on the explanation information, and a process of outputting an evaluation result regarding accuracy of the classification information associated with the document data based on a result obtained by inputting the first directive sentence to a large-scale language model stored in a server.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an evaluation program, an information processing device, and an evaluation method. [Background technology]

[0002] In recent years, various large-scale language models have been developed. Large-scale language models are language models trained using a huge amount of text data, and are trained to be able to process a variety of natural languages.

[0003] Non-Patent Document 1 discloses a method for improving the output accuracy of ChatGPT, an example of a large-scale language model. Specifically, Non-Patent Document 1 discloses that the accuracy of the output is improved by inputting the instruction "Let's think step by step" into ChatGPT. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "A spell to make ChatGPT smarter," Nikkei Newspaper, [online], March 24, 2023, [Retrieved June 11, 2024], Internet<URL:https: / / www.nikkei.com / article / DGXZQOUC22BVO0S3A320C2000000 / > Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, document data may be assigned classification information. The assigned classification information is used, for example, for analyzing the document data. When the number of document data is enormous or when many types of classification information are assigned, it becomes difficult to determine whether the correct classification information has been assigned.

[0006] The present disclosure has been made to solve the above-mentioned problems, and in one aspect, its purpose is to provide a technology for evaluating the accuracy of classification information assigned to document data using a large-scale language model. [Means for solving the problem]

[0007] In one example of the present disclosure, there is provided an evaluation program that causes a computer to execute the following processes: acquire document data and classification information associated with the document data; acquire explanatory information that explains the classification information; generate a first instruction statement including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and output an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction statement into a large-scale language model.

[0008] In one example of the present disclosure, the first instruction sentence further includes an instruction to output a reason for evaluation of whether the classification information associated with the document data is correct, and the evaluation result includes the reason for evaluation.

[0009] In one example of the present disclosure, the first instruction sentence further includes an instruction to cause the large-scale language model to output classification information of a correct answer candidate when classification information associated with the document data is incorrect. The evaluation result includes the classification information of the correct answer candidate.

[0010] In one example of the present disclosure, the process of obtaining the description information includes a process of generating a second instruction sentence for generating an explanation of the classification information, and a process of obtaining the result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model as the description information.

[0011] In one example of the present disclosure, the description information includes a sentence that defines a definition of the classification information.

[0012] In one example of the present disclosure, the explanation information includes information that associates other document data different from the document data with correct classification information corresponding to the other document data.

[0013] In one example of the present disclosure, the document data is patent information, and the classification information associated with the document data includes labels indicating elements of the invention.

[0014] In one example of the present disclosure, the classification information includes a label indicating whether the document data is related to other document data.

[0015] In one example of the present disclosure, the classification information associated with the document data includes classification information that has been assigned in advance by a user operation.

[0016] In one example of the present disclosure, the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

[0017] In one example of the present disclosure, the classification information associated with the document data includes classification information assigned by a classifier that is pre-trained to receive document data and output classification information related to the document data.

[0018] In another example of the present disclosure, there is provided an evaluation program that causes a computer to execute the following processes: acquiring document data, generating a first instruction statement for summarizing the document data, acquiring summary data of the document data based on a result obtained by inputting the first instruction statement into a first large-scale language model, generating a second instruction statement including an instruction for evaluating whether the summary data of the document data is correct, and outputting an evaluation result regarding the accuracy of the summary of the document data based on a result obtained by inputting the second instruction statement into a second large-scale language model.

[0019] In another example of the present disclosure, an information processing device is provided. The information processing device includes a control unit for controlling the information processing device. The control unit executes the following processes: acquiring document data and classification information associated with the document data; acquiring explanatory information that explains the classification information; generating a first instruction statement including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction statement into a large-scale language model.

[0020] Another example of the present disclosure provides a computer-implemented evaluation method, comprising the steps of: acquiring document data and classification information associated with the document data; acquiring explanatory information indicating an explanation of the classification information; generating a first instruction sentence including an instruction for evaluating whether the classification information associated with the document data is correct based on the explanatory information; and outputting an evaluation result regarding the accuracy of the classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

[0021] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 illustrates an example of a device configuration of an information processing system. [Figure 2] FIG. 1 is a diagram illustrating a classification evaluation process. [Figure 3] FIG. 1 is a schematic diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a user terminal. [Figure 5]It is a diagram showing an example of the data flow among an information processing apparatus, a user terminal, and a server. [Figure 6] It is a diagram showing an example of the setting screen displayed in step S110. [Figure 7] It is a diagram showing an example of the setting screen displayed in step S112. [Figure 8] It is a diagram showing an example of the instruction text generated in step S122. [Figure 9] It is a diagram showing an example of the evaluation result screen displayed in step S150. [Figure 10] It is a diagram showing a setting screen for accepting a plurality of patent information as evaluation targets. [Figure 11] It is a diagram showing an example of the evaluation result screen when a plurality of patent information is specified as evaluation targets. [Figure 12] It is a diagram for explaining an example of generating explanatory information using a large language model. [Figure 13] It is a diagram for explaining another example of the classification evaluation process. [Figure 14] It is a diagram for explaining an example of assigning classification labels using a large language model. [Figure 15] It is a diagram for explaining an example where the evaluation target is the summary of patent information.

Embodiments for Carrying Out the Invention

[0023] <00,data: Hereinafter, each embodiment according to the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Note that each of the embodiments and each modification example described below may be selectively combined as appropriate.

[0024] <A. Information Processing System 10> First, referring to FIG. 1, the device configuration of the information processing system 10 will be described.FIG. 1 is a diagram showing an example of the device configuration of the information processing system 10.

[0025] 1, the information processing system 10 includes an information processing device 100, a user terminal 200, and a server 300. The information processing device 100, the user terminal 200, and the server 300 are configured to be able to communicate with each other via a network NW (for example, the Internet).

[0026] The information processing device 100 is, for example, a notebook or desktop PC (Personal Computer), a tablet terminal, a smartphone, or any other computer equipped with a communication function. The number of information processing devices 100 constituting the information processing system 10 may be one, or may be two or more. The information processing device 100 is operated, for example, by a company "A."

[0027] The user terminal 200 is, for example, a notebook or desktop PC, a tablet terminal, a smartphone, or any other computer with a communication function. The number of user terminals 200 constituting the information processing system 10 may be one, or may be two or more. The user terminal 200 is owned, for example, by user "A," who is a general user.

[0028] The server 300 is, for example, a notebook or desktop personal computer (PC), a tablet terminal, a smartphone, or any other computer with a communication function. The number of servers 300 constituting the information processing system 10 may be one, or two or more. The server 300 is operated, for example, by a company "B."

[0029] The server 300 stores a large-scale language model 324. The large-scale language model 324 is a language model trained on a huge amount of text data, more than several billion pieces of text, and is trained to be able to process various natural languages. The large-scale language model 324 is also called an LLM (Large Language Model). The large-scale language model 324 is trained so that, when a prompt is input, it generates an output corresponding to the prompt.

[0030] Examples of the large-scale language model 324 include the GPT series such as GPT-3 (Generative Pretrained Transformer) and GPT-4, PaLM (Scaling Language Modeling with Pathways), LLaMA (Large Language Model Meta AI), and known LLM. In addition to the GPT series, various large-scale language models may be used regardless of their specific scale, such as a Transformer-based large-scale language model such as BERT (Bidirectional Encoder Representations from Transformers), a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or an LSTM (Long Short Term Memory).

[0031] For example, company "B" publishes an API (Application Programming Interface) for using the functions of the large-scale language model 324. This allows designers and general users of company "A" to use the functions of the large-scale language model 324 through the API.

[0032] The various processes described in this specification may be implemented in the information processing device 100, the user terminal 200, the server 300, or other computers.

[0033] In addition, in the above description, an example in which the information processing system 10 includes the server 300 has been described. However, the information processing system 10 may not include the server 300. In this case, the information processing system 10 is composed of one or more information processing devices 100 and one or more user terminals 200.

[0034] Furthermore, in the above description, an example in which the large language model 324 is stored in the server 300 has been described. However, the large language model 324 may be stored in the information processing device 100 or may be stored in the user terminal 200.

[0035] <B. Overview> The information processing device 100 provides various functions to the user terminal 200. As an example, the information processing device 100 provides the user terminal 200 with a function for evaluating whether the classification information attached to the document data is accurate.

[0036] "Document data" means data including a text. Examples of document data include technical information such as patent information and papers, and business documents such as securities reports.

[0037] Technical information is, for example, data including a technical idea. A technical idea is a technical means for solving a technical problem. Technical information is, for example, a description of technical means in a character string. Technical information includes, for example, invention information in which invention specific matters are described.

[0038] The technical information is, for example, patent information. The patent information includes, for example, patent documents such as published patent gazettes, patent publications, published patent gazettes, republished patents, and utility model publications. As an example, patent information is composed of bibliographic information, a description, claims, drawings, and an abstract. In addition to text, patent information may also include drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, and the like. The bibliographic information may include, for example, an application number, a publication number, a patent registration number, an application date, a publication date, a registration date, an applicant, a patent owner, a title of the invention, an agent, and an application country. Note that the technical information may also be non-patent literature such as a technical journal.

[0039] The "classification information" mentioned above is information for distinguishing document data. Examples of classification information assigned to document data as patent information include labels for classifying technical fields, labels indicating components of inventions such as invention-specific matters, and labels for classifying applicants or inventors. Examples of classification information assigned to document data other than patent information include labels for classifying company names, labels for classifying business fields, and labels for classifying authors. Classification information is not limited to information that distinguishes document data based on text, but may also be information that distinguishes document data based on drawings, tables, chemical structural formulas, mathematical formulas, gene sequences, bibliographic items, etc.

[0040] As another example, the classification information may be some evaluation result. When a user conducts a prior art search or an invalidity search, the user may evaluate whether or not the patent information is related to other documents. The classification information may be an evaluation label indicating whether or not the patent information is related to other documents.

[0041] The classification information may be specified by a character string, a numeric string, or a symbol string, or by a combination of at least one of letters, numbers, and symbols. The number of classification information assigned to one document data may be one or more.

[0042] The classification information may be used in any way. For example, a user may analyze document data based on the classification information provided. For another example, a user may search for document data based on the classification information provided. For yet another example, a user may use data in which document data and classification information are associated as training data. This allows a prediction model to be generated that receives input document data and outputs classification information corresponding to the document data.

[0043] In the following, patent information will be used as an example of "document data," but "document data" is not limited to patent information. Also, in the following, classification labels will be used as an example of "classification information," but "classification information" is not limited to classification labels.

[0044] A function for evaluating whether the classification label 123B assigned to the patent information 123A is accurate or not will be described with reference to Fig. 2. Fig. 2 is a diagram showing an outline of the evaluation process (hereinafter also referred to as "classification evaluation process").

[0045] First, the information processing device 100 acquires the evaluation target data 123. The source from which the evaluation target data 123 is acquired is arbitrary. As an example, the evaluation target data 123 is acquired from a patent database 124 (see FIG. 3 ) described below that is stored in a storage device within the information processing device 100.

[0046] Patent information 123A and a classification label 123B are associated with each other in the evaluation target data 123. The classification label 123B is assigned in advance to the patent information 123A by, for example, a user operation.

[0047] Next, the information processing device 100 acquires explanation information 126 related to the classification label 123B. In the explanation information 126, for example, the definition of the classification label 123B is specified in an explanatory sentence.

[0048] Next, the information processing device 100 generates a directive 128 to be input to the large-scale language model 324. The directive 128 is, for example, registered in advance as a template in the information processing device 100. The directive 128 includes argument sections 130, 132, and 136.

[0049] The argument section 130 specifies the acquired explanatory information 126. The argument section 132 specifies an instruction for evaluating whether the classification information associated with the patent information is correct. The instruction may be predefined or may be input by the user. The argument section 136 specifies a combination of the acquired patent information 123A and classification label 123B.

[0050] As a result, the information processing device 100 generates an instruction statement 128 for causing an evaluation based on the explanation information 126 as to whether or not the classification label 123B associated with the patent information 123A is correct.

[0051] The generated instruction sentence 128 is input to the large-scale language model 324. When the large-scale language model 324 receives the instruction sentence 128 as input, it generates a response according to the instruction sentence 128. The generated response is output to the information processing device 100.

[0052] The information processing device 100 outputs an evaluation result 230 regarding the accuracy of the classification label 123B associated with the patent information 123A based on the response result obtained from the large-scale language model 324. The accuracy may be indicated by two values, correct and incorrect, or may be indicated by a score indicating the degree of accuracy, or may be indicated by a sentence.

[0053] The evaluation result 230 from the information processing device 100 may be output to any destination. As an example, the output destination is the user terminal 200. The evaluation result 230 output to the user terminal 200 is displayed on the display of the user terminal 200, for example.

[0054] As described above, the information processing apparatus 100 evaluates the accuracy of the classification label 123B associated with the patent information 123A by using the large language model 324. It is novel to use the large language model 324 for such evaluation of the classification label 123B. Further, by providing the explanatory information 126 of the classification label 123B to the large language model 324, the accuracy of the classification evaluation process is improved.

[0055] <C. Hardware Configuration> Next, referring to FIGS. 3 and 4, the hardware configurations of the information processing apparatus 100 and the user terminal 200 shown in FIG. 1 above will be described in order.

[0056] Note that the hardware configuration of the server 300 shown in FIG. 1 is the same as that of the information processing apparatus 100, so the description thereof will not be repeated.

[0057] (C1. Information Processing Apparatus 100) First, referring to FIG. 3, the hardware configuration of the information processing apparatus 100 shown in FIG. 1 will be described. FIG. 3 is a schematic diagram showing an example of the hardware configuration of the information processing apparatus 100.

[0058] The information processing apparatus 100 includes a control device 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a communication interface 104, a display interface 105, an input interface 107, and an auxiliary storage device 120. These components are connected to a bus 110.

[0059] The control device 101 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or a combination thereof.

[0060] The control device 101 controls the operation of the information processing device 100 by executing various programs such as an evaluation program 122 and an operating system. Upon receiving an execution command for one of the various programs, the control device 101 reads the program from the auxiliary storage device 120 or the ROM 102 into the RAM 103. The RAM 103 functions as a working memory and temporarily stores various data required for executing the various programs.

[0061] A LAN (Local Area Network), an antenna, etc. are connected to the communication interface 104. The information processing device 100 exchanges data with external devices via the communication interface 104. The external devices include, for example, a user terminal 200, a server 300, and other communication devices.

[0062] A display 106 is connected to the display interface 105. The display interface 105 sends an image signal for displaying an image to the display 106 in accordance with a command from the control device 101 or the like. The display 106 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or other display device. The display 106 may be configured integrally with the information processing device 100 or may be configured separately from the information processing device 100.

[0063] An input device 108 is connected to the input interface 107. The input device 108 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 108 may be configured integrally with the information processing device 100, or may be configured separately from the information processing device 100.

[0064] The auxiliary storage device 120 is, for example, a hard disk, a flash memory, an SSD (Solid State Drive), or other storage medium. The auxiliary storage device 120 stores the evaluation program 122, the patent database 124, the above-mentioned explanatory information 126, the above-mentioned instruction statement 128, etc. The storage location of these is not limited to the auxiliary storage device 120, but may also be stored in a storage area (for example, a cache memory, etc.) of the control device 101, the ROM 102, the RAM 103, an external device, etc.

[0065] The evaluation program 122 may be provided not as a standalone program but as part of an arbitrary program. In this case, various processes defined in the evaluation program 122 are realized in cooperation with an arbitrary program such as the evaluation program 222 described below. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 122 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 122 may be realized by dedicated hardware. Furthermore, the information processing device 100 may be configured in the form of a so-called cloud service in which at least one server executes part of the processing of the evaluation program 122.

[0066] (C2. User terminal 200) Next, the hardware configuration of the user terminal 200 shown in Fig. 1 will be described with reference to Fig. 4. Fig. 4 is a schematic diagram showing an example of the hardware configuration of the user terminal 200.

[0067] The user terminal 200 includes a control device 201, a ROM 202, a RAM 203, a communication interface 204, a display interface 205, an input interface 207, and an auxiliary storage device 220. These components are connected to a bus 210.

[0068] The control device 201 is configured, for example, by at least one integrated circuit. The integrated circuit may be configured, for example, by at least one CPU, at least one GPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0069] The control device 201 controls the operation of the user terminal 200 by executing various programs such as an evaluation program 222 and an operating system. Upon receiving an execution command for one of the various programs, the control device 201 reads the program from the auxiliary storage device 220 or the ROM 202 into the RAM 203. The RAM 203 functions as a working memory and temporarily stores various data required for executing the program.

[0070] A LAN, an antenna, and the like are connected to the communication interface 204. The user terminal 200 exchanges data with external devices via the communication interface 204. The external devices include, for example, the information processing device 100, the server 300, and other communication devices. The user terminal 200 may be configured to be able to download the evaluation program 222 from the information processing device 100.

[0071] A display 206 is connected to the display interface 205. The display interface 205 sends an image signal for displaying an image to the display 206 in accordance with a command from the control device 201 or the like. The display 206 is, for example, a liquid crystal display, an organic EL display, or other display device. The display 206 may be configured integrally with the user terminal 200 or may be configured separately from the user terminal 200.

[0072] An input device 208 is connected to the input interface 207. The input device 208 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving user operations. The input device 208 may be configured integrally with the user terminal 200, or may be configured separately from the user terminal 200.

[0073] The auxiliary storage device 220 is, for example, a hard disk, a flash memory, an SSD, or other storage medium. The auxiliary storage device 220 stores the evaluation program 222 and the like. The storage location of the evaluation program 222 is not limited to the auxiliary storage device 220, and may be stored in a storage area (for example, a cache memory) of the control device 201, the ROM 202, the RAM 203, an external device (for example, a server), or the like.

[0074] The evaluation program 222 may be provided not as a standalone program but as part of an arbitrary program. In this case, various processes defined in the evaluation program 222 are realized in cooperation with an arbitrary program such as the above-mentioned evaluation program 122. Even a program that does not include some of these modules does not deviate from the spirit of the evaluation program 222 according to this embodiment. Furthermore, some or all of the functions provided by the evaluation program 222 may be realized by dedicated hardware. Furthermore, the user terminal 200 may be configured in the form of a so-called cloud service in which at least one server executes part of the processing of the evaluation program 222.

[0075] <D.データフロー> Next, the operation of the information processing system 10 will be described with reference to Fig. 5 to Fig. 9. Fig. 5 is a diagram showing an example of data flow between the information processing device 100, the user terminal 200, and the server 300.

[0076] (D1. Step S110) First, the process of step S110 shown in Fig. 5 will be described with reference to Fig. 6. Fig. 6 is a diagram showing a setting screen 400A as an example displayed in step S110.

[0077] The setting screen 400A receives various settings required for executing the classification evaluation process. The setting screen 400A includes, for example, input fields 412, 414, and 416.

[0078] The input field 412 receives input related to the above-mentioned explanatory information 126 (see FIG. 2). In the input field 412, for example, a definition related to the classification label 123B to be evaluated is input as the explanatory information 126. The definition is input by, for example, a user operation via the above-mentioned input device 208.

[0079] In the example of FIG. 6, in the input field 412, a definition relating to the classification label "patent analysis", a definition relating to the classification label "trademark analysis", and a definition relating to the classification label "other information analysis" are input.

[0080] The classification label "Patent Analysis" is a classification label assigned to patent information related to patent analysis techniques. The classification label "Trademark Analysis" is a classification label assigned to patent information related to trademark analysis techniques. The classification label "Other Information Analysis" is a classification label assigned to patent information related to analytical techniques for intellectual property rights and technical information other than patents and trademarks.

[0081] The input field 414 accepts input of a rule regarding the method of evaluating the correctness of the classification label 123B. The method of inputting the rule is arbitrary. The rule may be input by a text input operation by the user, or may be input by selecting a specific template from predefined templates.

[0082] The input field 416 accepts input of an output format for the evaluation result of the classification label 123B. The output format may be input by any method. The output format may be input by a text input operation by the user, or by a selection operation in which a specific template is selected from predefined templates.

[0083] (D2. Step S112) Next, the process of step S112 shown in Fig. 5 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a setting screen 400B displayed in step S112.

[0084] The setting screen 400B is displayed, for example, by scrolling the above-mentioned setting screen 400A (see FIG. 6). The setting screen 400B accepts settings of search conditions for searching the evaluation target data 123 from the above-mentioned patent database 124 (see FIG. 3). The setting screen 400B includes, for example, an input field 420, an acquire button 422, an input field 424, an expand button 426, a display field 428, a selection field 430, and an evaluation start button 432.

[0085] The input field 420 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input into the input field 420 include the patent information number (e.g., application number, publication number, or patent number) and the country in which the application was published.

[0086] The input field 424 accepts input of classification evaluation conditions. One example of classification evaluation conditions that can be input is the method of assigning the classification label 123B. The classification label 123B is assigned in advance using various methods, as will be described later. The input field 424 accepts settings regarding the method by which the classification label assigned will be evaluated.

[0087] In addition, an input field 424 accepts settings for evaluation points in the patent information 123A. Evaluation points that can be set include, for example, the abstract, the claims, the title of the invention, the technical field, the background art, the problem that the invention aims to solve, the effects of the invention, the means for solving the problem, and the specification. The evaluation points set in the patent information 123A are used in the classification evaluation process for the classification label.

[0088] In addition, the information processing device 100 may perform the classification evaluation process using the sentences themselves at the evaluation points set in the patent information 123A, or may perform the classification evaluation process using feature quantities extracted from the evaluation points set in the patent information 123A.

[0089] There is no particular method for inputting classification evaluation conditions into the input field 424. As an example, a list of settable classification evaluation conditions is displayed by pressing the expand button 426. The user can reflect the classification evaluation conditions in the input field 424 by performing a selection operation on the list.

[0090] When the acquire button 422 is pressed, the data to be evaluated is reflected as the target patent in a display field 428 based on the information set in the input fields 420 and 424. The user can edit the content reflected in the display field 428 as desired.

[0091] The selection field 430 accepts the selection of the large-scale language model 324 to be used. The large-scale language model 324 selected in the selection field 430 is used in the classification evaluation process.

[0092] When the user presses the evaluation start button 432, the user terminal 200 transmits the information entered on the setting screens 400A and 400B to the information processing device 100.

[0093] (D3. Step S122) Next, in step S122, the information processing device 100 generates a directive 128 to be input to the large-scale language model 324.

[0094] 8 is a diagram showing an example of a directive 128 generated in step S122. The directive 128 shown in FIG. 8 is a specific example of the directive 128 shown in FIG.

[0095] The directive 128 includes argument sections 130, 132, 134, and 136. The information processing device 100 specifies the explanation information 126 input in the above-mentioned input field 412 (see FIG. 6) in the argument section 130. The information processing device 100 also specifies the judgment rule input in the above-mentioned input field 414 (see FIG. 6) in the argument section 132. Furthermore, the information processing device 100 specifies the output format input in the above-mentioned input field 416 (see FIG. 6) in the argument section 134. Furthermore, the information processing device 100 specifies the data to be evaluated input in the above-mentioned display field 428 (see FIG. 7) in the argument section 136.

[0096] Preferably, the instruction sentence 128 includes an instruction 132A. The instruction 132A is an instruction to output the evaluation reason when evaluating whether the classification label 123B associated with the patent information 123A to be evaluated is correct or not. By checking the evaluation reason that is output, the user can more accurately determine whether the classification label 123B associated with the patent information 123A is correct or not.

[0097] The instruction sentence 128 also includes an instruction 132B. The instruction 132B is an instruction to cause the large-scale language model 324 to output a classification label of a correct answer candidate when the classification label 123B associated with the patent information 123A to be evaluated is incorrect. By checking the classification label 123B of the correct answer candidate output from the large-scale language model 324, the user can more accurately determine whether the classification label 123B is correct. This also facilitates the task of correcting the classification label 123B.

[0098] The information processing device 100 transmits the instruction statement 128 generated in step S122 to the server 300.

[0099] (D4. Step S130) Next, in step S130, the server 300 inputs the instruction text 128 received from the information processing apparatus 100 into the large language model 324. As a result, the large language model 324 generates response information corresponding to the instruction text 128. The server 300 transmits the generated response information to the information processing apparatus 100.

[0100] (D5. Steps S140, S150) Next, in step S140, the information processing apparatus 100 generates an evaluation result screen based on the response information received from the server 300. The evaluation result screen is described in a language such as HTML (HyperText Markup Language).

[0101] Next, in step S150, the user terminal 200 displays the evaluation result screen generated by the information processing apparatus 100. FIG. 9 is a diagram showing an example of the evaluation result screen 400C displayed in step S150. The evaluation result screen 400C is displayed, for example, on the display 206 of the user terminal 200.

[0102] The evaluation result screen 400C is output in the output format specified by the above-described argument section 134 (see FIG. 8). In the example of FIG. 9, the evaluation result screen 400C includes a classification label 442 assigned to the patent information to be evaluated, an evaluation result 444 indicating whether the assigned classification label 442 is correct, an evaluation reason 446 serving as the basis for the evaluation result 444, and a classification label 448 of a correct candidate.

[0103] <E <E1. Others> Next, other examples of the above embodiment will be described.

[0104] In the setting screen 400B shown in FIG. 7 described above, only one piece of patent information 123A to be evaluated was specified, but a plurality of pieces of patent information 123A may be specified as evaluation targets.

[0105] FIG. 10 is a diagram showing a setting screen 400D that accepts a plurality of pieces of patent information 123A as evaluation targets.

[0106] The setting screen 400D accepts settings of search conditions for searching the patent information 123A to be evaluated from the above-mentioned patent database 124 (see FIG. 3). By inputting the search conditions, the user can specify the patent information 123A to be used in the classification evaluation process. The setting screen 400D includes, for example, a selection field 452, an input field 454, and a button 456.

[0107] Selection field 452 accepts a selection input of a search type. For example, search types that can be specified in selection field 452 include "full text search," "field search," "command search," "semantic search," and "number search."

[0108] The input field 454 accepts input of search conditions for the patent information 123A to be evaluated. Examples of search conditions that can be input in the input field 454 include the number of the patent information 123A (e.g., application number, publication number, etc.), the country in which the application was published, etc. One or more pieces of patent information 123A can be specified in the input field 454.

[0109] When the button 456 is pressed, a classification evaluation process is executed for the patent information 123A entered in the input field 454.

[0110] 11 is a diagram showing an evaluation result screen 400E as an example when a plurality of patent information 123A are designated as evaluation targets. The evaluation result screen 400E is displayed on the display 206 of the user terminal 200, for example.

[0111] The evaluation result screen 400E includes a display field 460 for displaying information related to the classification evaluation conditions, and a display field 462 for the evaluation results.

[0112] The display column 460 displays information such as that input into the above-described setting screens 400A and 400B (see FIGS. 6 and 7). As an example, the display column 460 displays "ID" (Identification), "Audit Setting Name", "Model", "Number of Acquired Items", "Status", "Acquisition Time", "Audit Target", and "Classification Definition".

[0113] "ID" is an identifier for uniquely identifying the audit evaluation result. The "Audit Setting Name" displays the name of the process to be executed. In the example of FIG. 11, the process name indicating "Classification Evaluation" is displayed in the "Audit Setting Name". "Model" indicates the type of the large language model 324 used during the classification evaluation process and corresponds to the information set in the above-described selection column 430 (see FIG. 7). "Number of Acquired Items" corresponds to the number of patent information 123A to be evaluated. "Status" indicates whether the classification evaluation process has ended normally. "Acquisition Time" indicates the time required from the start to the end of the execution of the classification evaluation process. "Audit Target" corresponds to the information selected in the above-described input column 424 (see FIG. 7). The "Classification Definition" displays the explanatory information 126 set in the above-described input column 412 (see FIG. 6).

[0114] The display column 462 for evaluation results is provided according to the number of patent information 123A to be evaluated. In the example of FIG. 11, the display column 462A for the evaluation result regarding the first patent information and the display column 462B for the evaluation result regarding the second patent information are shown. Since the content displayed in each of the display columns 462A and 462B is as described in FIG. 9, the description thereof will not be repeated.

[0115] <E2. Others> Next, still other examples of the above-described embodiment will be described.

[0116] In the example of FIG. 6 described above, the explanatory information 126 was generated by the user inputting the definition of the classification label 123B in text into the input column 412 of the setting screen 400A. However, the method for generating the explanatory information 126 is not limited to this.

[0117] Next, referring to FIG. 12, another example of the method for generating the explanatory information 126 will be described. FIG. 12 is a diagram for explaining an example of generating the explanatory information 126 using the large language model 324.

[0118] The information processing apparatus 100 generates an instruction sentence 128A (second instruction sentence) for generating an explanation related to the classification label 123B. In the example of FIG. 12, the instruction sentence 128A is defined to generate definitions for the classification labels "Patent Analysis", "Trademark Analysis", and "Other Information Analysis".

[0119] The types of classification labels included in the instruction sentence 128A may be input by the user or by the information processing apparatus 100. As an example, the information processing apparatus 100 includes various classification labels associated with the patent information 123A to be evaluated in the instruction sentence 128A.

[0120] Thereafter, the information processing apparatus 100 acquires, as the explanatory information 126, the result obtained by inputting the instruction sentence 128A to the large language model 324. The acquired explanatory information 126 is sent to the user terminal 200 and reflected in the input field 412 (see FIG. 6) of the above-described setting screen 400A. Thereby, the user can easily generate the explanatory information 126.

[0121] Note that, in the above description, an example of generating the explanatory information 126 using the same large language model 324 as in the classification evaluation process has been described. However, the explanatory information 126 may be generated using another large language model different from the large language model 324.

[0122] ​​​​​​​In the example such as FIG. 2 described above, the explanatory information 126 was directly defined by the explanatory text indicating the definition of the classification label 123B. However, the method of defining the explanatory information 126 is not limited to this. As an example, the explanatory information 126 may be indirectly indicated by correct data showing the correspondence between patent information and classification labels, etc.

[0124] Referring to FIG. 13, more specific processing will be described. FIG. 13 is a diagram for explaining the classification evaluation processing in this example.

[0125] In this example, the information processing apparatus 100 acquires correct data 123X as the explanatory information 126. The correct data 123X includes patent information 123XA and the correct classification label 123XB associated with the patent information 123XA.

[0126] Next, the information processing apparatus 100 acquires an instruction statement 128B to be input to the large language model 324. The instruction statement 128B may be predefined or input by the user. The instruction statement 128B is defined to evaluate the evaluation target data 123 based on the correct data 123X. <E4. Others>

[0127]

[0128] <E4. Others> Next, still other examples of the above embodiment will be described.

[0129] In the example of FIG. 2 described above, the description was made on the premise that the classification label 123B associated with the patent information 123A was assigned by user operation. However, the method of assigning the classification label 123B to the patent information 123A is not limited to this.

[0130] <E4. Others> Hereinafter, referring to FIG. 14, another example of the method for assigning the classification label 123B will be described. FIG. 14 is a diagram for explaining an example of assigning the classification label 123B using the large language model 324.

[0131] The information processing apparatus 100 acquires an instruction sentence 128C for instructing to assign the classification label 123B to the patent information 123A. Next, the information processing apparatus 100 inputs the instruction sentence 128C into the large language model 324 and acquires response information from the large language model 324. Thereby, the relevant classification label 123B is assigned to each of the patent information 123A.

[0132] And in the above description, an example of assigning the classification label 123B using the same large language model 324 as that in the classification evaluation process has been described. However, the classification label 123B may be assigned using another large language model different from the large language model 324.

[0133] As another example, the classification label 123B may be assigned by a classifier. The classifier is pre-generated by a learning process using a learning dataset. In the learning dataset, the correct classification label is associated with each of the learning patent information. The internal parameters of the classifier are pre-optimized by a machine learning process using such a learning dataset. Thereby, when receiving the input of the patent information 123A, the classifier outputs the classification label 123B related to the patent information 123A as an estimation result.

[0134] Various machine learning algorithms can be adopted as the learning method for generating the classifier. As an example, deep learning, support vector machine, etc. are adopted as the machine learning algorithm.

[0135] <E5. Others> Next, still another example of the above embodiment will be described.

[0136] 2 and the like, the information processing device 100 evaluates whether the classification label 123B associated with the patent information 123A is correct or not by using the large-scale language model 324. However, the target of the accuracy evaluation is not limited to the classification label 123B.

[0137] Another example of the evaluation target will be described below with reference to Fig. 15. Fig. 15 is a diagram for explaining an example in which the evaluation target is the abstract of the patent information 123A.

[0138] First, the information processing device 100 acquires the patent information 123A to be evaluated. Next, the information processing device 100 specifies the acquired patent information 123A in a directive 128D for generating a summary. As a result, the information processing device 100 generates a directive 128D for summarizing the patent information 123A.

[0139] A directive 128D specifying patent information 123A is input to large-scale language model 324A. Upon receiving the input of directive 128D, large-scale language model 324A outputs summary data 127 corresponding to directive 128D. The output summary data 127 is output to information processing device 100.

[0140] Next, the information processing device 100 specifies the abstract data 127 for an instruction statement 128E for evaluating whether the abstract is correct or not. As a result, the information processing device 100 generates an instruction statement 128E including an instruction for evaluating whether the abstract data 127 of the patent information 123A is correct or not.

[0141] The instruction statement 128E specifying the summary data 127 is input to a large-scale language model 324B different from the large-scale language model 324A. When the large-scale language model 324B receives the instruction statement 128E, it outputs an evaluation result corresponding to the instruction statement 128E. The evaluation result is output to the information processing device 100. The evaluation result may be indicated by two values, correct or incorrect, or may be indicated by a score indicating the degree of accuracy, or may be indicated as a sentence.

[0142] As described above, the information processing device 100 evaluates the accuracy of the summary data 127 created by the large-scale language model 324A using another large-scale language model 324B.

[0143] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0144] 10 Information processing system, 100 Information processing device, 101 Control device, 102 ROM, 103 RAM, 104 Communication interface, 105 Display interface, 106 Display, 107 Input interface, 108 Input device, 110 Bus, 120 Auxiliary storage device, 122 Evaluation program, 123 Evaluation target data, 123A Patent information, 123B Classification label, 123X Correct answer data, 123XA Patent information, 123XB Classification label, 124 Patent database, 126 Explanation information, 127 Summary data, 128 Instruction, 128A Instruction, 128B Instruction, 128C Instruction, 128D Instruction, 128E Instruction, 130 Argument part, 132 Argument part, 132A Instruction, 132B Instruction, 134 Argument part, 136 Argument section, 200 user terminal, 201 control device, 202 ROM, 203 RAM, 204 communication interface, 205 display interface, 206 display, 207 input interface, 208 input device, 210 bus, 220 auxiliary storage device, 222 evaluation program, 230 evaluation result, 300 server, 324 large-scale language model, 324A large-scale language model, 324B large-scale language model, 400A setting screen, 400B setting screen, 400C evaluation result screen, 400D setting screen, 400E evaluation result screen, 412 input field, 414 input field, 416 input field, 420 input field, 422 acquire button, 424 input field, 426 expand button, 428 display field, 430 selection field, 432 start evaluation button, 442 classification label, 444 evaluation result, 446 Reason for evaluation, 448 classification label, 452 selection field, 454 input field, 456 button, 460 display field, 462 display field, 462A display field, 462B display field.

Claims

1. 1. An evaluation program comprising: The evaluation program is installed on a computer. A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; a process of generating a first instruction sentence including an instruction for evaluating whether classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first instruction sentence into a large-scale language model.

2. the first instruction statement further includes an instruction to output a reason for evaluation of whether classification information associated with the document data is correct, The evaluation program according to claim 1 , wherein the evaluation result includes the reason for the evaluation.

3. the first instruction sentence further includes an instruction to cause the large-scale language model to output classification information of a correct answer candidate when classification information associated with the document data is incorrect; The evaluation program according to claim 1 , wherein the evaluation result includes classification information of the correct answer candidates.

4. The process of acquiring the explanation information includes: a process of generating a second instruction sentence for generating a description of the classification information; The evaluation program according to claim 1 , further comprising a process of acquiring, as the explanation information, a result obtained by inputting the second instruction sentence into the large-scale language model or another large-scale language model.

5. The evaluation program according to claim 1 , wherein the explanatory information includes a sentence that defines a definition of the classification information.

6. 3. The evaluation program according to claim 1, wherein the explanatory information includes information associating other document data different from the document data with correct classification information corresponding to the other document data.

7. the document data is patent information, 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes a label indicating an element of the invention.

8. 3. The evaluation program according to claim 1, wherein the classification information includes a label indicating whether the document data is related to other document data.

9. 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes classification information previously assigned by a user operation.

10. 3. The evaluation program according to claim 1, wherein the classification information associated with the document data includes classification information previously assigned by the large-scale language model or another large-scale language model.

11. the classification information associated with the document data includes classification information assigned by a classifier; 3. The evaluation program according to claim 1, wherein the classifier is trained in advance to receive input of document data and output classification information related to the document data.

12. 1. An evaluation program comprising: The evaluation program is installed on a computer. A process of acquiring document data; a process of generating a first directive for summarizing the document data; obtaining summary data of the document data based on a result obtained by inputting the first directive sentence into a first large-scale language model; generating a second instruction sentence including an instruction for evaluating whether the summary data of the document data is correct; and outputting an evaluation result regarding the accuracy of the summary of the document data based on a result obtained by inputting the second directive sentence into a second large-scale language model.

13. An information processing device, a control unit for controlling the information processing device, The control unit A process of acquiring document data and classification information associated with the document data; A process of acquiring explanation information indicating an explanation of the classification information; a process of generating a first instruction sentence including an instruction for evaluating whether classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on a result obtained by inputting the first directive sentence into a large-scale language model.

14. 1. A computer-implemented evaluation method comprising: obtaining document data and classification information associated with the document data; obtaining explanation information indicating an explanation of the classification information; generating a first instruction sentence including an instruction for evaluating whether classification information associated with the document data is correct based on the explanation information; and outputting an evaluation result regarding the accuracy of classification information associated with the document data based on the result obtained by inputting the first instruction sentence into a large-scale language model.

Images