Program, method, information processing apparatus, and system

Abstract

To provide a program, a method, an information processing apparatus, and a system configured to prepare large amounts of teacher data for a deep learning model that performs processing for patents.SOLUTION: A method includes: a first patent extraction step of receiving population information related to a population including a plurality of patent documents, and extracting a first patent document out of the patent documents included in the received population; a first processing step of acquiring a first output result which is output in accordance with inputting, to a large language model, a prompt including the first patent document extracted in the first patent extraction step; a second patent extraction step of extracting a second patent document out of the patent documents included in the received population; and a second processing step of acquiring a second output result which is output in accordance with inputting the second patent document extracted in the second patent document extraction step, to a learning model trained with teacher data which is information based on the first output result acquired in the first processing step.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] Techniques for supporting patent work are known. Patent Document 1 discloses a technology for assisting in the understanding of claims, which identifies and presents key elements in claims, enabling readers to focus on reading the explanation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-096200 Summary of the Invention [Problem to be solved by the invention]

[0004] When processing patents using a deep learning model, there is a problem in that a large amount of training data needs to be prepared. Therefore, the present disclosure has been made to solve the above problem, and its purpose is to provide a technology that applies suitable processing to multiple patent documents without preparing a large amount of training data. [Means for solving the problem]

[0005] A program to be executed by a computer having a processor and a memory unit, wherein the processor executes the following steps: a population reception step of receiving population information regarding a population including a plurality of patent documents; a first patent extraction step of extracting a first patent document from the plurality of patent documents included in the population received in the population reception step; a first processing step of obtaining a first output result that is output in response to inputting a prompt including the first patent document extracted in the first patent extraction step into a large-scale language model; a second patent extraction step of extracting a second patent document from the plurality of patent documents included in the population received in the population reception step; and a second processing step of obtaining a second output result that is output in response to inputting the second patent document extracted in the second patent extraction step into a learning model trained using information based on the first output result obtained in the first processing step as training data. [Effects of the Invention]

[0006] According to the present disclosure, suitable processing can be applied to multiple patent documents without preparing a large amount of training data. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of the system 1. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the server 10. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a user terminal 20. [Figure 4] FIG. 10 is a diagram showing the data structure of a user table 1012. [Figure 5] FIG. 10 is a diagram showing the data structure of a document table 1013. [Figure 6] FIG. 10 is a diagram showing the data structure of a population table 1014. [Figure 7] FIG. 10 is a diagram showing the data structure of a first support table 1015. [Figure 8] FIG. 10 is a diagram showing the data structure of a second support table 1016. [Figure 9] FIG. 10 is a diagram showing the data structure of a teacher table 1017. [Figure 10] FIG. 10 is a diagram showing the data structure of a task master 1021. [Figure 11] FIG. 10 is a diagram showing the data structure of a parameter master 1022. [Figure 12] 10 is a flowchart showing a document processing operation. [Figure 13] 10 is a screen example showing the operation of a population creation processing page. [Figure 14] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0009] <System 1 Configuration> The system 1 in this disclosure is an information processing system that provides a patent document processing support service. The system 1 includes an information processing device including a server 10, a user terminal 20, and an artificial intelligence system 40, all of which are connected via a network N. FIG. 1 is a block diagram showing the functional configuration of the system 1. As shown in FIG. FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10, the user terminal 20, and the artificial intelligence system 40, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0011] <Server 10 configuration> The server 10 is an information processing device that provides a patent document processing support service. The server 10 includes a storage unit 101 and a control unit 104 .

[0012] <Configuration of the storage unit 101 of the server 10> The storage unit 101 of the server 10 includes an application program 1011 , a user table 1012 , a document table 1013 , a population table 1014 , a first support table 1015 , a second support table 1016 , a teacher table 1017 , a task master 1021 , and a parameter master 1022 .

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. Application programs 1011 include applications such as a web browser application.

[0014] User table 1012 is a table that stores and manages information about member users (hereinafter, "users") who use the service. When a user registers to use the service, the user's information is stored in a new record in user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having columns of user IDs and user names, with the user ID as the primary key. FIG. 4 is a diagram showing the data structure of the user table 1012. As shown in FIG.

[0015] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item that is set with a unique value for each user. The user name is an item for storing the name of the user. The user name may be set to any character string such as a nickname instead of a name.

[0016] The document table 1013 is a table for storing and managing information (document information) relating to patent documents. The document table 1013 is a table having columns for document ID, user ID, document name, and document content. FIG. 5 is a diagram showing the data structure of the document table 1013.

[0017] The document ID is an item that stores document identification information for identifying a patent document. The user ID is an item for storing user identification information for identifying a user. The document name is an item that stores the name of the patent document. Any character string can be set as the document name. For example, the document name stores the publication number, registration number, application number, or other reference number of the patent document. The document content is an item that stores the document content of a patent document. Specifically, the text information of the patent document is stored. For example, the document content may include some or all of the "claims," "abstract," "detailed description of the invention," "examples," "embodiments," etc., contained in the patent document (specification). Specifically, the publication text of the unexamined patent publication, patent gazette, etc. is stored.

[0018] The population table 1014 is a table for storing and managing a population (population information) including a plurality of patent documents. In the present disclosure, information for specifying a population consisting of a plurality of patent documents to be processed in various information processing is stored. The population table 1014 is a table having a population ID as a primary key, and columns of population ID, population data, task ID, and query data. FIG. 6 is a diagram showing the data structure of the population table 1014.

[0019] The population ID is an item for storing population identification information for identifying a population. The population identification information is an item for which a unique value is set for each piece of population information. Population data is an item that stores information that identifies a population consisting of document IDs of multiple patent documents. The population data may be configured to store not only document IDs, but also document names, document contents, etc. in association with the document IDs. Furthermore, multiple patent documents do not necessarily need to be identified by document IDs; a population consisting of multiple patent documents may be identified by a search formula, search conditions, or logical formula for searching patents. The task ID is an item for storing task identification information indicating a task to be applied to a population including a plurality of patent documents identified by population identification information. The query data is an item for storing a query to be applied to a population including a plurality of patent documents identified by the population identification information. Specifically, the query data stores the following queries according to the type of task: Classification task: A sentence (character string) defining the classification target and a list of example sentences (character strings) expressing the classification target are stored. Example) Category A is "···", Category B is "···", Category C is "···" Screening: A sentence (character string) describing the target invention is stored. Example: A device comprising... means,... means, and... · Applicability determination: A sentence (character string) describing the target product is stored. Example: A device comprising... means,... means, and... Invention evaluation: Sentences expressing evaluation items and evaluation criteria are memorized. Example) Innovation: A rank is... B rank is... C rank is... Economics: What is A rank? What is B rank? What is C rank?

[0020] The first support table 1015 is a table for storing and managing prompts related to query sentences to the artificial intelligence system and the contents of responses from the artificial intelligence system (first support information). The first support table 1015 is a table having columns of a first document ID, a prompt, and a first output value. FIG. 7 is a diagram showing the data structure of the first support table 1015. As shown in FIG.

[0021] The first document ID is an item that stores document identification information for identifying the first patent document. A prompt is an item that stores a prompt related to a query sentence to an artificial intelligence system. A prompt is mainly a query sentence (text) that is input to an artificial intelligence system. Specifically, a user can input a prompt to an artificial intelligence system so that the artificial intelligence system outputs a desired output result. Note that a prompt does not have to be a character string, but can also be a prompt in the form of an image, video, audio, etc. For example, a gesture or audio instruction by the user can also be a prompt. The first output value is an item for storing output data from the artificial intelligence system in response to a prompt. Specifically, the output data (character string information) corresponding to the task information used to generate the prompt is stored.

[0022] The second support table 1016 is a table for storing and managing output data (second support information) from the classification model 1051 for patent documents. Specifically, the second support table 1016 stores information such as classification labels and classification accuracy that are output in response to applying patent documents as input data to the classification model 1051. Note that, although the classification results using the classification model are described as an example in this disclosure, a configuration may be adopted in which screening results using a screening model, pertinence determination results using a pertinence determination model, and invention evaluation results using an invention evaluation model are stored instead of the classification results. The second support table 1016 is a table having columns for a second document ID, a second output value, and precision. FIG. 8 is a diagram showing the data structure of the second support table 1016. As shown in FIG.

[0023] The second document ID is an item for storing document identification information for identifying the second patent document. The second output value is an item that stores output data (character string information) such as a classification label that is output in response to applying the patent document identified by the second document ID as input data to the classification model 1051. The second output value may store output data such as a plurality of classification labels. The accuracy is an item that stores information indicating the quality of the second output value, such as the classification accuracy for the classification label that is output in response to applying the patent document identified by the second document ID as input data to the classification model 1051. The accuracy may store information indicating the quality for each of a plurality of classification labels, etc.

[0024] The teacher table 1017 is a table for storing and managing teacher information for patent documents. The teacher table 1017 is a table having columns of teacher document ID and teacher label. FIG. 9 is a diagram showing the data structure of the teacher table 1017.

[0025] The teaching document ID is an item for storing document identification information for identifying a patent document. The teacher label is an item for storing teacher data (character string information) such as a classification label of a patent document identified by the current private document ID.

[0026] The task master 1021 is a table for storing and managing information (task information) relating to tasks executed on patent documents. The task master 1021 is a table having the task ID as a primary key and columns of task ID, task type, extracted item, and task data. FIG. 10 is a diagram showing the data structure of the task master 1021.

[0027] The task ID is an item that stores task identification information for identifying a task. The task identification information is an item that has a unique value set for each piece of task information. The task type is an item that stores task type information related to the type of task. Specifically, the task type stores a character string that identifies the content of the task according to the purpose of the processing (task) performed on the patent document. In the present disclosure, the task type stores information such as "classification," "screening," "applicability determination," and "invention evaluation" according to the content of the following tasks. Note that in the present disclosure, the above tasks are examples, and the present disclosure can be applied to any task performed on a patent document. Classification task: A task to assign a classification to multiple patent documents according to their respective contents. Screening task: A task that determines whether each of multiple patent documents is related to the target document. Specifically, the task can be subdivided into "prior art search," "prior art search," "invalidation document search," etc. Relevance determination task: A task that determines whether each of multiple patent documents is related to the target product. For example, this task is performed in patent work such as "clearance search," "FTO search," and "investigation of infringement products of other companies." Invention evaluation task: A task that evaluates the value of the inventions in multiple patent documents, such as their innovativeness, economic viability, practicality, feasibility of other companies implementing them, and ease of detecting infringement. For example, this is a task performed in patent work, such as "determining whether to file an application," "considering whether to file an application abroad and the country in which to file," and "whether to maintain a registered patent." The extracted item is an item that stores information (extracted item information) indicating extracted items of a patent document when executing a task. Specifically, the extracted item includes character strings indicating extracted items of the patent document that provides the task, such as "abstract," "specification," "claims," and "drawings." The extracted item also includes character strings indicating extracted items of the patent document that provides the task, such as "title of the invention," "detailed description of the invention," "technical field," "background art," "prior art documents," "problem to be solved by the invention," "means for solving the problem," "effects of the invention," "brief explanation of the drawings," "form for carrying out the invention," "embodiment," "industrial applicability," "explanation of symbols, etc.", "problem," and "solution." Specifically, the following extracted item information is stored for each task type: Classification: Abstract, claims, independent claims, claim 1, specification, etc. Screening: Abstract, claims, independent claims, claim 1, specification, etc. - Applicability determination: Claim 1, claims, independent claims Invention evaluation: Claim 1, independent claims Task data is an item that stores information (character strings) related to a task. Specifically, the task data stores the following character strings depending on the task type. Note that the <query> item contains query data (described later), and the <extracted document> contains extracted portions (extracted content, partial sentences) from the text contained in the patent document, identified based on the extraction items, to generate a prompt. If the task type is "Classification" 『 Define technology classification as follows. Please classify the inventions described in the "Target Patents" below into either Class A, B, or C. Please calculate the relevance between the target patents and the classification on a scale of 0 to 100. Please output the reasons for your decision. #Technical classification: <Query> (Category A "...", Class B "...", Class C "...") #Target patent: <Extracted document> (abstract and claims of patent documents (contents)) 』 If the task type is "Screening" 『 Please consider whether the invention described in the "Target Invention" below is described in the invention described in the "Target Patent" below. Calculate the relevance between the target invention and the target patent on a scale of 0 to 100. Please output the reason for your decision. #Target invention: <Query> (a device comprising...means,...means, and...) #Target patent: <Extracted documents> (abstract, claims, and specification of patent documents) 』 If the task type is "Applicability determination" 『 Please consider whether the products, services, etc. listed in the "Target Products" below fall within the scope of the inventions listed in the "Target Patents" below. Please calculate the relevance between the target products and the target patents on a scale of 0 to 100. Please output the reasons for your decision. #Target product:<Query> (a device including a...means, a...means, and...) #Target patent: <Extracted document> (Claim 1 in the patent document (content)) 』 - Task type is "Invention evaluation" 『 The evaluation criteria are defined as follows: Please rate the inventions described in the "Target Patents" below as A, B, or C. Please output the reason for your decision. #Evaluation criteria:<Query>(Rating A "···", Rating B "···", Rating C "···") #Target patent: <Extracted document> (abstract and claims of patent documents (contents))

[0028] The parameter master 1022 is a table for storing and managing information relating to parameters (parameter information). The parameter master 1022 is a table having columns for extraction ratio, precision threshold, and application conditions. FIG. 11 is a diagram showing the data structure of the parameter master 1022.

[0029] The sampling rate is an item for storing a sampling rate when identifying a first patent document consisting of a plurality of patent documents from a population. Specifically, the sampling rate takes a value from 0 to 100%. The accuracy threshold is an item for storing conditions related to information indicating the quality of the second output value, such as accuracy, when identifying a second patent set consisting of one or more second patent documents. For example, based on the second output value and accuracy output by the classification model 1051, only those documents whose accuracy is greater than the accuracy threshold are identified and stored as second patent documents. Specifically, the accuracy threshold takes a value between 0 and 1. The application conditions are items for storing application conditions when applying the extraction ratio and the accuracy threshold. Specifically, the application conditions include conditions related to the number of times document processing has been performed, the number of unprocessed documents, the ratio, and the like.

[0030] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, a document processing unit 1042, and a classification model 1051. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0031] The user registration control unit 1041 performs processing to store information about users who wish to use the service according to the present disclosure in the user table 1012. The information stored in the user table 1012 is generated when a user opens a web page operated by a service provider from any information processing terminal, enters information into a predetermined input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Before the user registration control unit 1041 registers the user information in the user table 1012, the service provider may conduct a predetermined examination to restrict whether or not the user is permitted to use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or may be automatically set by the user registration control unit 1041.

[0032] The document processing unit 1042 executes document processing, the details of which will be described later.

[0033] The classification model 1051 is an item that stores a classification model for classifying patent documents. The classification model 1051 includes any learning model such as a machine learning model, a deep learning model, an artificial intelligence model, etc. In the present disclosure, the classification model 1051 includes a learning model that can output a classification name (classification label) and accuracy (information indicating inference accuracy and inference quality) corresponding to the classification name in response to input of a document (text data) such as a patent document. For example, the classification model 1051 includes Bayesian classification, SVM (Support Vector Machines), decision trees, random forests, logistic regression, a deep learning model, and BERT. The classification model 1051 may include an image classification model such as CNN, ResNet, Inception, VGGNet, etc. Since patent documents include images such as drawings, a configuration may be adopted in which a classification name can be output according to the images of the drawings, etc. The classification model 1051 can be realized by many general-purpose libraries used in natural language processing, such as NLTK (Natural Language Toolkit), spaCy, Scikit-learn, TensorFlow and Keras, PyTorch, FastText, and Gensim.

[0034] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or tablet, a stationary personal computer (PC) or a laptop PC, or a wearable terminal such as a head mounted display (HMD) or a wristwatch terminal. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 , and an output device 208 .

[0035] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012 .

[0036] The user ID 2011 is the user's account ID. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID 2011 includes information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20.

[0037] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server or the like operated by a service provider via a communication IF. The application programs 2012 include applications such as a web browser application. The application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20.

[0038] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 executes an application program 2012 stored in the storage unit 201, thereby realizing each functional unit.

[0039] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061 , a microphone 2062 , a position information sensor 2063 , a motion sensor 2064 , and a touch device 2065 .

[0040] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082 .

[0041] <Configuration of Artificial Intelligence System 40> The artificial intelligence system 40 is an information processing device that outputs a response to a prompt. Examples of the artificial intelligence system 40 include ChatGPT, OpenAI GPT, PerplexityAsk, and BingAI. These artificial intelligence systems have a dialogue (chat) function, and a user can provide any inquiry or command to the artificial intelligence system in text to receive a response to the inquiry or command. In the present disclosure, a user can obtain a text response that assists in the reading comprehension of a patent document by sending a prompt created in the query processing to the artificial intelligence system 40. In addition, in this disclosure, the artificial intelligence system is not limited to a text-based dialogue system. For example, it may be an image generation AI system such as Midjourney or Stable Diffusion. For example, a user can send a prompt created in an inquiry process to such an image generation AI system to obtain an image or video as a response that assists in reading and understanding patent documents. Additionally, the present disclosure can also be applied to an artificial intelligence system that outputs answer content in the form of video, audio, etc. The artificial intelligence system 40 includes a large-scale language model system (LLM system).

[0042] <System 1 Operation> Each process of the system 1 will be explained below. FIG. 12 is a flowchart showing the document processing operation. FIG. 13 is an example of a screen showing the operation of the population creation processing page.

[0043] <Document Processing> Document processing is the process of applying processing to patent documents.

[0044] <Document Processing Overview> Document processing is a series of steps that create a population to be processed, identify a first set from the population by sampling, generate a prompt to be applied to the first set, obtain a first output value output from the large-scale language model by sending the prompt to a large-scale language model, present the first output value to a user, create training data in response to input operations from the user, identify a second set from the population, perform classification processing on the second set using a classification model trained based on the training data, and repeat the above steps until there are no unprocessed documents in the population.

[0045] <Details of document processing> The document processing is described in detail below.

[0046] <Population creation step> In step S101, the document processing unit 1042 of the server 10 executes a population receiving step of receiving population information relating to a population including a plurality of patent documents. Specifically, the user operates the input device 206 of the user terminal 20 to input the URL of a page for executing the population creation processing (population creation processing page D1) into a web browser or the like, and opens the population creation processing page. The control unit 204 of the user terminal 20 sends a request to open the population creation processing page to the server 10. Based on the received request, the control unit 104 of the server 10 generates the population creation processing page D1 and sends it to the user terminal 20. The control unit 204 of the user terminal 20 displays the received population creation processing page D1 on the display 2081 of the user terminal 20. 13 is an example screen of a population creation processing page D1 in document processing. The population creation processing page D1 is displayed on the display 2081 of the user terminal 20. The population creation processing page D1 includes a task selection field D101, an extraction item selection field D111, a query input field D102, a document input field D103, a prompt display field D104, and a send button D105.

[0047] <Population creation step (task reception step)> The control unit 204 of the user terminal 20 executes a task presenting step of presenting a plurality of pieces of task information to the user in a selectable manner. Specifically, the document processing unit 1042 of the server 10 references the task master 1021, acquires the task ID and task type items, and transmits them to the user terminal 20. The control unit 204 of the user terminal 20 displays the received multiple task types in a selectable manner in the task selection field D101. For example, the control unit 204 of the user terminal 20 presents to the user each of the multiple task types displayed in the task selection field D101 in association with a radio button for selecting the task type.

[0048] The document processing unit 1042 of the server 10 executes a task receiving step of receiving task information related to a task to be executed on a patent document. The task receiving step executes a step of receiving, from a user, an input operation of predetermined task information from among the plurality of pieces of task information presented in the task presentation step. Specifically, the user operates the input device 206 of the user terminal 20 to select a predetermined task type to be executed on the patent document from among the multiple task types displayed in the task selection field D101.

[0049] The task receiving step executes a step of receiving task information indicating at least one of a classification task for assigning a classification to a patent document, a screening task for determining the relevance between the patent document and a target document, a relevance determination task for determining the relevance between the patent document and a target product, and an invention evaluation task for evaluating the value of an invention related to the patent document. Specifically, the user can select a predetermined task type from among task types such as classification assignment task, screening task, applicability determination task, invention evaluation task, etc. Note that a configuration may be adopted in which multiple task types can be selected.

[0050] The control unit 204 of the user terminal 20 transmits a task ID corresponding to the task information selected in the task selection field D101 to the server 10. The document processing unit 1042 of the server 10 receives and accepts the task ID.

[0051] In the present disclosure, a configuration in which multiple pieces of task information are presented as selectable pieces and selected by the user has been described as an example, but this is not limiting. For example, the task information may be of one type (one of a classification task, a screening task, a classification task, and an invention evaluation task), and the task information may be predetermined as predetermined task information. In this case, too, the document processing unit 1042 of the server 10 can be interpreted as having received input of the predetermined task information. Furthermore, the task information may not be received by selection, but may be specified based on the content of a query received in a query receiving step described later. In addition, instead of the task selection field D101, a task input field may be provided in which the user can input task information, task type, etc. The user may input information indicating task information such as a classification task, screening task, applicability determination task, invention evaluation task, etc. into the task input field, and the control unit 204 of the user terminal 20 or the document processing unit 1042 of the server 10 may accept the task type.

[0052] <Population creation step (query reception step)> The document processing unit 1042 of the server 10 executes a query receiving step of receiving query information to be applied to a patent document. Specifically, the user inputs a character string related to a query to be applied to the patent document into the query input field D102 by operating the input device 206 of the user terminal 20. For example, the user inputs the following query into the query input field D102 according to the task type selected in step S501 by operating the input device 206 of the user terminal 20:

[0053] [Query for classification task] Category A is a category related to information processing equipment. Classification B is a classification for agricultural machinery. Classification C is a classification for food products.

[0054] [Query for screening task] Query example 1) A device comprising: a means, a means, and a Query example 2) A method that includes a process, a process, and a Query example 3) A program that executes a... step, a... step, and...

[0055] [Query for the applicability determination task] Query example 1) A device comprising: a means, a means, and a Query example 2) A method that includes a process, a process, and a Query example 3) A program that executes a... step, a... step, and...

[0056] [Query for invention evaluation task] Innovation: What is A rank? What is B rank? What is C rank? Economics: What is A rank? What is B rank? What is C rank?

[0057] The control unit 204 of the user terminal 20 transmits the character string entered in the query input field D102 to the server 10. The document processing unit 1042 of the server 10 receives the character string and accepts it as query data.

[0058] <Population creation step (document acquisition step)> The document processing unit 1042 of the server 10 executes a document acquisition step for acquiring patent documents relating to patents. Specifically, the user operates the input device 206 of the user terminal 20 to input the document ID of the patent document in the document input field D103. Note that the patent document does not need to be specified by the document ID in the document table 1013; the user may input the patent number of the published patent publication or the like, the abstract of each patent document, the scope of claims, and the patent specification, etc., into the document input field D103. The user may also be configured to directly select patent document information contained in a Word, Excel, CSV file, etc. stored in the memory unit 201 of the user terminal 20 as the file to be uploaded. Any means may be used as long as the user can transmit a specified patent document to the server 10 by any means.

[0059] The document acquisition step executes a step of acquiring patent documents relating to a plurality of patents. The user may operate the input device 206 of the user terminal 20 to transmit multiple patent documents to the server 10. The user may input document IDs of multiple patent documents into the document input field D103 by operating the input device 206 of the user terminal 20. The user may also transmit an Excel or CSV file containing information on multiple patent documents to the server 10.

[0060] The control unit 204 of the user terminal 20 transmits one or more patent documents input or selected by the user to the server 10. The document processing unit 1042 of the server 10 acquires and accepts one or more patent documents by reception.

[0061] The document processing unit 1042 of the server 10 stores the task ID, query data, and one or more patent documents received from the user terminal 20 in the task ID, query data, and population data items of a new record in the population table 1014.

[0062] <Population creation steps (second time onwards)> In step S101, the population receiving step includes a step of receiving first population information regarding a first population, and a step of receiving second population information regarding a second population including one or more patent documents from the first population information excluding one or more patent documents stored in association with the second output result in the second storage step. Specifically, when document processing is performed up to step S109 and then step S101 is performed again (when performed for the second time or later), the patent documents identified by the second document ID associated with the second output value in the second support table 1016 are excluded from the patent documents included in the population data and accepted as a new population. This allows the first and second processes to be performed again on patent documents for which the quality of the second process results from the learning model is not satisfactory. Patent documents for which the quality of the second process results is satisfactory are gradually removed from the initially given population, so the first and second processes can be gradually applied to multiple patent documents included in the initially given population. By applying the second process to the population as much as possible, the application of the first process can be suppressed.

[0063] In step S101, the population receiving step includes a step of receiving first population information regarding a first population, and a step of receiving second population information regarding a second population including one or more patent documents from the first population information excluding one or more patent documents stored in association with information based on the first output result in the first storage step and one or more patent documents stored in association with the second output result in the second storage step. Specifically, when document processing is performed up to step S109 and then step S101 is performed again (when performed for the second time or later), patent documents included in the population data may be excluded from the data and accepted as a new population, excluding patent documents identified by the teacher document ID associated with the teacher label in the teacher table 1017 and patent documents identified by the second document ID associated with the second output value in the second support table 1016. Similarly, patent documents identified by the first document ID associated with the first output value in the first support table 1015 may be excluded from the patent documents included in the population data and accepted as a new population. This allows the first process and the second process to be performed again on patent documents excluding patent documents that have been given the first output result or training data based on the first output result, and patent documents for which the quality of the second process result is favorable. Since patent documents that have been given training data or for which the quality of the second process result is favorable are gradually removed from the initially given population, the first process and the second process can be gradually applied to multiple patent documents included in the initially given population. By applying the second process to the population as much as possible, the application of the first process can be suppressed.

[0064] <Sampling Step (First Embodiment)> In step S102, the document processing unit 1042 of the server 10 executes a first patent extraction step of extracting a first patent document from the plurality of patent documents included in the population received in the population receiving step. Specifically, the document processing unit 1042 of the server 10 randomly extracts (samples) some patent documents (first patent documents) from the multiple patent documents included in the population data stored in the population table 1014 in step S101. For example, in the present disclosure, the first patent document is intended to create training data for processing to be performed on multiple patent documents included in the population data, and therefore, patent documents are extracted at an extraction rate of 10 to 20% from the multiple patent documents included in the population data. For example, if the population data contains 1,000 patent documents, 100 to 200 patent documents are extracted.

[0065] <Sampling Step (Second Embodiment)> In step S102, the first patent extraction step includes a step of dividing the multiple patent documents included in the population received in the population reception step into multiple groups, and a step of extracting one or more patent documents as first patent documents from each of the multiple groups. Specifically, the document processing unit 1042 of the server 10 performs clustering processing on the plurality of patent documents included in the population data stored in the population table 1014 in step S101. Specifically, the document processing unit 1042 of the server 10 performs clustering processing on the plurality of patent documents based on the similarity of the document contents of the plurality of patent documents (text data such as claims, abstracts, detailed descriptions of the invention, examples, embodiments, and other information, and image data such as drawings). For example, any method such as k-means, hierarchical clustering, DBSCAN, or spectral clustering can be used for the clustering processing. This allows the plurality of patent documents included in the population data to be classified into groups (clusters) such as group A, group B, group C, etc. The document processing unit 1042 of the server 10 extracts a predetermined number of patent documents from each group. Specifically, the document processing unit 1042 of the server 10 extracts from each group a number of patent documents equal to 10 to 20% of the number of patent documents included in the population data, and divides this total number by the number of groups. For example, if the population data contains 1,000 patent documents and there are four groups, A, B, C, and D, and the total number of patent documents to be extracted is 100, 25 patent documents are extracted from each group. Similarly, if the total number of patent documents to be extracted is 200, 50 patent documents are extracted from each group. It is preferable to extract patent documents from groups randomly. This allows the first process to be applied in a balanced manner across the entire expression space of the patent documents included in the population, even if the contents of the patent documents included in the population are biased, by sampling a predetermined number of patent documents for each group. For example, the user can prepare training data in a balanced manner across the entire expression space of the patent documents included in the population. This allows the second process to be performed with high quality and also saves processing time and costs. The number of first patent documents extracted for each group may be a predetermined fixed number, or a predetermined number of first patent documents may be extracted according to the number of patent documents included in the group or the proportion of the number of documents in each group.

[0066] <Sampling steps (second and subsequent times)> In step S102, the first patent extraction step includes a step of extracting a number of first patent documents according to a first ratio from the first population information, and a step of extracting a number of first patent documents according to a second ratio from the second population information. Specifically, when document processing is performed up to step S109 and then step S101 is performed again (when performed for the second time or later), first patent documents are extracted from the population at a rate of P% (first rate) in step S102 for the first time, and first patent documents are extracted from the population at a rate of Q% (second rate), which is different from P%, in step S102 for the second time.

[0067] For example, the second percentage may be greater than the first percentage. Specifically, P% may be 10% and Q% may be 15%. In this case, if there are many patent documents for which the quality of the second processing result is not satisfactory, the convergence of the processing may be poor. By making the second ratio larger than the first ratio, the proportion of patent documents processed by the large-scale language model in the population increases, thereby increasing the amount of training data used in the second processing and reducing the proportion of patent documents for which the quality of the second processing result is not satisfactory. This may reduce the number of processing times in the processing disclosed herein, thereby shortening the total processing time.

[0068] For example, the first percentage may be greater than the second percentage. Specifically, P% may be 10% and Q% may be 5%. In this case, if the second proportion is made smaller than the first proportion, the proportion of patent documents in the population that are processed by the large-scale language model will decrease, thereby making it possible to save on processing costs associated with the first processing.

[0069] The document processing unit 1042 of the server 10 may search for the applicable condition items in the parameter master 1022 and acquire the extraction ratio (P%, Q%). For example, the number of times document processing is executed (first time, second time, etc.) may be used as an applicable condition, and the extraction ratio (P%, Q%) corresponding to the number of executions may be acquired.

[0070] The document processing unit 1042 of the server 10 stores one or more document IDs of the extracted one or more first patent documents in the document ID field of a new record in the first support table 1015. As a result, the processing target of the prompt generation step is stored in the record of the first support table 1015 as first support information.

[0071] <Prompt generation step> In step S103, a step of creating a prompt, which is input data for causing the large-scale language model based on the first patent document to execute processing, is executed. In this disclosure, the process when one patent document is input will be described below as an example. Note that the document processing unit 1042 of the server 10 executes a prompt generation step for each of the multiple patent documents included in the first patent document.

[0072] <Prompt Generation Step (First Embodiment)> In step S103, the document processing unit 1042 of the server 10 executes a prompt generation step of generating a prompt including the first patent document extracted in the first patent extraction step in the large-scale language model. The prompt generation step may include a step of generating a prompt for causing the large-scale language model to output supplemental information including support information for assisting in reading comprehension of the first document. Specifically, the document processing unit 1042 of the server 10 searches the task ID item of the task master 1021 based on the task ID of the target record in the population table 1014, and acquires the extracted items and task data items. The document processing unit 1042 of the server 10 searches the document ID item of the document table 1013 based on the first document ID of the target record in the first support table 1015, and acquires the document content (contents of the patent document such as claims, abstract, detailed description of the invention, examples, and embodiments). The document processing unit 1042 of the server 10 extracts predetermined extracted parts (extracted content, partial sentences) from the document content based on the acquired extracted items. Note that extraction of extracted parts does not necessarily have to be performed. The document processing unit 1042 of the server 10 generates a prompt based on the acquired task data, the query data of the target record in the population table 1014, and the extracted extraction portion. The document processing unit 1042 of the server 10 stores the generated prompt in the prompt field of the record of the first support information identified by the document ID of the patent document for which the prompt is to be generated in the first support table 1015. In other words, the document processing unit 1042 of the server 10 generates a prompt for each piece of first support information stored in the first support table 1015, and stores it in association with the first support information.

[0073] <Prompt Generation Step (Second Embodiment)> In step S103, the prompt generation step includes a step of generating a prompt including a classification definition sentence that specifies the first patent document and classification rules, for outputting first classification information regarding the classification of the first patent document to the large-scale language model.

[0074] Specifically, the document processing unit 1042 of the server 10 generates a prompt by applying the query data of the target record in the population table 1014 and the extracted extraction portion to the acquired task data. The document processing unit 1042 of the server 10 can generate the following prompt by replacing "<query>" with the query data and "<extracted document>" with the extraction portion in the character string of the task data. In this disclosure, a classification task, a screening task, a classification determination task, and an invention evaluation task are disclosed as examples, but the applicable processes and tasks are not limited to these. The present invention can be applied to any task that can be handled by a large-scale language model.

[0075] [Prompt for classification task] Define technology classification as follows. Please classify the inventions described in the "Target Patents" below into either Class A, B, or C. Please calculate the relevance between the target patents and the classification on a scale of 0 to 100. Please output the reasons for your decision. #Technical classification: Category A is a category related to information processing equipment. Classification B is a classification for agricultural machinery. Classification C is a classification for food products. #Applicable patents: [Abstract] ... [Claims] ...

[0076] [Prompts for screening tasks] Please consider whether the invention described in the "Target Invention" below is described in the invention described in the "Target Patent" below. Calculate the relevance between the target invention and the target patent on a scale of 0 to 100. Please output the reason for your decision. #Target Invention: ...means, ...means, ...apparatus comprising #Applicable patents: [Abstract] ... [Claims] ... [Specification] ...

[0077] [Prompt for the relevant / inappropriate task] Please consider whether the products, services, etc. listed in the "Target Products" below fall within the scope of the inventions listed in the "Target Patents" below. Please calculate the relevance between the target products and the target patents on a scale of 0 to 100. Please output the reasons for your decision. #Applicable products: ...means, ...means, ...apparatus comprising #Applicable patents: [Claim 1] ...

[0078] [Prompt for Invention Evaluation Task] Please rate the inventions described in the "Target Patents" below according to the evaluation criteria for each evaluation item below. #Evaluation items and criteria: Innovation: What is A rank? What is B rank? What is C rank? Economics: What is A rank? What is B rank? What is C rank? #Applicable patents: [Claim 1] ...

[0079] <Prompt generation step (classification definition creation process)> When executing a classification task or an invention evaluation task in the prompt generation step (second embodiment), it is necessary to define the classification definition and the evaluation items and evaluation criteria in advance. The document processing unit 1042 of the server 10 may execute a third patent extraction step of extracting a third patent document from the plurality of patent documents included in the population received in the population receiving step. These classification definitions and evaluation items / criteria can also be created using the artificial intelligence system 40. The following mainly describes how to create classification definitions in the classification task, but is not limited to this. Similarly, the evaluation items and evaluation criteria in the invention evaluation task may be created by the artificial intelligence system 40.

[0080] The document processing unit 1042 of the server 10 may execute a classification perspective acquisition step of acquiring classification perspective information regarding the classification perspective of the third patent document, which is output in response to inputting into the large-scale language model the third patent document extracted in the third patent extraction step and a prompt including a classification perspective sentence in which the classification perspective is defined. Specifically, in step S101, the document processing unit 1042 of the server 10 randomly extracts (samples) some patent documents (third patent documents) from the plurality of patent documents included in the population data stored in the population table 1014. Note that the extraction of the patent documents may be performed by executing the same process as the extraction of the first patent document in step S102. The document processing unit 1042 of the server 10 creates the following prompt based on the document contents of the multiple patent documents included in the extracted third patent document and the classification viewpoint sentence that defines the classification viewpoint. As a result, the document processing unit 1042 of the server 10 creates a prompt for each of the multiple extracted patent documents.

[0081] [Classification creation task prompt] Please classify the inventions described in the "Target Patents" below according to the following classification criteria. #Classification perspective: Technical field, application, problem, purpose, configuration, solution, effect [Abstract] ... [Claims] ...

[0082] The document processing unit 1042 of the server 10 sends a request including the generated prompt (character string) to an API endpoint provided by the artificial intelligence system 40 as input data for the artificial intelligence system 40. The artificial intelligence system 40 sends a response including an answer to the prompt to the server 10. The document processing unit 1042 of the server 10 receives a response to the sent prompt. The response includes a character string related to the answer to the prompt. Specifically, the following answer is obtained in the classification creation task: In this disclosure, the following response content will be described as an example, but it is preferable to receive the response content as structured data such as JSON that has a field (key) related to the "classification name." For example, in a large-scale language model (LLM) called ChatGPT, the response result can be structured (typed) data by using a function called Function Calling. In response to the prompts created for each of the plurality of patent documents, answer contents for each of the plurality of patent documents (answer contents for each of the plurality of patent documents) are obtained.

[0083] [Answers to the classification task] #Classification perspective 1: Technical field #Category:Semiconductor Technology #Reason: In the abstract,... #Classification perspective 2: Purpose #Classification name: Refinement #Reason: In the abstract,...

[0084] The document processing unit 1042 of the server 10 may execute a classification definition acquisition step to acquire a classification definition sentence obtained by generalizing the classification perspective information related to the third patent document, which is output in response to inputting a prompt including the classification perspective information acquired in the classification perspective acquisition step into the large-scale language model. Specifically, the document processing unit 1042 of the server 10 creates the following prompt based on multiple classification names obtained as response results from the artificial intelligence system 40 for one or multiple patent documents: Classification name A, classification name B, and classification name C are input with classification names (classification labels) created in the classification creation task, such as "semiconductor technology," "miniaturization," and "high integration."

[0085] [Classification definition task prompt] Please create a classification definition by grouping the classification labels listed in the "Classification Name" below according to the classification perspective. #Category name: [Category name A] [Category name B] [Category name C]

[0086] The document processing unit 1042 of the server 10 sends a request including the generated prompt (character string) to an API endpoint provided by the artificial intelligence system 40 as input data for the artificial intelligence system 40. The artificial intelligence system 40 sends a response including an answer to the prompt to the server 10. The document processing unit 1042 of the server 10 receives a response to the sent prompt. The response includes a character string related to the answer to the prompt. Specifically, the following answer (category definition) is obtained in the category definition task: In this way, classification definitions can be automatically created using classification names created by applying the artificial intelligence system 40 to the multiple patent documents included in the third patent document. The classification definitions thus obtained can be used as queries for the classification task described above. This makes it possible to automatically classify multiple patent documents included in a population containing multiple patent documents without manually preparing classification definitions.

[0087] [Answers to the classification definition task] #Category 1: High-performance semiconductor technology #Classification explanation: Inventions related to technology that improves the performance of semiconductors #Subcategory: Wiring innovations, high integration innovations, etc. #Category 2: Low-cost semiconductor technology #Classification explanation: Inventions related to technology that reduces the design and manufacturing costs of semiconductors #Subcategory: Material innovation, processing innovation, etc.

[0088] <First Output Value Acquisition Step (First Embodiment)> In step S104, the document processing unit 1042 of the server 10 executes a first processing step of obtaining a first output result that is output in response to inputting a prompt including the first patent document extracted in the first patent extraction step into the large-scale language model. In step S104, the first processing step executes a step of acquiring first classification information regarding the classification of the first patent document, which is output in response to inputting a prompt including the first patent document and a classification definition sentence that specifies a classification rule into the large-scale language model. In step S104, the first processing step executes a step of acquiring first classification information based on the classification definition sentence acquired in the classification definition acquisition step.

[0089] Specifically, the document processing unit 1042 of the server 10 transmits a request including the prompt (character string) generated in step S103 to an API endpoint provided by the artificial intelligence system 40 as input data for the artificial intelligence system 40. The artificial intelligence system 40 transmits a response including an answer to the prompt to the server 10. The document processing unit 1042 of the server 10 receives a response to the sent prompt. The response includes a character string related to the answer to the prompt. Specifically, the AI system 40 outputs a character string related to the answer to each of the classification task, screening task, pertinence determination task, and invention evaluation task, for each of the classification task, screening task, pertinence determination task, and invention evaluation task. For example, the answer content includes the following information: Note that in this disclosure, the answer content below will be described as an example, but it is preferable to configure the answer content to be received as structured data such as JSON having fields (keys) such as "classification name." For example, in a large-scale language model (LLM) called ChatGPT, the answer result can be structured (typed) data by using a function called Function Calling. In response to the prompts created for each of the plurality of patent documents, answer contents for each of the plurality of patent documents (answer contents for each of the plurality of patent documents) are obtained. The document processing unit 1042 of the server 10 stores the response result in the first output value field of the record of the first support information specified by the document ID of the patent document for which the prompt is to be generated in the first support table 1015.

[0090] [Answer for classification task] #Classification:A #Relevance:80 #reason: The target patent has the structure..., structure.... On the other hand, the target patent does not have the structure..., structure.... It meets the classification definition of Classification A, but does not meet the classification definitions of Classification B and Classification C. The relevance to Classification A is about 80.

[0091] [Answers for screening tasks] #Relevance:40 #reason: The target invention is consistent with the subject patent in terms of... and differs from it in terms of... There are significant differences, and the relevance to the target patent is around 40.

[0092] [Answer content for the relevant / non-relevant task] #Relevance:20 #reason: The subject product is consistent with the subject patent in terms of... and differs from the subject patent in terms of... The product in question is not included in the scope of the patent in question. The relevance to the patent in question is approximately 20.

[0093] [Answer content for invention evaluation task] #Innovativeness: Rank B #Reason: The target patent has a configuration such as... #Economical: Rank A #Reason: The target patent has a configuration such as...

[0094] In this case, the index value (relevance) is an index value indicating the accuracy of the assigned classification if the task information is a classification assignment task, an index value indicating the relevance with the target invention if the task information is a screening task, and an index value indicating the relevance with the target product if the task information is a pertinence determination task.

[0095] <First output value presentation step> In step S105, the document processing unit 1042 of the server 10 executes a first presentation step of presenting the first output result acquired in the first processing step to the user. Specifically, the document processing unit 1042 of the server 10 transmits the received response content to the user terminal 20. The control unit 204 of the user terminal 20 displays the received response content on the display 2081 of the user terminal 20. Note that the configuration may be such that the user can download information related to the response result in a file format such as a Word, Excel, or CSV file by operating the input device 206 of the user terminal 20. The document processing unit 1042 of the server 10 may present the extracted portion of the patent document to the user in addition to the index value and the basis, thereby enabling the user to confirm which portion of the patent document the index value was calculated based on and on what basis.

[0096] The document processing unit 1042 of the server 10 transmits to the user terminal 20 the response contents corresponding to the prompts generated for each of the multiple patent documents included in the first patent document. The control unit 204 of the user terminal 20 may display and present the received multiple response contents on the display 2081 of the user terminal 20. Specifically, information identifying the multiple patent documents, such as the document ID, publication number, and document name, may be associated with the prompts generated for each document, and the response contents received from the artificial intelligence system 40 in response to the prompts, and presented to the user. Furthermore, in addition to the index values and the rationale, extracted portions of the patent documents may be presented to the user. In addition, the user may be able to download, by operating the input device 206 of the user terminal 20, information identifying multiple patent documents, such as the document ID, publication number, and document name, as well as a file such as a Word, Excel, or CSV file that stores prompts generated for each document and the answers received from the artificial intelligence system 40 in response to the prompts.

[0097] <Steps for creating training data> In step S106, the document processing unit 1042 of the server 10 accepts input operations from the user regarding the first output result presented in the first presentation step, and executes a training data creation step to create training data regarding the first patent document in accordance with the input operations. Specifically, the user can edit the first output result displayed and presented on the display 2081 of the user terminal 20 by operating the input device 206 of the user terminal 20. Specifically, the user can modify or delete the content of the first output result by operating the input device 206 of the user terminal 20. The user transmits the edited content of the first output result, such as modification or deletion, to the server 10 by operating the input device 206 of the user terminal 20. Since the content of the first output value output from the large-scale language model often contains incorrect or inaccurate information, it is preferable for the user to create training data based on the first output value by correcting, editing, or deleting the content of the first output value.

[0098] In step S106, the document processing unit 1042 of the server 10 executes a first storage step of storing at least a part or all of the first patent documents extracted in the first patent extraction step in association with information based on the first output result. Specifically, the document processing unit 1042 of the server 10 associates the first document ID with the corrected and edited first output result based on the editing content received from the user terminal 20, and stores them in the teacher document ID and teacher label items of a new record in the teacher table 1017. If the content of the first output value is valid, the user sends the first document ID and the first output result to the server 10 as is, and the document processing unit 1042 of the server 10 stores the first document ID and the first output result received in the teacher document ID and teacher label items of the teacher table 1017. This allows the user to create training data with less work load by referring to the first output result. Compared to using the first output result as training data as is, the second processing is performed based on training data created and edited by the user, so higher quality processing results are expected.

[0099] In the present disclosure, an example in which teacher data is created in response to a user's input operation based on the first output result output from the artificial intelligence system 40 in steps S105 and S106 has been described as an example, but the present disclosure is not limited to this. For example, the first output result (first output value) for the first patent document may be stored directly in the teacher label field of the teacher table 1017. In this case, steps S105 and S106 may be omitted.

[0100] <Processing target identification step> In step S107, the document processing unit 1042 of the server 10 executes a second patent extraction step of extracting a second patent document from the plurality of patent documents included in the population received in the population receiving step. Specifically, the document processing unit 1042 of the server 10 extracts all of the plurality of patent documents included in the population data stored in the population table 1014 in step S101 as second patent documents. The document processing unit 1042 of the server 10 may identify the second patent document from among the multiple patent documents included in the population data, excluding the patent document identified by the training document ID item in the training table 1017. Similarly, the document processing unit 1042 of the server 10 may identify a second patent document from among the multiple patent documents included in the population data, excluding the patent document identified by the first document ID item in the first support table 1015. Alternatively, the document processing unit 1042 of the server 10 may randomly extract (sample) some patent documents (first patent documents) from the plurality of patent documents and use them as the second patent documents. Note that the sampling process may be the same as the sampling step in step S102. Detailed explanations will be omitted.

[0101] The document processing unit 1042 of the server 10 stores one or more document IDs of the extracted one or more second patent documents in the document ID field of a new record in the second support table 1016. As a result, the processing target of the second output value acquisition step is stored in the record of the second support table 1016 as first support information.

[0102] <Learning process> In step S108, the document processing unit 1042 of the server 10 executes a learning process for the classification model 1051. The learning process for the classification model 1051 will be described below.

[0103] <Classification model 1051 learning process> The learning process of the classification model 1051 is a process of learning the learning parameters of the deep neural network included in the classification model 1051 by deep learning.

[0104] <Overview of the learning process for classification model 1051> The learning process of the classification model 1051 is a process of learning the learning parameters of the deep neural network included in the classification model 1051 by deep learning, using the document content of the patent document (which may be an extraction point extracted based on the extraction items of the task master 1021 by referring to the task ID of the population information, or any extraction point such as the claims, abstract, detailed description of the invention, examples, and embodiments) as input data (input vector) and the teacher label (first output result) as output data (teaching data).

[0105] <Details of the training process for classification model 1051> The control unit 104 of the server 10 acquires training information including a training document ID and a training label from the training table 1017. The document processing unit 1042 of the server 10 searches the document ID item in the document table 1013 based on the training document ID and acquires the document content. The document processing unit 1042 of the server 10 preferably extracts an extraction portion from the document content based on the extraction item in the task master 1021. In other words, it is preferable to extract a predetermined extraction portion from the patent document so that it is the same as the input data when the first output value was acquired in step S104. The control unit 104 of the server 10 creates learning data using document contents, extracted portions, etc. as input data (input vectors) and teacher labels as output data (teaching data). The control unit 104 of the server 10 creates data sets such as training data, test data, and validation data for training the deep neural network of the classification model 1051 based on the learning data. The control unit 104 of the server 10 uses deep learning to train the learning parameters of the deep neural network included in the classification model 1051 based on the created dataset.

[0106] <Second output value acquisition step> In step S108, the step of obtaining a second output value is performed by inputting input data based on the second patent document into the classification model 1051. In the present disclosure, the processing when one patent document is input will be described below as an example. Note that the document processing unit 1042 of the server 10 executes the second output value acquisition step for each of the multiple patent documents included in the second patent document.

[0107] <Second Output Value Acquisition Step (First Embodiment)> In step S108, the second processing step executes a step of acquiring second classification information regarding the classification of the second patent document, which is output in response to inputting the second patent document into the learning model, which is a classification model. The second processing step includes a step of obtaining second quality information indicating the quality of classification for one or more patent documents contained in the second patent document output in response to inputting the second patent document into the learning model. If the second quality information obtained in the second processing step satisfies a predetermined condition, the document processing unit 1042 of the server 10 executes a second classification storage step in which at least a portion of one or more patent documents included in the second patent document is stored in association with the second classification information. Specifically, the document processing unit 1042 of the server 10 searches the task ID item of the task master 1021 based on the task ID of the target record in the population table 1014, and acquires the extracted item. The document processing unit 1042 of the server 10 searches the document ID item of the document table 1013 based on the second document ID of the target record in the second support table 1016, and acquires the document content (contents of the patent document such as claims, abstract, detailed description of the invention, examples, and embodiments). The document processing unit 1042 of the server 10 extracts predetermined extracted parts (extracted content, partial sentences) from the document content based on the acquired extracted items. Note that extraction of extracted parts does not necessarily have to be performed. The document processing unit 1042 of the server 10 inputs the document content or the extracted portion as input data to the classification model 1051. Note that the document processing unit 1042 of the server 10 may input keywords and the like extracted by applying processing such as morphological analysis to the document content or the extracted portion as input data to the classification model 1051. In response to input data, the classification model 1051 outputs a classification label (character string) and an index (quality index) indicating the quality of the classification process when classifying the input data into the classification label, such as accuracy. Specifically, accuracy is an index indicating how accurately the classification model 1051 predicted the classification label. Accuracy is the number of samples correctly predicted by the classification model 1051 divided by the total number of samples. Accuracy takes a value between 0 and 1 and can be expressed as a percentage (0% to 100%). Indicators indicating the quality of the classification process include a confusion matrix, precision, recall, F1 score, etc. For example, for a particular patent document A, the classification model 1051 outputs a second output value such as "classification label: semiconductor technology (accuracy 0.7)".

[0108] In step S108, the document processing unit 1042 of the server 10 stores at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step satisfies a predetermined condition, and does not store at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step does not satisfy the predetermined condition. A second storage step is performed. Specifically, the document processing unit 1042 of the server 10 acquires a classification label and a quality index for each of the multiple patent documents included in the second patent document set. The document processing unit 1042 of the server 10 compares the quality index with a predetermined accuracy threshold, and if the condition that the quality index is equal to or greater than the accuracy threshold is met, stores the acquired classification label and quality index in the second output value and accuracy fields of the record of the patent document identified based on the document ID of the processing target in the second support table 1016. In other words, the document processing unit 1042 of the server 10 does not store all of the second output results output from the classification model 1051 in the second support table 1016, but stores only the second output results whose quality indexes satisfy predetermined conditions in the second support table 1016. In other words, if the quality index of the second output result is insufficient, it is not stored in the second support table 1016. In this case, if the learning of the classification model 1051 progresses sufficiently in the second or subsequent document processing, the quality index of the second output result will have a sufficient value, and will be stored in the second support table 1016. The document processing unit 1042 of the server 10 may delete records of second support information stored in the second support table 1016 whose quality index does not satisfy specified conditions (only the second document ID is stored, and the second output value and accuracy values are blank).

[0109] <Second Output Value Acquisition Step (Second Embodiment)> In step S108, the document processing unit 1042 of the server 10 executes a second processing step to obtain a second output result that is output in response to inputting the second patent document extracted in the second patent extraction step into a learning model that has been trained using information based on the first output result obtained in the first processing step as training data. The second processing step executes a step of obtaining a second output result that is output in response to inputting a second patent document into a learning model that has been trained based on the training data created in the training data creation step. The second processing step includes a step of acquiring second quality information indicating the quality of processing for one or more patent documents included in the second patent document output in response to inputting the second patent document into the learning model. Specifically, in the present disclosure, a case where the second output value is output by the classification model 1051 is taken as an example, but the present disclosure is not limited to this. For example, instead of the classification model 1051, a screening model may be used that outputs a relevance to a target patent and a quality index (accuracy, etc.) related to the relevance in response to input of the document content and extracted portion of a patent document. In this case, the screening model is trained using training data based on the answer content (first output value) in the case of a screening task. The screening model outputs a relevance to a target patent and a quality index (accuracy, etc.) related to the relevance in response to input of the document content and extracted portion of a second patent document. For example, for a specific patent document A, the screening model outputs a second output value such as "relevance: 80% (accuracy 0.7)." For example, instead of the classification model 1051, a pertinence determination model may be used that outputs a degree of relevance with a target product and a quality index (such as accuracy) related to the degree of relevance in response to input of the document content and extracted portion of a patent document. In this case, the pertinence determination model is trained using training data based on the answer content (first output value) in the pertinence determination task. In response to input of the document content and extracted portion of a second patent document, the pertinence determination model outputs a degree of relevance with a target product and a quality index (such as accuracy) related to the degree of relevance. For example, for a specific patent document A, the pertinence determination model outputs a second output value such as "relevance: 80% (accuracy 0.7)." For example, instead of the classification model 1051, an invention evaluation model may be used that outputs an evaluation value (evaluation rank) for each evaluation item of the target invention and a quality index (accuracy, etc.) related to the evaluation value (evaluation rank) for each evaluation item in response to input of the document content and extracted portions of the patent document. In this case, the invention evaluation model is trained using training data based on the answer content (first output value) in the invention evaluation task. In response to input of the document content and extracted portions of the patent document of the second patent document, the invention evaluation model outputs an evaluation value (evaluation rank) for each evaluation item and a quality index (accuracy, etc.) related to the degree of relevance. For example, for a specific patent document A, the invention evaluation judgment model outputs a second output value such as "Innovativeness: Rank A (accuracy 0.7), Economic efficiency: Rank B (accuracy 0.6)."

[0110] <Second output value acquisition step (second time onwards)> In step S108, the second storage step includes the steps of, when the population information received in the population receiving step is the first population information, if the second quality information satisfies the first condition, storing at least a portion of one or more patent documents included in the second patent document in association with the second output result, and if the second quality information does not satisfy the first condition, not storing at least a portion of the one or more patent documents included in the second patent document in association with the second output result, and when the population information received in the population receiving step is the second population information, if the second quality information satisfies the second condition, storing at least a portion of the one or more patent documents included in the second patent document in association with the second output result, and if the second quality information does not satisfy the second condition, not storing at least a portion of the one or more patent documents included in the second patent document in association with the second output result. Specifically, when step S108 is executed again (when executed for the second or subsequent time) after document processing has been executed up to step S109, the accuracy threshold S for the first time may be set to a different value than the accuracy threshold T for the second time. The document processing unit 1042 of the server 10 may store, in the second support table 1016, second output results that satisfy a second condition based on the accuracy thresholds (S, T) among the second output results output for the multiple patent documents included in the second patent document, and may not store in the second support table 1016 second output results that do not satisfy the second condition. The document processing unit 1042 of the server 10 may delete records of second support information stored in the second support table 1016 whose second output results do not satisfy specified conditions (only the second document ID is stored, and the second output value and accuracy values are blank).

[0111] The document processing unit 1042 of the server 10 may search the application condition items of the parameter master 1022 and acquire the accuracy thresholds (S, T). For example, the number of document processing executions may be set as an application condition, and the accuracy thresholds (S, T) corresponding to the number of executions may be acquired. For example, the number of document processing executions (first time, second time, etc.) may be set as an application condition, and the accuracy thresholds (S, T) corresponding to the number of executions may be acquired.

[0112] For example, the first condition may include the second condition. For example, the accuracy threshold for the first condition may be 0.5 or more, and the accuracy threshold for the second condition may be 0.7 or more. As a result, when the first process and the second process are executed multiple times, the conditions for the quality of the second process result become gradually stricter, thereby improving the final process quality.

[0113] For example, the second condition may include the first condition. For example, the accuracy threshold for the first condition may be 0.7 or more, and the accuracy threshold for the second condition may be 0.5 or more. As a result, when the first and second processes are executed multiple times, the requirements for the quality of the second process result are gradually relaxed, which can shorten the total processing time and reduce processing costs.

[0114] <Unprocessed document determination step> In step S109, the document processing unit 1042 of the server 10 executes an unprocessed document determination step to determine whether or not there are any patent documents included in the population received in step S101 that are not stored in association with the first output result, training data, or second output result. Specifically, in step S101, the document processing unit 1042 of the server 10 searches and acquires the second document ID field in the second support table 1016 based on the document IDs of the multiple patent documents included in the population data stored in the population table 1014. At this time, only records in the second support table 1016 in which a value is stored in the second output value field are targeted for search. The document processing unit 1042 of the server 10 compares the document IDs of the multiple patent documents included in the population data with the acquired second document ID, and determines that there are no unprocessed documents if all of the document IDs of the multiple patent documents included in the population data are included in the acquired second document ID. The document processing unit 1042 of the server 10 may acquire the teacher document ID field of the teacher table 1017, compare the document IDs of the multiple patent documents included in the population data with the acquired second document ID and teacher document ID, and determine that there are no unprocessed documents if all of the document IDs of the multiple patent documents included in the population data are included in the acquired second document ID and teacher document ID. In other words, patent documents to which teacher labels have been assigned may be excluded from the determination of unprocessed documents. The document processing unit 1042 of the server 10 may acquire the first document ID item of the first support table 1015, compare the document IDs of the multiple patent documents included in the population data with the acquired second document ID and first document ID, and determine that there are no unprocessed documents if all of the document IDs of the multiple patent documents included in the population data are included in the acquired second document ID and first document ID. In other words, patent documents for which the first output value is stored in the first support table 1015 may be excluded from the determination of unprocessed documents.

[0115] In step S109, if the document processing unit 1042 of the server 10 determines that there are no unprocessed documents, the process proceeds to step S110. On the other hand, if it determines that there are unprocessed documents, document processing (second or subsequent times) is executed again from step S101. Specifically, the population creation step (second or subsequent times) is executed in step S101. That is, the document processing unit 1042 of the server 10 automatically iterates steps S101 to S109 as long as there are unprocessed documents in the document creation process.

[0116] <Ending step> In step S110, the document processing unit 1042 of the server 10 executes a processing termination step to terminate processing if there are no patent documents in the population information received in the population reception step, excluding one or more patent documents stored in association with information based on the first output result in the first storage step and one or more patent documents stored in association with the second output result in the second storage step. As a result, the first process, the second process, etc. are repeatedly performed until information based on the first process result (teaching data) and the second process result are associated and stored for all patent documents included in the population received in the population reception step. The document processing unit 1042 of the server 10 repeatedly executes document processing steps S101 to S109, and the processing results for all patent documents included in the population data are stored in the teacher table 1017, the second support table 1016, and the first support table 1015. The document processing disclosed herein allows for suitable processing to be applied to a population containing multiple patent documents using a small amount of training data or no training data at all, without the need for a large amount of training data. The first processing using a large-scale language model tends to be more expensive and time-consuming than the second processing. The present disclosure eliminates the need to apply the first processing to all patent documents in the population, thereby saving processing time and costs.

[0117] <Basic computer hardware configuration> 14 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0118] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0119] The main memory device 902 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0120] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0121] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0122] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0123] <Basic functional configuration of computer 90> The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 14) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0124] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0125] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0126] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0127] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0128] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0129] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0130] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0131] (Appendix 1) A program to be executed by a computer having a processor and a memory unit, wherein the processor executes the following steps: a population reception step (S101) of receiving population information regarding a population including a plurality of patent documents; a first patent extraction step (S102) of extracting a first patent document from the plurality of patent documents included in the population received in the population reception step; a first processing step (S104) of obtaining a first output result that is output in response to inputting a prompt including the first patent document extracted in the first patent extraction step into a large-scale language model; a second patent extraction step (S107) of extracting a second patent document from the plurality of patent documents included in the population received in the population reception step; and a second processing step (S108) of obtaining a second output result that is output in response to inputting the second patent document extracted in the second patent extraction step into a learning model trained using information based on the first output result obtained in the first processing step as training data. This makes it possible to apply suitable processing to a population containing multiple patent documents using a small amount of training data, or no training data at all, without preparing a large amount of training data. The first process using a large-scale language model tends to be more expensive and time-consuming than the second process. Because the first process does not need to be applied to all patent documents in the population, processing time and costs can be saved.

[0132] (Appendix 2) The program described in Appendix 1, which executes a second processing step (S108) including a step of acquiring second quality information indicating the quality of processing for one or more patent documents included in the second patent document that is output in response to inputting the second patent document into the learning model, and a second storage step (S108) in which the processor stores at least a portion of the one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step satisfies a predetermined condition, and does not store at least a portion of the one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step does not satisfy the predetermined condition. This allows the second patent document to be processed using information based on the first output result as training data. Only the second processing results obtained by the learning model that are of good quality can be stored in association with the second patent document.

[0133] (Appendix 3) The program described in Appendix 2, wherein the population receiving step (S101) includes a step of receiving first population information regarding a first population, and a step of receiving second population information regarding a second population including one or more patent documents from the first population information excluding one or more patent documents stored in association with the second output result in the second storage step. This makes it possible to perform the first and second processes again on patent documents for which the quality of the second processing results from the learning model is not satisfactory. Since patent documents for which the quality of the second processing results is favorable are gradually removed from the initially given population, the first processing and the second processing can be gradually applied to multiple patent documents included in the initially given population. By applying the second process to the population as much as possible, it is possible to suppress the application of the first process.

[0134] (Appendix 4) A program as described in Appendix 2, wherein the processor executes a first storage step (S106) in which at least some or all of the first patent documents extracted in the first patent extraction step are stored in association with information based on the first output result, and the population reception step (S101) includes the steps of receiving first population information regarding the first population, and receiving second population information regarding a second population including one or more patent documents excluding from the first population information one or more patent documents stored in association with information based on the first output result in the first storage step and one or more patent documents stored in association with the second output result in the second storage step. This allows the first and second processes to be performed again on patent documents excluding the first output result or patent documents to which training data has been provided based on the first output result, and patent documents for which the quality of the second processing result is suitable. Patent documents that have been given training data or for which the quality of the second processing results is favorable are gradually removed from the initially given population, so that the first and second processing can be gradually applied to multiple patent documents included in the initially given population. By applying the second process to the population as much as possible, it is possible to suppress the application of the first process.

[0135] (Appendix 5) The program described in Appendix 3, wherein the first patent extraction step (S102) includes a step of extracting a number of first patent documents according to a first ratio from the first population information, and a step of extracting a number of first patent documents according to a second ratio from the second population information. This makes it possible to control the extraction ratio of the first patent document when the first process and the second process are executed multiple times. By controlling the convergence of the process, the total processing time can be shortened, and by reducing the proportion of patent documents to which the first process is applied, processing costs can be reduced.

[0136] (Appendix 6) 6. The program of claim 5, wherein the second rate is greater than the first rate. For example, if there are many patent documents for which the quality of the second processing result is not satisfactory, the convergence of the processing may be poor. By making the second ratio larger than the first ratio, the proportion of patent documents processed by the large-scale language model in the population increases, thereby increasing the amount of training data used in the second processing and reducing the proportion of patent documents for which the quality of the second processing result is not satisfactory. This may reduce the number of processing times in the processing disclosed herein, thereby shortening the total processing time.

[0137] (Appendix 7) 6. The program of claim 5, wherein the second rate is less than the first rate. If the second proportion is smaller than the first proportion, the proportion of patent documents in the population that are processed by the large-scale language model decreases, thereby making it possible to save on processing costs associated with the first processing.

[0138] (Appendix 8) The program described in Appendix 3, wherein the second storage step (S108) includes the steps of, when the population information received in the population receiving step is the first population information, storing at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information satisfies the first condition, and not storing at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information does not satisfy the first condition, and when the population information received in the population receiving step is the second population information, storing at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information satisfies the second condition, and not storing at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information does not satisfy the second condition. This makes it possible to control the conditions for determining whether the quality of the second processing result is suitable when the first processing and the second processing are executed multiple times. For example, by making the conditions more lenient, the total processing time can be shortened and processing costs can be reduced, while by making the conditions more strict, processing quality can be improved.

[0139] (Appendix 9) 10. The program according to claim 8, wherein the first condition includes the second condition. As a result, when the first process and the second process are executed multiple times, the conditions for the quality of the second process result become gradually stricter, thereby improving the final process quality.

[0140] (Appendix 10) The second condition is a program according to Appendix 8, which includes the first condition. As a result, when the first and second processes are executed multiple times, the requirements for the quality of the second process result are gradually relaxed, which can shorten the total processing time and reduce processing costs.

[0141] (Appendix 11) A program described in Appendix 4, in which a processor executes a processing termination step (S110) that terminates processing when there are no patent documents among the population information received in the population reception step, excluding one or more patent documents stored in association with information based on the first output result in the first storage step and one or more patent documents stored in association with the second output result in the second storage step. As a result, the first process, the second process, etc. are repeatedly performed until information based on the first process result and the second process result are associated and stored for all patent documents included in the population received in the population reception step.

[0142] (Appendix 12) The program described in Appendix 1, wherein the first processing step (S104) is a step of acquiring first classification information regarding the classification of the first patent document, which is output in response to inputting a prompt including the first patent document and a classification definition sentence that specifies the classification rules into a large-scale language model, and the second processing step (S108) is a step of acquiring second classification information regarding the classification of the second patent document, which is output in response to inputting the second patent document into a learning model that is a classification model. This makes it possible to classify multiple patent documents contained in a population containing multiple patent documents using a small amount of training data, or no training data at all, without preparing a large amount of training data. The first process using a large-scale language model tends to be more expensive and time-consuming than the second process. Because the first process does not need to be applied to all patent documents in the population, processing time and costs can be saved.

[0143] (Appendix 13) A program as described in Appendix 12, wherein a processor executes a third patent extraction step (S103) of extracting a third patent document from a plurality of patent documents included in a population received in a population receiving step, a classification perspective acquisition step (S103) of acquiring classification perspective information related to the classification perspective of the third patent document, which is output in response to inputting into a large-scale language model a prompt including the third patent document extracted in the third patent extraction step and a classification perspective sentence that defines the classification perspective, and a classification definition acquisition step (S103) of acquiring classification definition sentences obtained by generalizing the classification perspective information related to the third patent document, which is output in response to inputting into the large-scale language model a prompt including the classification perspective information acquired in the classification perspective acquisition step, wherein a first processing step (S104) is a step of acquiring first classification information based on the classification definition sentences acquired in the classification definition acquisition step. This makes it possible to classify a plurality of patent documents contained in a population that includes a plurality of patent documents without preparing a classification definition.

[0144] (Appendix 14) The program described in Appendix 12, wherein the second processing step (S108) includes a step of acquiring second quality information indicating the quality of classification for one or more patent documents included in the second patent document that is output in response to inputting the second patent document into the learning model, and the processor executes a second classification storage step (S108) of storing at least a portion of the one or more patent documents included in the second patent document in association with the second classification information if the second quality information acquired in the second processing step satisfies a predetermined condition. This allows the second patent document to be classified using information based on the first output result as training data. Only those documents for which the quality of classification by the classification model is favorable can be stored in association with the second patent document.

[0145] (Appendix 15) A program as described in Appendix 1, wherein the processor executes a first presentation step (S105) of presenting to the user the first output result obtained in the first processing step, and a teacher data creation step (S106) of receiving from the user an input operation on the first output result presented in the first presentation step and creating teacher data related to the first patent document in accordance with the input operation, and a second processing step (S108) of obtaining a second output result that is output in accordance with inputting the second patent document into a learning model learned based on the teacher data created in the teacher data creation step. This allows the user to create training data with less work load by referring to the first output result. Compared to using the first output result as training data as is, the second processing is performed based on training data created and edited by the user, so higher quality processing results are expected.

[0146] (Appendix 16) The program described in Appendix 1, wherein the first patent extraction step (S102) includes a step of dividing the multiple patent documents included in the population received in the population reception step into multiple groups, and a step of extracting one or more patent documents as first patent documents from each of the multiple groups. This allows the first process to be applied in a balanced manner across the entire expression space of the patent documents included in the population, even if the contents of the patent documents included in the population are biased, by sampling a predetermined number of patent documents for each group. For example, the user can prepare training data in a balanced manner across the entire expression space of the patent documents included in the population. This allows the second process to be performed with high quality and also saves processing time and costs. The number of first patent documents extracted for each group may be a predetermined fixed number, or a predetermined number of first patent documents may be extracted according to the number of patent documents included in the group or the proportion of the number of documents in each group.

[0147] (Appendix 17) A computer-implemented method comprising a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of appendices 1 to 16. This makes it possible to apply suitable processing to a population containing multiple patent documents using a small amount of training data, or no training data at all, without preparing a large amount of training data. The first process using a large-scale language model tends to be more expensive and time-consuming than the second process. Because the first process does not need to be applied to all patent documents in the population, processing time and costs can be saved.

[0148] (Appendix 18) An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of Supplementary Note 1 to Supplementary Note 16. This makes it possible to apply suitable processing to a population containing multiple patent documents using a small amount of training data, or no training data at all, without preparing a large amount of training data. The first process using a large-scale language model tends to be more expensive and time-consuming than the second process. Because the first process does not need to be applied to all patent documents in the population, processing time and costs can be saved.

[0149] (Appendix 19) A system comprising means for performing all steps performed in any of the inventions according to any one of appendixes 1 to 16. This makes it possible to apply suitable processing to a population containing multiple patent documents using a small amount of training data, or no training data at all, without preparing a large amount of training data. The first process using a large-scale language model tends to be more expensive and time-consuming than the second process. Because the first process does not need to be applied to all patent documents in the population, processing time and costs can be saved. [Explanation of symbols]

[0150] 1 System, 10 Server, 101 Memory Unit, 104 Control Unit, 106 Input Device, 108 Output Device, 20 User Terminal, 201 Memory Unit, 204 Control Unit, 206 Input Device, 208 Output Device, 40 Artificial Intelligence System, 401 Memory Unit, 404 Control Unit, 406 Input Device, 408 Output Device

Claims

1. A program to be executed by a computer having a processor and a storage unit, the processor: a population receiving step of receiving population information regarding a population including a plurality of patent documents; a first patent extraction step of extracting a first patent document from the plurality of patent documents included in the population received in the population receiving step; a first processing step of obtaining a first output result output in response to inputting a prompt including the first patent document extracted in the first patent extraction step into a large-scale language model; a second patent extraction step of extracting a second patent document from the plurality of patent documents included in the population received in the population receiving step; a second processing step of acquiring a second output result output in response to inputting the second patent document extracted in the second patent extraction step into a learning model trained using information based on the first output result acquired in the first processing step as training data; A program that executes.

2. the second processing step includes a step of acquiring second quality information indicating a quality of processing of one or more patent documents included in the second patent document output in response to inputting the second patent document into the learning model; the processor: If the second quality information acquired in the second processing step satisfies a predetermined condition, at least a part of one or more patent documents included in the second patent document is stored in association with the second output result; If the second quality information acquired in the second processing step does not satisfy a predetermined condition, at least a part of one or more patent documents included in the second patent document is not stored in association with the second output result. A second storing step; To execute The program according to claim 1.

3. The population receiving step includes: receiving first population information relating to a first population; receiving second population information regarding a second population including one or more patent documents from the first population information, excluding one or more patent documents stored in association with the second output result in the second storing step; Including, The program according to claim 2.

4. the processor: a first storage step of storing at least a part or all of the first patent documents extracted in the first patent extraction step in association with information based on the first output result; Run The population receiving step includes: receiving first population information relating to a first population; From the first population information, one or more patent documents stored in association with information based on the first output result in the first storage step; one or more patent documents stored in association with the second output result in the second storage step; receiving second population information regarding a second population including one or more patent documents excluding Including, The program according to claim 2.

5. The first patent extraction step extracting the first patent documents in a number corresponding to a first ratio from the first population information; extracting the first patent documents in a number corresponding to a second ratio from the second population information; Including, The program according to claim 3.

6. The second ratio is greater than the first ratio. The program according to claim 5.

7. The second rate is less than the first rate. The program according to claim 5.

8. The second storing step When the population information received in the population receiving step is the first population information, If the second quality information satisfies a first condition, storing at least a portion of one or more patent documents included in the second patent document in association with the second output result; If the second quality information does not satisfy the first condition, at least a part of one or more patent documents included in the second patent document is not stored in association with the second output result. When the population information received in the population receiving step is the second population information, If the second quality information satisfies a second condition, storing at least a portion of one or more patent documents included in the second patent document in association with the second output result; If the second quality information does not satisfy the second condition, at least a part of one or more patent documents included in the second patent document is not stored in association with the second output result. Including steps, The program according to claim 3.

9. the first condition includes the second condition, The program according to claim 8.

10. the second condition includes the first condition, The program according to claim 8.

11. the processor: Among the population information received in the population receiving step, one or more patent documents stored in association with information based on the first output result in the first storage step; one or more patent documents stored in association with the second output result in the second storage step; a processing termination step in which the processing is terminated if there is no patent document other than To execute The program according to claim 4.

12. the first processing step is a step of acquiring first classification information regarding the classification of the first patent document, which is output in response to inputting the first patent document and a prompt including a classification definition sentence in which a classification rule is defined, into the large-scale language model; the second processing step is a step of acquiring second classification information regarding the classification of the second patent document, which is output in response to inputting the second patent document into the learning model, which is a classification model; The program according to claim 1.

13. the processor: a third patent extraction step of extracting a third patent document from the plurality of patent documents included in the population received in the population receiving step; a classification viewpoint acquisition step of acquiring classification viewpoint information regarding the classification viewpoint of the third patent document, which is output in response to a prompt input including the third patent document extracted in the third patent extraction step and a classification viewpoint sentence in which a classification viewpoint is defined, into a large-scale language model; a classification definition acquisition step of acquiring the classification definition sentence obtained by aggregating the classification viewpoint information regarding the third patent document, the classification definition sentence being output in response to inputting a prompt including the classification viewpoint information acquired in the classification viewpoint acquisition step into a large-scale language model; Run the first processing step is a step of acquiring the first classification information based on the classification definition sentence acquired in the classification definition acquisition step; The program according to claim 12.

14. the second processing step includes a step of acquiring second quality information indicating a quality of classification of one or more patent documents included in the second patent document output in response to inputting the second patent document into the learning model; the processor: a second classification storage step of storing at least a portion of one or more patent documents included in the second patent document in association with the second classification information when the second quality information acquired in the second processing step satisfies a predetermined condition; To execute The program according to claim 12.

15. the processor: a first presentation step of presenting the first output result acquired in the first processing step to a user; a training data creation step of receiving an input operation from the user for the first output result presented in the first presentation step, and creating training data related to the first patent document in accordance with the input operation; Run the second processing step is a step of acquiring the second output result output in response to inputting the second patent document into the learning model trained based on the training data created in the training data creation step; The program according to claim 1.

16. The first patent extraction step a step of dividing the plurality of patent documents included in the population received in the population receiving step into a plurality of groups; extracting one or more patent documents from each of the plurality of groups as the first patent document; Including, The program according to claim 1.

17. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 16.

18. 17. An information processing device comprising a control unit and a storage unit, wherein the control unit executes all of the steps executed in the invention according to any one of claims 1 to 16.

19. A system comprising means for executing all steps performed in any one of the inventions according to claims 1 to 16.

Images