Programs, methods, information processing devices, systems

Abstract

The present invention provides a program, method, information processing device, and system that apply a suitable processing method to multiple patent documents without requiring the preparation of a large amount of training data. [Solution] The method involves: a population reception step of receiving population information relating to a population containing multiple patent documents; a first patent extraction step of extracting a first patent document from among the multiple patent documents included in the received population; a first processing step of obtaining a first output result that is output in response to inputting a prompt containing the extracted first patent document to a large-scale language model; a second patent extraction step of extracting a second patent document from among the multiple patent documents included in the population received in the population reception step; and a second processing step of obtaining a second output result that is output in response to inputting the second patent document extracted in the second patent extraction step to a learning model that has been trained using information based on the first output result obtained in the first processing step as training data.

Description

[[Technical Field]]

[0001] The present disclosure relates to a program, a method, an information processing apparatus, and a system. [[Background Art]]

[0002] Technologies for supporting patent operations are known. Patent Document 1 discloses a technology for supporting understanding of claims, which specifies and presents key constituent elements in a claim so that a user can focus on reading the description thereof. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Publication No. 2011-096200 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] When processing patents using a deep learning model, there is a problem that a large amount of training data needs to be prepared. Accordingly, the present disclosure has been made to solve the above problem, and an object of the present disclosure is to provide a technology for applying suitable processing to a plurality of patent documents without preparing a large amount of training data. [[Means for Solving the Problem]]

[0005] A program to be executed by a computer having a processor and a memory unit, wherein the processor executes: a population reception step of receiving population information relating to a population including a plurality of patent documents; a first patent extraction step of extracting a first patent document from among the plurality of patent documents included in the population received in the population reception step; a first processing step of obtaining a first output result output in response to inputting a prompt including the first patent document extracted in the first patent extraction step to a large-scale language model; a second patent extraction step of extracting a second patent document from among the plurality of patent documents included in the population received in the population reception step; and a second processing step of obtaining a second output result output in response to inputting the second patent document extracted in the second patent extraction step to a learning model that has been trained using information based on the first output result obtained in the first processing step as training data. [Effects of the Invention]

[0006] According to this disclosure, suitable processing can be applied to multiple patent documents without having to prepare a large amount of training data. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing the functional configuration of System 1. [Figure 2] This block diagram shows the functional configuration of Server 10. [Figure 3] This is a block diagram showing the functional configuration of user terminal 20. [Figure 4] This diagram shows the data structure of user table 1012. [Figure 5] This diagram shows the data structure of document table 1013. [Figure 6] This diagram shows the data structure of population table 1014. [Figure 7] This diagram shows the data structure of the first support table 1015. [Figure 8] This diagram shows the data structure of the second support table 1016. [Figure 9] This diagram shows the data structure of the teacher table 1017. [Figure 10] This diagram shows the data structure of Task Master 1021. [Figure 11] This diagram shows the data structure of parameter master 1022. [Figure 12] This is a flowchart showing the document processing operation. [Figure 13] This is an example screen showing the operation of the population creation processing page. [Figure 14] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0008] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0009] <System 1 Configuration> System 1 in this disclosure is an information processing system that provides support services for processing patent documents. System 1 comprises a server 10, a user terminal 20, and an artificial intelligence system 40, all connected via a network N. Figure 1 is a block diagram showing the functional configuration of System 1. Figure 2 is a block diagram showing the functional configuration of server 10. Figure 3 is a block diagram showing the functional configuration of the user terminal 20.

[0010] Each information processing device is configured by a computer including an arithmetic unit and a storage device. The basic hardware configuration of a computer and the basic functional configuration of a computer realized by said hardware configuration will be described later. Descriptions overlapping the basic hardware configuration of a computer and the basic functional configuration of a computer described later will be omitted for each of the server 10, the user terminal 20, and the artificial intelligence system 40.

[0011] <Configuration of Server 10> The server 10 is an information processing device that provides patent document processing support services. The server 10 includes a storage unit 101 and a control unit 104.

[0012] <Configuration of Storage Unit 101 of Server 10> The storage unit 101 of the server 10 includes an application program 1011, a user table 1012, a document table 1013, a population table 1014, a first support table 1015, a second support table 1016, a training table 1017, a task master 1021, and a parameter master 1022.

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. The application program 1011 includes applications such as a web browser application.

[0014] The user table 1012 is a table that stores and manages information of member users (hereinafter referred to as users) who use the service. When a user completes service usage registration, the user's information is stored in a new record of the user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having a user ID as a primary key, and columns for a user ID and a user name. Figure 4 is a diagram showing the data structure of the user table 1012.

[0015] The User ID is an item that stores user identification information to identify a user. User identification information is an item that is set to a unique value for each user. The username field is used to store the user's real name. However, the username can also be a nickname or any other string of characters.

[0016] Document table 1013 is a table for storing and managing information (document information) related to patent documents. Document table 1013 is a table that has columns for document ID, user ID, document name, and document content. Figure 5 shows the data structure of document table 1013.

[0017] The document ID is an item that stores document identification information for identifying a patent document. The User ID is an item that stores user identification information used to identify a user. The document name is an item that stores the name of the patent document. The document name can be set to any string of characters. For example, the document name may store the publication number, registration number, application number, or other reference number of the patent document. The "Document Content" field stores the content of the patent document. Specifically, it stores the text information of the patent document. For example, the "Document Content" field includes some or all of the "Claims," ​​"Abstract," "Detailed Description of the Invention," "Examples," and "Embodiments" contained in the patent document (specification). Specifically, it stores the text of publications such as published gazettes and patent gazettes.

[0018] The population table 1014 is a table for storing and managing a population (population information) that includes multiple patent documents. In this disclosure, it stores information that identifies a population consisting of multiple patent documents that are the subject of various information processing. Population table 1014 is a table with Population ID as the primary key and containing columns for Population ID, Population Data, Task ID, and Query Data. Figure 6 shows the data structure of the population table 1014.

[0019] The Population ID is an item that stores population identification information used to identify a population. Each population identification information field has a unique value assigned to it. Population data is an item that stores information to identify a population consisting of document IDs of multiple patent documents. Population data may also be configured to store not only document IDs, but also document names, document contents, etc., associated with the document IDs. Furthermore, it is not always necessary to identify multiple patent documents by document ID; a population consisting of multiple patent documents may also be identified by a search expression, search conditions, or logical expression used to search for patents. The Task ID is an item that stores task identification information indicating a task to be applied to a population that includes multiple patent documents identified by the population identification information. Query data is an item that stores queries applied to a population that includes multiple patent documents identified by population identification information. Specifically, query data stores queries such as the following, depending on the type of task. • Classification assignment task: The system stores a text (string) that defines the item to be classified, and a list of multiple exemplary texts (strings) that represent the item to be classified. Example: Classification A is "...", Classification B is "...", Classification C is "..." • Screening: A text (string of characters) describing the target invention is stored. Example) A device comprising... means,... means, and... • Classification determination: A sentence (string of characters) describing the target product is stored. Example) A device comprising... means,... means, and... • Invention evaluation: A document describing the evaluation items and criteria is stored. Example) Innovation: What is A rank...? What is B rank...? What is C rank...? Economic efficiency: What is A rank...? What is B rank...? What is C rank...?

[0020] The first support table 1015 is a table for storing and managing prompts related to inquiry texts to the artificial intelligence system and the content of the responses from the artificial intelligence system (first support information). The first support table 1015 is a table that has columns for the first document ID, prompt, and first output value. Figure 7 shows the data structure of the first support table 1015.

[0021] The first document ID is an item that stores document identification information for identifying the first patent document. A prompt is an item that stores prompts related to query texts to an artificial intelligence system. A prompt is primarily a query text (text) input to an artificial intelligence system. Specifically, a user can input prompts to the artificial intelligence system so that it produces the desired output. Note that prompts do not have to be text strings; they can also be images, videos, audio, etc. For example, gestures and voice instructions from the user can also be prompts. The first output value is an item that stores the output data from the artificial intelligence system in response to a prompt. Specifically, it stores output data (string information) corresponding to the task information used when generating the prompt.

[0022] The second support table 1016 is a table for storing and managing output data (second support information) from the classification model 1051 for patent documents. Specifically, the second support table 1016 stores information such as classification labels and classification accuracy that are output when a patent document is applied to the classification model 1051 as input data. In this disclosure, the classification results by the classification model are explained as an example, but instead of classification results, the system may also store screening results by a screening model, compliance determination results by a compliance determination model, and invention evaluation results by an invention evaluation model. The second support table 1016 is a table that has columns for second document ID, second output value, and precision. Figure 8 shows the data structure of the second support table 1016.

[0023] The second document ID is an item that stores document identification information for identifying the second patent document. The second output value is an item that stores output data (string information) such as classification labels, which are output in response to applying the patent document identified by the second document ID to the classification model 1051 as input data. The second output value may store multiple sets of output data such as classification labels. The accuracy item stores information indicating the quality of the second output value, such as the classification accuracy for the classification labels output when the patent document identified by the second document ID is applied to the classification model 1051 as input data. The accuracy item may also store information indicating the quality for each of multiple classification labels.

[0024] The teacher table 1017 is a table for storing and managing teacher information for patent documents. Teacher table 1017 is a table that has columns for Teacher Document ID and Teacher Label. Figure 9 shows the data structure of the teacher table 1017.

[0025] The teacher document ID is an item that stores document identification information for identifying a patent document. The teacher label is an item that stores teacher data (string information) such as classification labels for patent documents identified by private document IDs today.

[0026] Task Master 1021 is a table for storing and managing information (task information) about tasks performed on patent documents. Task Master 1021 is a table with Task ID as the primary key, and has columns for Task ID, Task Type, Extraction Item, and Task Data. Figure 10 shows the data structure of task master 1021.

[0027] The Task ID is an item that stores task identification information to identify a task. Task identification information is an item with a unique value assigned to each task. The task type is an item that stores task type information related to the type of task. Specifically, the task type stores a string that identifies the content of the task according to the purpose of the processing (task) performed on the patent document. In this disclosure, the task type stores information such as "classification assignment," "screening," "applicability determination," and "invention evaluation" according to the content of the following tasks. Note that the above tasks are examples in this disclosure, and this disclosure is applicable to any task on a patent document. • Classification assignment task: This task involves assigning classifications to multiple patent documents according to their respective contents. • Screening task: This task involves determining whether each of several patent documents is related to the target document. Specifically, the task may be subdivided into tasks such as "prior art search," "prior art search," and "invalidation search." • Determination of applicability task: This task involves determining whether each of multiple patent documents is related to the target product. For example, it is a task performed in patent work such as "clearance investigation," "FTO investigation," and "infringement investigation." • Invention Evaluation Task: This task involves evaluating the value of inventions related to patent documents, such as their innovativeness, economic efficiency, practicality, applicability by other companies, and ease of infringement detection, for each of multiple patent documents. For example, it is a task performed in patent work such as "determining whether to file an application," "considering foreign applications and the countries in which to file," and "determining whether to maintain a registered patent." Extraction items are items that store information (extraction item information) indicating the extraction items of a patent document when performing a task. Specifically, extraction items include strings that indicate the extraction items of a patent document that provide tasks such as "abstract," "specification," "claims," ​​and "drawings." Extraction items also include strings that indicate the extraction items of a patent document that provide tasks such as "title of invention," "detailed description of invention," "technical field," "background art," "prior art documents," "problem to be solved by the invention," "means for solving the problem," "effects of the invention," "brief description of the drawings," "modes for carrying out the invention," "examples," "industrial applicability," "explanation of symbols, etc.," "problem," and "solution." Specifically, the extracted items are stored according to the task type, with the following information being collected: • Classification: Abstract, claims, independent claims, claim 1, specification, etc. • Screening: Abstract, claims, independent claims, claim 1, specification, etc. • Determination of applicability: Claim 1, Patent Claims, Independent Claim • Invention evaluation: Claim 1, Independent Claim Task data is an item that stores information (strings) related to a task. Specifically, task data stores strings like the following depending on the task type. Note that the <Query> item will contain query data, which will be described later, and the <Extracted Document> item will contain the extracted portion (extracted content, partial text) from the text contained in the patent document, which is identified based on the extraction item, and a prompt will be generated. • When the task type is "Classification Assignment" ' The following technical classifications are defined. Please classify the invention described in the "Target Patent" below into one of the following classifications: A, B, or C. Please calculate the degree of relevance between the target patent and the classification on a scale of 0 to 100. Please output the reason for your judgment. #Technical classification: <Query> (Category A "...", Class B "...", Class C "...") #Target Patent: <Extracted Document> (Abstract and claims sections (content) from the patent document) 』 • If the task type is "Screening" ' Please examine whether the invention described in the "Target Invention" below is described in the invention described in the "Target Patent" below. Calculate the degree of relevance between the Target Invention and the Target Patent on a scale of 0 to 100. Output the reason for your determination. #Target Invention: <Query> (An apparatus comprising... means,... means, and...) #Target Patent: <Extracted Document> (Abstract, Claims, and Specification sections (content) from the patent document) 』 • When the task type is "Applicability Determination" ' Please consider whether the products, services, etc. listed under "Target Products" below fall within the scope of the rights of the inventions listed under "Target Patents" below. Calculate the degree of relevance between the target products and the target patents on a scale of 0 to 100. Output the reason for your determination. #Target product:<Query>(a device comprising... means,... means, and...) #Target Patent: <Extracted Document> (The portion of Claim 1 in the patent document (content)) 』 • Task type is "Invention Evaluation" ' The evaluation criteria are defined as follows: Please evaluate the invention described in the "Target Patent" below as A, B, or C. Please output the reason for your judgment. #Evaluation criteria: <Query> (Evaluation A "...", Evaluation B "...", Evaluation C "...") #Target Patent: <Extracted Document> (Abstract and claims sections (content) from the patent document)

[0028] The parameter master 1022 is a table for storing and managing information about parameters (parameter information). The parameter master 1022 is a table having columns for extraction rate, precision threshold, and application conditions. Figure 11 shows the data structure of the parameter master 1022.

[0029] The sampling rate is an item that stores the sampling ratio used to identify the first patent document, which consists of multiple patent documents, from the population. Specifically, the sampling rate can take values ​​between 0 and 100%. The precision threshold is an item that stores conditions related to information indicating the quality of the second output value, such as the precision, when identifying a second patent set consisting of one or more second patent documents. For example, based on the second output value and precision output by the classification model 1051, only those whose precision is greater than the precision threshold are identified and stored as second patent documents. Specifically, the precision threshold takes a value between 0 and 1. The application conditions are items that store the application conditions when applying the extraction rate and precision threshold. Specifically, the application conditions store conditions related to the number of times document processing has been performed, the number of unprocessed documents, the percentage, etc.

[0030] <Configuration of the control unit 104 of server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, a document processing unit 1042, and a classification model 1051. The control unit 104 realizes each functional unit by executing the application program 1011 stored in the storage unit 101.

[0031] The user registration control unit 1041 processes information of users who wish to use the services related to this disclosure and stores it in the user table 1012. Information stored in the user table 1012 is obtained when a user opens a web page operated by the service provider from any information processing terminal, enters information into a designated input form, and sends it to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, and user registration is completed. As a result, users stored in the user table 1012 can use the service. Prior to the registration of user information in the user table 1012 by the user registration control unit 1041, the service provider may perform a prescribed review and restrict whether or not the user can use the service. The user ID can be any string or number that can identify the user, and may be any string or number desired by the user, or the user registration control unit 1041 may automatically set any string or number.

[0032] The document processing unit 1042 performs document processing. Details will be described later.

[0033] Classification model 1051 is an item that stores a classification model for classifying patent documents. The classification model 1051 includes any learning model such as a machine learning model, a deep learning model, or an artificial intelligence model. In this disclosure, the classification model 1051 includes a learning model that can output a classification name (classification label) and accuracy (inference accuracy, information indicating the quality of inference) corresponding to the classification name, in response to input of a document (text data) such as a patent document. For example, the classification model 1051 includes Bayesian classification, SVM (Support Vector Machines), decision trees, random forests, logistic regression, deep learning models, and BERT. Furthermore, the classification model 1051 may include image classification models such as CNN, ResNet, Inception, and VGGNet. Since patent documents include images such as drawings, the system may be configured to output classification names according to such images. Classification model 1051 can be implemented using many general-purpose libraries used for natural language processing. For example, classification model 1051 can be implemented using NLTK (Natural Language Toolkit), spaCy, Scikit-learn, TensorFlow and Keras, PyTorch, FastText, Gensim, and others.

[0034] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user of the service. The user terminal 20 may be, for example, a mobile device such as a smartphone or tablet, or a stationary PC (Personal Computer) or laptop PC. It may also be a wearable device such as an HMD (Head Mount Display) or a smartwatch. The user terminal 20 includes a storage unit 201, a control unit 204, an input device 206, and an output device 208.

[0035] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012.

[0036] User ID 2011 is the user's account ID. The user sends User ID 2011 from user terminal 20 to server 10. Server 10 identifies the user based on User ID 2011 and provides the services related to this disclosure to the user. User ID 2011 includes information such as a session ID that is temporarily assigned by server 10 to identify the user using user terminal 20.

[0037] The application program 2012 may be pre-stored in the memory unit 201, or it may be configured to be downloaded from a web server operated by the service provider via a communication interface. Application Program 2012 includes applications such as web browser applications. Application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20.

[0038] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 comprises an input control unit 2041 and an output control unit 2042. The control unit 204 realizes each functional unit by executing an application program 2012 stored in the memory unit 201.

[0039] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, and a touch device 2065.

[0040] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082.

[0041] <Configuration of Artificial Intelligence System 40> The artificial intelligence system 40 is an information processing device that outputs the content of the response to a prompt. For example, the artificial intelligence system 40 includes ChatGPT, OpenAI GPT, PerplexityAsk, BingAI, etc. These artificial intelligence systems have a dialogue (chat) function, and users can obtain answers to inquiries or commands by giving the artificial intelligence system arbitrary inquiries or commands in text. In this disclosure, the user can obtain text that assists in reading patent documents as the response content by sending a prompt created in the inquiry processing to the artificial intelligence system 40. Furthermore, in this disclosure, the artificial intelligence system is not limited to text-based dialogue. For example, it may be an image generation AI system such as Midjourney or Stable Diffusion. For instance, a user can send a prompt created during inquiry processing to such an image generation AI system to obtain images or videos as responses that assist in reading patent documents. Furthermore, this disclosure is also applicable to artificial intelligence systems that output response content via video, audio, etc. The artificial intelligence system 40 includes a large-scale language model system (LLM system).

[0042] <System 1 operation> The following describes each process in System 1. Figure 12 is a flowchart showing the operation of document processing. Figure 13 shows an example screen illustrating the operation of the population creation processing page.

[0043] <Document Processing> Document processing is the process of applying processing to patent documents.

[0044] <Overview of Document Processing> Document processing is a series of processes that involves creating a population to be processed, identifying a first set from the population by sampling, generating a prompt to apply to the first set, obtaining a first output value from a large-scale language model by sending the prompt to the large-scale language model, presenting the first output value to the user, creating training data in response to the user's input, identifying a second set from the population, performing classification processing on the second set using a classification model trained on the training data, and repeating the above process until there are no more unprocessed documents left in the population.

[0045] <Details of document processing> The details of the document processing are described below.

[0046] <Steps to create a population> In step S101, the document processing unit 1042 of the server 10 executes a population reception step that receives population information relating to a population that includes multiple patent documents. Specifically, the user operates the input device 206 of the user terminal 20 to enter the URL of the page for executing the population creation process (population creation page D1) into a web browser or the like, and opens the population creation page. The control unit 204 of the user terminal 20 sends a request to the server 10 to open the population creation page. The control unit 104 of the server 10 generates the population creation page D1 based on the received request and sends it to the user terminal 20. The control unit 204 of the user terminal 20 displays the received population creation page D1 on the display 2081 of the user terminal 20. Figure 13 shows an example of the population creation processing page D1 in document processing. The population creation processing page D1 is displayed on the user terminal 20's display 2081. The population creation processing page D1 includes a task selection field D101, an extraction item selection field D111, a query input field D102, a document input field D103, a prompt display field D104, and a send button D105.

[0047] <Population Creation Step (Task Acceptance Step)> The control unit 204 of the user terminal 20 executes a task presentation step that presents the user with multiple task options for selection. Specifically, the document processing unit 1042 of the server 10 refers to the task master 1021 to obtain the task ID and task type items and sends them to the user terminal 20. The control unit 204 of the user terminal 20 displays the received multiple task types in the task selection field D101 in a selectable manner. For example, the control unit 204 of the user terminal 20 presents each of the multiple task types displayed in the task selection field D101 to the user, associating them with radio buttons for selecting the task type.

[0048] The document processing unit 1042 of the server 10 executes a task reception step that receives task information relating to a task to be performed on a patent document. The task reception step also executes a step that receives input operations from the user for predetermined task information from among the multiple task information presented in the task presentation step. Specifically, the user selects a predetermined task type to be performed on the patent document from among multiple task types displayed in the task selection field D101 by operating the input device 206 of the user terminal 20.

[0049] The task acceptance step involves receiving task information indicating at least one of the following tasks: a classification assignment task for assigning a classification to a patent document; a screening task for determining the relationship between a patent document and a target document; a compliance determination task for determining the relationship between a patent document and a target product; and an invention evaluation task for evaluating the value of the invention related to the patent document. Specifically, users can select a predetermined task type from among task types such as classification tasks, screening tasks, applicability determination tasks, and invention evaluation tasks. It is also possible to configure the system to allow selection of multiple task types.

[0050] The control unit 204 of the user terminal 20 sends a task ID corresponding to the task information selected in the task selection field D101 to the server 10. The document processing unit 1042 of the server 10 receives and accepts the task ID.

[0051] In this disclosure, a configuration in which multiple task information items are presented for selection and selected by the user is disclosed as an example, but the disclosure is not limited to this. For example, there may be only one type of task information (one of the following: classification assignment task, screening task, applicability determination task, or invention evaluation task), and the task information may be predetermined as a set of task information. In this case as well, the document processing unit 1042 of the server 10 can be interpreted as having received input of predetermined task information. Furthermore, task information does not necessarily have to be selected; it may be configured to be identified based on the content of the query received in the query reception step described later. Alternatively, instead of the task selection field D101, a task input field may be provided where the user can input task information, task type, etc. The user may input information indicating task information such as classification assignment task, screening task, applicability determination task, invention evaluation task, etc., into the task input field, and the control unit 204 of the user terminal 20 or the document processing unit 1042 of the server 10 may accept the task type.

[0052] <Population creation step (Query reception step)> The document processing unit 1042 of the server 10 executes a query reception step that receives query information to be applied to the patent document. Specifically, the user enters a string relating to the query to be applied to the patent document into the query input field D102 by operating the input device 206 of the user terminal 20. For example, the user enters the following query into the query input field D102 by operating the input device 206 of the user terminal 20, depending on the task type selected in step S501.

[0053] [Query for classification assignment tasks] Classification A is a classification related to information processing equipment. Classification B is a classification related to agricultural machinery. Classification C is a classification related to food products.

[0054] [Query for screening tasks] Query Example 1) A device comprising... means,... means, and... Query Example 2) A method comprising... process, ... process, and ... Query Example 3) A program that executes the following steps: ...step, ...step, and ...

[0055] [Query for the applicability determination task] Query Example 1) A device comprising... means,... means, and... Query Example 2) A method comprising... process, ... process, and ... Query Example 3) A program that executes the following steps: ...step, ...step, and ...

[0056] [Query for the invention evaluation task] Innovation: What is A-rank...? What is B-rank...? What is C-rank...? Economic efficiency: What is A rank...? What is B rank...? What is C rank...?

[0057] The control unit 204 of the user terminal 20 sends the string entered in the query input field D102 to the server 10. The document processing unit 1042 of the server 10 receives the string and accepts it as query data.

[0058] <Population Creation Step (Document Acquisition Step)> The document processing unit 1042 of server 10 executes a document acquisition step to acquire patent documents relating to patents. Specifically, the user enters the document ID of the patent document into the document input field D103 by operating the input device 206 of the user terminal 20. Note that the patent document does not need to be specified by its document ID in the document table 1013; the user may also enter the patent number (such as a published patent gazette), the abstract for each patent document, the claims, and the contents of the patent document (such as the patent specification) into the document input field D103. The user may also be able to directly select information about patent documents stored in Word, Excel, CSV files, etc., stored in the storage unit 201 of the user terminal 20 as the upload target. Any means by which the user can send the specified patent document to the server 10 is acceptable.

[0059] The document acquisition step involves performing the step of acquiring patent documents relating to multiple patents. The user may also transmit multiple patent documents to the server 10 by operating the input device 206 of the user terminal 20. The user may also input the document IDs of multiple patent documents into the document input field D103 by operating the input device 206 of the user terminal 20. Alternatively, the user may transmit an Excel, CSV file, or the like containing information on multiple patent documents to the server 10.

[0060] The control unit 204 of the user terminal 20 transmits one or more patent documents entered or selected by the user to the server 10. The document processing unit 1042 of the server 10 receives and accepts one or more patent documents.

[0061] The document processing unit 1042 of server 10 stores the task ID, query data, and one or more patent documents received from the user terminal 20 in the task ID, query data, and population data fields of a new record in the population table 1014.

[0062] <Population Creation Steps (2nd time onwards)> In step S101, the population reception step includes receiving first population information relating to a first population, and receiving second population information relating to a second population which includes one or more patent documents, excluding one or more patent documents stored in association with the second output result in the second storage step from the first population information. Specifically, when step S101 is executed again after the document processing has been completed up to step S109 (when it is executed for the second time or later), the patent documents identified by the second document ID associated with the second output value in the second support table 1016 are excluded from the patent documents included in the population data and accepted as a new population. This allows the first and second processing to be performed again on patent documents whose second processing result quality is unsatisfactory based on the learning model. As patent documents with satisfactory second processing results are gradually removed from the initial population, the first and second processing can be gradually applied to multiple patent documents included in the initial population. By applying the second processing to the population as much as possible, the application of the first processing can be suppressed.

[0063] In step S101, the population reception step includes receiving first population information relating to the first population, and receiving second population information relating to the second population, which includes one or more patent documents from the first population information, excluding one or more patent documents stored in association with information based on the first output result in the first storage step, and one or more patent documents stored in association with the second output result in the second storage step. Specifically, if step S101 is executed again after the document processing has been performed up to step S109 (when executed for the second time or later), the patent documents included in the population data may be excluded from the patent documents identified by the teacher document ID associated with the teacher label in the teacher table 1017, and the patent documents identified by the second document ID associated with the second output value in the second support table 1016, and accepted as a new population. Similarly, patent documents identified by the first document ID associated with the first output value in the first support table 1015 may be excluded from the patent documents included in the population data and accepted as a new population. This allows the first and second processing to be performed again on patent documents excluding those for which training data has been provided based on the first output result or the first output result, and those for which the quality of the second processing result is satisfactory. As patent documents for which training data has been provided or for which the quality of the second processing result is satisfactory are gradually removed from the initial population, the first and second processing can be gradually applied to multiple patent documents included in the initial population. By applying the second processing to the population as much as possible, the application of the first processing can be suppressed.

[0064] <Sampling Step (First Embodiment)> In step S102, the document processing unit 1042 of the server 10 executes a first patent extraction step, which extracts a first patent document from among multiple patent documents included in the population received in the population reception step. Specifically, in step S101, the document processing unit 1042 of server 10 randomly selects (performs sampling) some patent documents (first patent documents) from among multiple patent documents included in the population data stored in the population table 1014. For example, in this disclosure, the first patent document is for creating training data for processing performed on multiple patent documents included in the population data, so patent documents are extracted from the multiple patent documents included in the population data at an extraction rate of 10-20%. For example, if the population data contains 1000 patent documents, 100 to 200 patent documents will be extracted.

[0065] <Sampling step (second embodiment)> In step S102, the first patent extraction step includes dividing the multiple patent documents included in the population received in the population reception step into multiple groups, and extracting one or more patent documents from each of the multiple groups as the first patent document. Specifically, the document processing unit 1042 of server 10 performs clustering on multiple patent documents included in the population data stored in the population table 1014 in step S101. Specifically, the document processing unit 1042 of server 10 performs clustering on multiple patent documents based on the similarity of the document content of the multiple patent documents (text data such as claims, abstract, detailed description of the invention, examples, embodiments, and other specifications, and image data such as drawings). For example, any method can be used for clustering, such as k-means, hierarchical clustering, DBSCAN, and spectral clustering. This makes it possible to classify multiple patent documents included in the population data into groups (clusters) such as Group A, Group B, Group C, etc. The document processing unit 1042 of server 10 extracts a predetermined number of patent documents from each group. Specifically, the document processing unit 1042 of server 10 takes 10-20% of the total number of patent documents included in the population data as the total number, and extracts a number of patent documents from each group obtained by dividing this total number by the number of groups. For example, if the population data contains 1000 patent documents, and there are four groups A, B, C, and D, then if the total number of patent documents to be extracted is 100, then 25 patent documents should be extracted from each group. Similarly, if the total number of patent documents to be extracted is 200, then 50 patent documents should be extracted from each group. It is preferable to extract the patent documents from the groups randomly. This allows the first processing to be applied evenly across the entire representation space of the patent documents in the population, even if there is a bias in the content of the patent documents included in the population, by sampling a predetermined number of patent documents for each group. For example, the user can prepare training data that is well-balanced across the entire representation space of the patent documents included in the population. This enables the second processing to be performed with high quality, while also saving processing time and costs. The number of first patent documents extracted for each group may be a predetermined number, or it may be configured to extract a predetermined number of first patent documents corresponding to the number of patent documents included in the group or the proportion of each group.

[0066] <Sampling Step (2nd time onwards)> In step S102, the first patent extraction step includes extracting a number of first patent documents from first population information corresponding to a first proportion, and extracting a number of first patent documents from second population information corresponding to a second proportion. Specifically, when step S101 is executed again after the document processing has been performed up to step S109 (when it is executed for the second time or later), in the first step S102, first patent documents in a proportion of P% (first proportion) are extracted from the population, and in the second step S102, first patent documents in a proportion of Q% (second proportion) different from P% are extracted from the population.

[0067] For example, the second proportion may be larger than the first proportion. Specifically, P% may be 10% and Q% may be 15%. In this case, if there are many patent documents whose second processing result is of unsatisfactory quality, the convergence of the processing may be poor. If the second proportion is made larger than the first proportion, the proportion of patent documents processed by the large-scale language model in the population increases, which allows for an increase in the training data used for the second processing and a reduction in the proportion of patent documents whose second processing result is of unsatisfactory quality. This can reduce the number of processing steps in the processing described in this disclosure, and in some cases, shorten the total processing time.

[0068] For example, the first proportion may be larger than the second proportion. Specifically, P% may be 10% and Q% may be 5%. In this case, if the second proportion is made smaller than the first proportion, the proportion of patent documents processed by the large-scale language model within the population will decrease, thus saving on processing costs associated with the first processing.

[0069] The document processing unit 1042 of server 10 may search for the application condition items in the parameter master 1022 and obtain the extraction percentage (P%, Q%). For example, the number of document processing executions (1st time, 2nd time, etc.) can be used as an application condition, and the system can be configured to obtain the extraction percentage (P%, Q%) according to the number of executions.

[0070] The document processing unit 1042 of server 10 stores the document IDs of one or more extracted first patent documents in the document ID field of a new record in the first support table 1015. As a result, the processing target of the prompt generation step is stored as first support information in the record of the first support table 1015.

[0071] <Prompt generation step> In step S103, the step of creating prompts, which are input data for causing a large-scale language model based on the first patent document to perform processing, is performed. In this disclosure, the processing when one patent document is input as an example is described below. The document processing unit 1042 of the server 10 executes a prompt generation step for each of the multiple patent documents included in the first patent document.

[0072] <Prompt generation step (first embodiment)> In step S103, the document processing unit 1042 of the server 10 executes a prompt generation step to generate a prompt for the large-scale language model that includes the first patent document extracted in the first patent extraction step. The prompt generation step may also include a step to generate a prompt for the large-scale language model to output supplementary information that includes support information to assist in reading the first document. Specifically, the document processing unit 1042 of server 10 searches the task ID field in the task master 1021 based on the task ID of the target record in the population table 1014 and obtains the extraction items and task data items. The document processing unit 1042 of server 10 searches the document ID field in the document table 1013 based on the first document ID of the target record in the first support table 1015 and obtains the document content (the content of the patent document, such as claims, abstract, detailed description of the invention, examples, embodiments, etc.). Based on the obtained extraction items, the document processing unit 1042 of server 10 extracts predetermined extraction parts (extracted content, partial text) from the document content. Note that extraction of extraction parts is optional. The document processing unit 1042 of server 10 generates a prompt based on the acquired task data, the query data of the target record in the population table 1014, and the extracted location. The document processing unit 1042 of server 10 stores the generated prompt in the prompt field of the record of the first support information identified by the document ID of the patent document for which the prompt is to be generated in the first support table 1015. In other words, the document processing unit 1042 of server 10 generates a prompt for each of the first support information stored in the first support table 1015 and stores them in association.

[0073] <Prompt generation step (second embodiment)> In step S103, the prompt generation step includes generating a prompt that includes a first patent document and a classification definition document specifying classification rules, and causes a large-scale language model to output first classification information relating to the classification of the first patent document.

[0074] Specifically, the document processing unit 1042 of server 10 generates prompts by applying the query data of the target record in the population table 1014 and the extracted location to the acquired task data. The document processing unit 1042 of server 10 can generate the following prompts by replacing "<query>" with the query data and "<extracted document>" with the extracted location in the string of the task data. In this disclosure, classification tasks, screening tasks, applicability determination tasks, and invention evaluation tasks are disclosed as examples, but the applicable processes and tasks are not limited to these. It can be applied to any task that a large-scale language model can handle.

[0075] [Prompt for classification assignment tasks] The following technical classifications are defined. Please classify the invention described in the "Target Patent" below into one of the following classifications: A, B, or C. Please calculate the degree of relevance between the target patent and the classification on a scale of 0 to 100. Please output the reason for your judgment. #Technical classification: Classification A is a classification related to information processing equipment. Classification B is a classification related to agricultural machinery. Classification C is a classification related to food products. #Target Patent: [Summary] ... [Claims] ...

[0076] [Prompts for screening tasks] Please examine whether the invention described in the "Target Invention" below is described in the invention described in the "Target Patent" below. Calculate the degree of relevance between the Target Invention and the Target Patent on a scale of 0 to 100. Output the reason for your determination. #Target invention: ...means, ...means, ...device #Target Patent: [Summary] ... [Claims] ... [Specification] ...

[0077] [Prompt for the applicability determination task] Please consider whether the products, services, etc. listed under "Target Products" below fall within the scope of the rights of the inventions described under "Target Patents" below. Calculate the degree of relevance between the Target Products and the Target Patents on a scale of 0 to 100. Output the reason for your determination. #Target Products: ...means, ...means, ...device #Target Patent: [Claim 1] ...

[0078] [Prompt for the invention evaluation task] Please evaluate the inventions described in the "Target Patents" below according to the evaluation criteria for each evaluation item below. #Evaluation items and criteria: Innovation: What is A-rank...? What is B-rank...? What is C-rank...? Economic efficiency: What is A rank...? What is B rank...? What is C rank...? #Target Patent: [Claim 1] ...

[0079] <Prompt generation step (classification definition creation process)> In the prompt generation step (second embodiment), when performing a classification assignment task or an invention evaluation task, it is necessary to define the classification definition and evaluation items / criteria in advance. The document processing unit 1042 of server 10 may execute a third patent extraction step to extract a third patent document from among multiple patent documents included in the population received in the population reception step. These classification definitions and evaluation items / evaluation criteria can also be created using the artificial intelligence system 40. The following primarily describes methods for creating classification definitions in classification assignment tasks, but is not limited to these methods. Similarly, evaluation items and criteria in invention evaluation tasks may be created using the artificial intelligence system 40.

[0080] The document processing unit 1042 of the server 10 may perform a classification perspective acquisition step to acquire classification perspective information relating to the classification perspective of the third patent document, which is output in response to inputting a prompt to the large-scale language model that includes the third patent document extracted in the third patent extraction step and a classification perspective document that defines the classification perspective. Specifically, in step S101, the document processing unit 1042 of server 10 randomly extracts (samples) some patent documents (third patent documents) from among multiple patent documents included in the population data stored in the population table 1014. Note that the extraction of patent documents may be performed using the same process as the extraction of the first patent document in step S102. The document processing unit 1042 of server 10 creates the following prompts based on the document contents of the multiple patent documents included in the extracted third patent document and the classification perspective document that defines the classification perspective. As a result, the document processing unit 1042 of server 10 creates a prompt for each of the multiple patent documents that were extracted.

[0081] [Prompt for classification creation task] Please classify the inventions described in the "Target Patents" below according to the following classification criteria. #Classification criteria: Technical field, application, problem, objective, configuration, solution, effect [Summary] ... [Claims] ...

[0082] The document processing unit 1042 of server 10 sends a request containing the generated prompt (string) as input data to the artificial intelligence system 40 and sends it to the API endpoint provided by the artificial intelligence system 40. The artificial intelligence system 40 sends a response containing the answer to the prompt back to server 10. The document processing unit 1042 of server 10 receives a response to the sent prompt. The response contains a string of characters related to the answer to the prompt. Specifically, in the classification creation task, the following type of answer is obtained. In this disclosure, the following response content is described as an example, but it is preferable that the response content be received as structured data such as JSON with a field (key) related to the "classification name". For example, in a large-scale language model (LLM) called ChatGPT, the response result can be made into structured (typed) data by using a function called Function Calling. Furthermore, the responses obtained will be the responses for each of the multiple patent documents (responses for each of the multiple patent documents) to the prompts created for each of the multiple patent documents.

[0083] [Responses to the classification creation task] #Classification perspective 1: Technical field #Category: Semiconductor Technology #Reason: In the summary, ... #Classification perspective 2: Purpose #Classification name: Refinement #Reason: In the summary, ...

[0084] The document processing unit 1042 of the server 10 may execute a classification definition acquisition step to acquire a classification definition document obtained by broadly grouping the classification perspective information relating to the third patent document, which is output in response to inputting a prompt containing the classification perspective information acquired in the classification perspective acquisition step to the large-scale language model. Specifically, the document processing unit 1042 of the server 10 creates a prompt based on multiple classification names obtained as response results from the artificial intelligence system 40 for one or more patent documents, as follows: Classification name A, classification name B, and classification name C are entered with classification names (classification labels) created in a classification creation task, such as "semiconductor technology," "miniaturization," and "high integration."

[0085] [Prompt for classification definition task] Please create a classification definition by broadening the classification labels listed under "Classification Name" below according to your classification criteria. #Category name: [Category name A] [Category name B] [Category name C]

[0086] The document processing unit 1042 of server 10 sends a request containing the generated prompt (string) as input data to the artificial intelligence system 40 and sends it to the API endpoint provided by the artificial intelligence system 40. The artificial intelligence system 40 sends a response containing the answer to the prompt back to server 10. The document processing unit 1042 of server 10 receives a response to the sent prompt. The response contains a string of characters related to the answer to the prompt. Specifically, in the classification definition task, the following answer (classification definition) is obtained. In this way, classification definitions can be automatically created using classification names generated by applying the artificial intelligence system 40 to multiple patent documents included in the third patent document. The classification definitions thus obtained can be used as queries for the classification assignment task described above. This makes it possible to automatically classify multiple patent documents included in a population containing multiple patent documents without manually preparing classification definitions.

[0087] [Responses to the classification definition task] #Category 1: High-performance semiconductor technology #Description of classification: Inventions related to technologies for improving the performance of semiconductors. #Subcategory: Wiring techniques, High integration techniques, ... #Category 2: Semiconductor Cost Reduction Technologies #Classification explanation: Inventions related to technologies that reduce the design and manufacturing costs of semiconductors #Subcategory: Material innovations, processing innovations, ...

[0088] <First output value acquisition step (first embodiment)> In step S104, the document processing unit 1042 of the server 10 executes a first processing step to obtain a first output result that is output in response to inputting a prompt to the large-scale language model that includes the first patent document extracted in the first patent extraction step. In step S104, the first processing step performs the step of obtaining first classification information relating to the classification of the first patent document, which is output in response to inputting a prompt to a large-scale language model that includes a first patent document and a classification definition document that specifies the classification rules. In step S104, the first processing step is to perform the step of obtaining first classification information based on the classification definition document obtained in the classification definition acquisition step.

[0089] Specifically, the document processing unit 1042 of the server 10 sends a request containing the prompt (string) generated in step S103 to the API endpoint provided by the artificial intelligence system 40 as input data for the artificial intelligence system 40. The artificial intelligence system 40 sends a response containing the answer to the prompt back to the server 10. The document processing unit 1042 of server 10 receives a response to a sent prompt. The response includes a string of characters relating to the answer to the prompt. Specifically, depending on the task—classification assignment task, screening task, applicability determination task, and invention evaluation task—the artificial intelligence system 40 outputs strings of characters relating to the answer to the classification assignment result, screening result, applicability determination result, and invention evaluation result, respectively. For example, the response content includes the following information. In this disclosure, the following response content is described as an example, but it is preferable that the response content be received as structured data such as JSON with fields (keys) such as "classification name". For example, in a large-scale language model (LLM) called ChatGPT, the response result can be made into structured (typed) data by using a function called Function Calling. Furthermore, the responses obtained will be the responses for each of the multiple patent documents (responses for each of the multiple patent documents) to the prompts created for each of the multiple patent documents. The document processing unit 1042 of server 10 stores the response result in the first output value item of the first support information record, which is identified by the document ID of the patent document to be used to generate the prompt in the first support table 1015.

[0090] [Response content for classification assignment tasks] #Classification: A #Relevance: 80 #reason: The patent in question includes configuration .... and configuration . On the other hand, the patent in question does not include configuration .... and configuration . It conforms to the classification definition of Classification A, but does not conform to the classification definitions of Classification B and Classification C. The degree of relevance to Classification A is approximately 80.

[0091] [Response content for screening tasks] #Relevance: 40 #reason: The target invention is consistent with the target patent in terms of ..., and differs in terms of .... There are significant differences, and the degree of relevance to the target patent is approximately 40.

[0092] [Response content for the task of determining whether or not the item applies] #Relevance:20 #reason: The product in question is consistent with the patent in the respect of..., and differs in the respect of.... The product in question is not included in the scope of the patent in question. Its relevance to the patent in question is approximately 20.

[0093] [Response content for the invention evaluation task] #Innovation: B rank #Reason: The patent in question has a configuration including ..., ... #Economy: A rank #Reason: The patent in question has a configuration including ..., ...

[0094] In this case, the index value (relevance) is an index value indicating the accuracy of the assigned classification when the task information is a classification assignment task, an index value indicating the degree of relevance to the target invention when the task information is a screening task, and an index value indicating the degree of relevance to the target product when the task information is a compliance determination task.

[0095] <First output value presentation step> In step S105, the document processing unit 1042 of the server 10 executes a first presentation step in which it presents the first output result obtained in the first processing step to the user. Specifically, the document processing unit 1042 of the server 10 sends the received response content to the user terminal 20. The control unit 204 of the user terminal 20 displays the received response content on the user terminal 20's display 2081. The user may also download the response information in file formats such as Word, Excel, or CSV by operating the input device 206 of the user terminal 20. The document processing unit 1042 of server 10 may also present the user with the extracted location in the patent document, in addition to the index value and its basis. This allows the user to confirm which location in the patent document the index value was calculated based on and on what basis.

[0096] The document processing unit 1042 of server 10 sends the response content corresponding to the prompts generated for each of the multiple patent documents included in the first patent document to the user terminal 20. The control unit 204 of user terminal 20 may display and present the received response content on the display 2081 of user terminal 20. Specifically, information identifying the documents, such as the document ID, publication number, and document name of the multiple patent documents, the prompts generated for each document, and the response content received from the artificial intelligence system 40 in response to those prompts may be associated and presented accordingly. In addition to the index value and justification, the extracted location of the patent document may also be presented to the user. Furthermore, the user may operate the input device 206 of the user terminal 20 to download files such as Word, Excel, or CSV files containing information identifying multiple patent documents, such as document IDs, publication numbers, and document names, as well as prompts generated for each document and responses received from the artificial intelligence system 40 in response to those prompts.

[0097] <Steps for creating training data> In step S106, the document processing unit 1042 of the server 10 receives an input operation from the user regarding the first output result presented in the first presentation step, and executes a training data creation step to create training data relating to the first patent document in accordance with the input operation. Specifically, the user can edit the first output result displayed on the user terminal 20's display 2081 by operating the input device 206 of the user terminal 20. Specifically, the user can modify or delete the content of the first output result by operating the input device 206 of the user terminal 20. The user sends the edited content, such as modifications and deletions to the first output result, to the server 10 by operating the input device 206 of the user terminal 20. Since the content of the first output values ​​produced by large-scale language models often contains incorrect or inaccurate information, it is preferable for users to create training data based on the first output values ​​by correcting, editing, or deleting the content of the first output values.

[0098] In step S106, the document processing unit 1042 of the server 10 performs a first storage step in which it stores at least part or all of the first patent documents extracted in the first patent extraction step in association with information based on the first output result. Specifically, the document processing unit 1042 of server 10 associates the first document ID with the modified and edited first output result based on the edited content received from the user terminal 20, and stores it in the teacher document ID and teacher label fields of a new record in the teacher table 1017. If the content of the first output value is valid, the user sends the first document ID and first output result as is to server 10, and the document processing unit 1042 of server 10 stores the received first document ID and first output result in the teacher document ID and teacher label fields of the teacher table 1017. This allows users to create training data with minimal effort by referring to the first output result. The second processing is performed based on the training data created and edited by the user, compared to when the first output result is used directly as training data, so higher quality processing results can be expected.

[0099] In this disclosure, an example is provided in which training data is created in response to user input operations based on the first output result output from the artificial intelligence system 40 in steps S105 and S106, but the disclosure is not limited to this example. For example, the first output result (first output value) for the first patent document may be stored directly in the training label item of the training table 1017. In this case, steps S105 and S106 may be omitted.

[0100] <Step to identify the object to be processed> In step S107, the document processing unit 1042 of the server 10 executes a second patent extraction step, which extracts a second patent document from among multiple patent documents included in the population received in the population reception step. Specifically, in step S101, the document processing unit 1042 of the server 10 extracts all of the multiple patent documents included in the population data stored in the population table 1014 as second patent documents. Furthermore, the document processing unit 1042 of server 10 may identify the second patent document from among multiple patent documents included in the population data, excluding the patent document identified by the teacher document ID item in the teacher table 1017. Similarly, the document processing unit 1042 of the server 10 may identify the second patent document from among multiple patent documents included in the population data, excluding the patent document identified by the first document ID item in the first support table 1015. In addition, the document processing unit 1042 of server 10 may randomly select (sample) some patent documents (first patent documents) from multiple patent documents and designate them as second patent documents. The sampling process may be the same as the sampling step in step S102. A detailed explanation is omitted.

[0101] The document processing unit 1042 of server 10 stores the document IDs of one or more extracted second patent documents in the document ID field of a new record in the second support table 1016. As a result, the processing target of the second output value acquisition step is stored as first support information in the record of the second support table 1016.

[0102] <Learning Process> In step S108, the document processing unit 1042 of the server 10 executes the training process for the classification model 1051. The training process for the classification model 1051 is described below.

[0103] <Training process for classification model 1051> The training process for classification model 1051 involves training the training parameters of the deep neural network included in classification model 1051 using deep learning.

[0104] <Overview of the training process for classification model 1051> The learning process for classification model 1051 involves using the document content of a patent document (which may be extracted based on the extraction items of task master 1021 by referring to the task ID of the population information, or it may be any other extracted section such as the claims, abstract, detailed description of the invention, examples, embodiments, etc.) as input data (input vector), and training the learning parameters of the deep neural network included in classification model 1051 using deep learning so that the training labels (first output result) become the output data (training data).

[0105] <Details of the training process for classification model 1051> The control unit 104 of server 10 obtains teacher information, including the teacher document ID and teacher label, from the teacher table 1017. The document processing unit 1042 of server 10 searches the document ID item in the document table 1013 based on the teacher document ID and obtains the document content. It is preferable for the document processing unit 1042 of server 10 to extract the extraction points from the document content based on the extraction items of the task master 1021. In other words, it is preferable to extract predetermined extraction points from the patent document so that they are the same as the input data when the first output value was obtained in step S104. The control unit 104 of the server 10 creates training data using document content, extracted locations, etc. as input data (input vectors) and teacher labels as output data (teaching data). The control unit 104 of server 10 creates datasets such as training data, test data, and validation data for training the deep neural network of the classification model 1051, based on the training data. The control unit 104 of server 10 trains the learning parameters of the deep neural network included in the classification model 1051 using deep learning, based on the created dataset.

[0106] <Second output value acquisition step> In step S108, the second output value is obtained by inputting the input data based on the second patent document into the classification model 1051. In this disclosure, the processing when one patent document is input as an example is described below. The document processing unit 1042 of the server 10 executes the second output value acquisition step for each of the multiple patent documents included in the second patent document.

[0107] <Second output value acquisition step (first embodiment)> In step S108, the second processing step is to obtain second classification information relating to the classification of the second patent document, which is output in response to inputting the second patent document into the learning model, which is a classification model. The second processing step includes the step of obtaining second quality information indicating the quality of classification for one or more patent documents contained in the second patent document, which is output in response to inputting the second patent document into the learning model. The document processing unit 1042 of the server 10 executes a second classification storage step, which stores at least a portion of one or more patent documents included in the second patent document in association with the second classification information, if the second quality information acquired in the second processing step satisfies predetermined conditions. Specifically, the document processing unit 1042 of server 10 searches the task ID field in the task master 1021 based on the task ID of the target record in the population table 1014 and obtains the extraction item field. The document processing unit 1042 of server 10 searches the document ID field in the document table 1013 based on the second document ID of the target record in the second support table 1016 and obtains the document content (content of the patent document such as claims, abstract, detailed description of the invention, examples, embodiments, etc.). Based on the obtained extraction item, the document processing unit 1042 of server 10 extracts predetermined extraction parts (extracted content, partial text) from the document content. Note that extraction of extraction parts is optional. The document processing unit 1042 of server 10 inputs the document content or extracted sections as input data for the classification model 1051. Alternatively, the document processing unit 1042 of server 10 may also input keywords, etc., extracted by applying processing such as morphological analysis to the document content or extracted sections as input data for the classification model 1051. The classification model 1051 outputs classification labels (strings) and indicators (quality indicators) that show the quality of the classification process when classifying input data into classification labels, such as accuracy, in response to the input data. Specifically, accuracy is an indicator that shows how accurately the classification model 1051 predicted the classification labels. Accuracy is calculated by dividing the number of samples correctly predicted by the classification model 1051 by the total number of samples. Accuracy takes a value between 0 and 1 and can be expressed as a percentage (0% to 100%). Indicators that show the quality of the classification process include the confusion matrix, precision, recall, and F1 score. For example, for a specific patent document A, the classification model 1051 outputs a second output value such as "Classification label: Semiconductor technology (accuracy 0.7)".

[0108] In step S108, the document processing unit 1042 of the server 10 stores at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step satisfies predetermined conditions, and does not store at least a portion of one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step does not satisfy predetermined conditions. The second memory step is executed. Specifically, the document processing unit 1042 of server 10 acquires a classification label and quality index for each of the multiple patent documents included in the second patent document. The document processing unit 1042 of server 10 compares the quality index with a predetermined precision threshold, and if the condition that it is equal to or greater than the precision threshold is met, it stores the acquired classification label and quality index in the second output value and precision field of the patent document record identified based on the document ID to be processed in the second support table 1016. In other words, the document processing unit 1042 of server 10 does not store all of the second output results output from the classification model 1051 in the second support table 1016, but only stores in the second support table 1016 the second output results whose quality indicators satisfy predetermined conditions. That is, if the quality indicators of the second output results are insufficient, they are not stored in the second support table 1016. In this case, as the classification model 1051 learns sufficiently in subsequent document processing, the quality indicators of the second output results will become sufficient and will be stored in the second support table 1016. The document processing unit 1042 of the server 10 may delete records of second support information stored in the second support table 1016 whose quality indicators do not meet predetermined conditions (records in which only the second document ID is stored, and the second output value and precision value are blank).

[0109] <Second output value acquisition step (second embodiment)> In step S108, the document processing unit 1042 of the server 10 executes a second processing step in which it obtains a second output result, which is output in response to inputting the second patent document extracted in the second patent extraction step into a learning model that has been trained using information based on the first output result obtained in the first processing step as training data. The second processing step involves obtaining a second output result, which is output in response to inputting a second patent document into a learning model that has been trained based on the training data created in the training data creation step. The second processing step includes obtaining second quality information indicating the quality of processing for one or more patent documents included in the second patent document, which is output in response to inputting the second patent document into the learning model. Specifically, this disclosure uses the example of a case where a second output value is generated by classification model 1051, but is not limited to this. For example, instead of classification model 1051, a screening model may be used that outputs the degree of relevance to the target patent and quality indicators (accuracy, etc.) related to that relevance, based on the input of the document content and extracted locations of the patent document. In this case, the screening model is trained using training data based on the answers (first output value) for the screening task. The screening model outputs the degree of relevance to the target patent and quality indicators (accuracy, etc.) related to that relevance, based on the input of the document content and extracted locations of the second patent document. For example, for a specific patent document A, the screening model outputs a second output value such as "Relevance: 80% (accuracy 0.7)". For example, instead of classification model 1051, a classification model may be used that outputs the degree of relevance to the target product and quality indicators (accuracy, etc.) related to that relevance, in response to input of the document content and extracted locations of the patent document. In this case, the classification model is trained using training data based on the response content (first output value) for the classification task. The classification model outputs the degree of relevance to the target product and quality indicators (accuracy, etc.) related to that relevance, in response to input of the document content and extracted locations of the second patent document. For example, for a specific patent document A, the classification model outputs a second output value such as "Relevance: 80% (accuracy 0.7)". For example, instead of classification model 1051, an invention evaluation model may be used that outputs evaluation values ​​(evaluation ranks) for each evaluation item of the target invention, and quality indicators (accuracy, etc.) related to the evaluation values ​​(evaluation ranks) for each evaluation item, in response to input of the document content and extracted locations of the patent document. In this case, the invention evaluation model is trained using training data based on the answers (first output values) for the invention evaluation task. The invention evaluation model outputs evaluation values ​​(evaluation ranks) for each evaluation item and quality indicators (accuracy, etc.) related to the degree of relevance, in response to input of the document content and extracted locations of the second patent document. For example, for a specific patent document A, the invention evaluation judgment model outputs second output values ​​such as "Innovativeness: A rank (accuracy 0.7), Economic efficiency: B rank (accuracy 0.6)."

[0110] <Second output value acquisition step (2nd time onwards)> In step S108, the second storage step includes the steps of: if the population information received in the population reception step is the first population information, and the second quality information satisfies the first condition, then storing at least a portion of one or more patent documents included in the second patent document in association with the second output result; and if the second quality information does not satisfy the first condition, then not storing at least a portion of one or more patent documents included in the second patent document in association with the second output result. Specifically, if step S108 is executed again after the document processing has been performed up to step S109 (when it is executed for the second time or later), the precision threshold may be set to a different value for the first precision threshold S and the second precision threshold T. The document processing unit 1042 of the server 10 may store in the second support table 1016 the second output results that satisfy the second condition based on the precision thresholds (S, T) from among the multiple patent documents included in the second patent document, and may not store in the second support table 1016 the second output results that do not satisfy the second condition. The document processing unit 1042 of the server 10 may delete records of second support information stored in the second support table 1016 whose second output result does not satisfy predetermined conditions (records in which only the second document ID is stored, and the second output value and precision value are blank).

[0111] The document processing unit 1042 of server 10 may search for the application condition items in the parameter master 1022 and obtain the precision threshold (S, T). For example, the number of document processing executions can be used as an application condition, and the precision threshold (S, T) can be obtained according to the number of executions. For example, the number of document processing executions (1st time, 2nd time, etc.) can be used as an application condition, and the precision threshold (S, T) can be obtained according to the number of executions.

[0112] For example, the first condition may include the second condition. For example, the precision threshold for the first condition may be 0.5 or higher, and the precision threshold for the second condition may be 0.7 or higher. This allows the quality requirements for the second process result to gradually become stricter when the first and second processes are executed multiple times, thereby improving the final processing quality.

[0113] For example, the second condition may include the first condition. For example, the precision threshold for the first condition may be 0.7 or higher, and the precision threshold for the second condition may be 0.5 or higher. This allows the quality requirements for the second process result to gradually loosen when the first and second processes are executed multiple times. This can reduce the total processing time and save on processing costs.

[0114] <Unprocessed Document Determination Step> In step S109, the document processing unit 1042 of the server 10 performs an unprocessed document determination step to determine whether there are any patent documents among the patent documents included in the population received in step S101 that are not stored in association with the first output result, training data, or second output result. Specifically, in step S101, the document processing unit 1042 of server 10 searches for and retrieves the second document ID item in the second support table 1016 based on the document IDs of multiple patent documents included in the population data stored in the population table 1014. At this time, only records in the second support table 1016 in which a value is stored in the second output value item are searched. The document processing unit 1042 of server 10 compares the document IDs of multiple patent documents included in the population data with the acquired second document IDs. If all of the document IDs of the multiple patent documents included in the population data are included in the acquired second document ID, it determines that there are no unprocessed documents. The document processing unit 1042 of server 10 retrieves the teacher document ID item from the teacher table 1017, compares the document IDs of multiple patent documents included in the population data with the retrieved second document ID and teacher document ID, and may determine that there are no unprocessed documents if all of the document IDs of the multiple patent documents included in the population data are included in the retrieved second document ID and teacher document ID. In other words, patent documents to which a teacher label has been assigned may be excluded from the determination of whether they are unprocessed documents. The document processing unit 1042 of server 10 may obtain the first document ID item from the first support table 1015, compare the document IDs of multiple patent documents included in the population data with the obtained second document ID and first document ID, and determine that there are no unprocessed documents if all of the document IDs of multiple patent documents included in the population data are included in the obtained second document ID and first document ID. In other words, patent documents for which the first output value is stored in the first support table 1015 may be excluded from the determination of whether they are unprocessed documents.

[0115] In step S109, if the document processing unit 1042 of the server 10 determines that there are no unprocessed documents, it proceeds to step S110. On the other hand, if it determines that there are unprocessed documents, document processing (second time onwards) is executed again from step S101. Specifically, in step S101, the population creation step (second time onwards) is executed. In other words, the document processing unit 1042 of server 10 automatically iterates through each step from S101 to S109 as long as there are unprocessed documents in the document creation process.

[0116] <Ending Steps> In step S110, the document processing unit 1042 of the server 10 executes a processing termination step to terminate processing if there are no patent documents among the population information received in the population reception step, excluding one or more patent documents stored in the first storage step in association with information based on the first output result, and one or more patent documents stored in the second storage step in association with the second output result. As a result, the first and second processes are repeatedly executed until the information based on the first processing result (training data) and the second processing result are associated and stored for all patent documents included in the population received in the population reception step. The document processing unit 1042 of server 10 iteratively executes document processing steps S101 to S109, thereby storing the processing results for all patent documents included in the population data in the teacher table 1017, the second support table 1016, and the first support table 1015. The document processing methods described in this disclosure allow for the application of suitable processing to a population containing multiple patent documents with a small amount of training data, or even without any training data, without the need to prepare a large amount of training data. The first processing method using large-scale language models tends to be more expensive and time-consuming than the second processing method. This disclosure eliminates the need to apply the first processing method to all patent documents in the population, thus saving processing time and costs.

[0117] <Basic Computer Hardware Configuration> Figure 14 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary memory 903, and a communication interface 991. These are electrically connected to each other by a communication bus 921.

[0118] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0119] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0120] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0121] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0122] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0123] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 14) is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0124] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0125] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0126] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0127] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0128] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0129] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0130] <Note> The details described in each of the above embodiments are noted below.

[0131] (Note 1) A program to be executed by a computer having a processor and a memory unit, wherein the processor executes: a population reception step (S101) in which population information relating to a population including a plurality of patent documents is received; a first patent extraction step (S102) in which a first patent document is extracted from the plurality of patent documents included in the population received in the population reception step; a first processing step (S104) in which a first output result is output in response to inputting a prompt including the first patent document extracted in the first patent extraction step to a large-scale language model; a second patent extraction step (S107) in which a second patent document is extracted from the plurality of patent documents included in the population received in the population reception step; and a second processing step (S108) in which a second output result is output in response to inputting the second patent document extracted in the second patent extraction step to a learning model that has been trained using information based on the first output result obtained in the first processing step as training data. This makes it possible to apply suitable processing to a population containing multiple patent documents with a small amount of training data, or even without any training data at all, without having to prepare a large amount of training data. The first processing method, using a large-scale language model, tends to be more expensive and time-consuming than the second processing method. Since it is not necessary to apply the first processing method to all patent documents included in the population, processing time and costs can be saved.

[0132] (Note 2) The program described in Appendix 1, which performs a second processing step (S108) which includes a step of acquiring second quality information indicating the quality of processing for one or more patent documents included in the second patent document output in response to inputting the second patent document to the learning model, and a second storage step (S108) in which the processor stores at least a portion of the one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step satisfies predetermined conditions, and does not store at least a portion of the one or more patent documents included in the second patent document in association with the second output result if the second quality information acquired in the second processing step does not satisfy predetermined conditions. This allows processing to be performed on the second patent document using information based on the first output result as training data. Only the second processing results from the learning model that are of suitable quality can be associated with and stored in the second patent document.

[0133] (Note 3) The program as described in Appendix 2, wherein the population reception step (S101) includes a step of receiving first population information relating to a first population, and a step of receiving second population information relating to a second population which includes one or more patent documents, excluding one or more patent documents stored in association with the second output result in the second storage step from the first population information. This allows the first and second processing to be performed again on patent documents where the quality of the second processing result from the learning model is unsatisfactory. Since patent documents with favorable quality results from the second processing are gradually removed from the initial population, the first and second processing can be gradually applied to multiple patent documents included in the initial population. By applying the second treatment to the population as much as possible, the application of the first treatment can be minimized.

[0134] (Note 4) The program as described in Appendix 2, wherein the processor performs a first storage step (S106) in which it stores at least some or all of the first patent documents extracted in the first patent extraction step in association with information based on the first output result, and the population reception step (S101) includes a step of receiving first population information relating to the first population, and a step of receiving second population information relating to the second population, which includes one or more patent documents from the first population information, excluding one or more patent documents stored in association with information based on the first output result in the first storage step and one or more patent documents stored in association with the second output result in the second storage step. This allows the first and second processes to be performed again on patent documents, excluding those for which training data has been provided based on the first output result or the first output result, and those for which the quality of the second processing result is satisfactory. As patent documents for which training data has been provided or for which the quality of the second processing result is satisfactory are gradually removed from the initial population, the first and second processing can be gradually applied to multiple patent documents included in the initial population. By applying the second treatment to the population as much as possible, the application of the first treatment can be minimized.

[0135] (Note 5) The first patent extraction step (S102) is the program described in Appendix 3, which includes the steps of extracting a number of first patent documents from first population information corresponding to a first proportion, and extracting a number of first patent documents from second population information corresponding to a second proportion. This makes it possible to control the extraction rate of the first patent document when the first and second processes are performed multiple times. By controlling the convergence of the processing, the total processing time can be reduced, and processing costs can be saved by reducing the proportion of patent documents to which the first processing is applied.

[0136] (Note 6) The second proportion is the program described in Appendix 5, which is larger than the first proportion. For example, if there are many patent documents whose second processing results are of unsatisfactory quality, the convergence of the processing may be poor. If the second proportion is made larger than the first proportion, the proportion of patent documents processed by the large-scale language model in the population increases, which allows for an increase in the training data used for the second processing and a reduction in the proportion of patent documents whose second processing results are of unsatisfactory quality. This can reduce the number of processing steps in the process described in this disclosure, and in some cases, shorten the total processing time.

[0137] (Note 7) The second proportion is the program described in Appendix 5, which is smaller than the first proportion. Making the second proportion smaller than the first proportion reduces the proportion of patent documents processed by the large-scale language model within the population, thereby saving on processing costs associated with the first processing.

[0138] (Note 8) The program as described in Appendix 3, which includes the steps: if the population information received in the population reception step is the first population information, and the second quality information satisfies the first condition, then at least a portion of one or more patent documents included in the second patent document is stored in association with the second output result; and if the second quality information does not satisfy the first condition, then at least a portion of one or more patent documents included in the second patent document is not stored in association with the second output result; if the population information received in the population reception step is the second population information, and the second quality information satisfies the second condition, then at least a portion of one or more patent documents included in the second patent document is stored in association with the second output result; and if the second quality information does not satisfy the second condition, then at least a portion of one or more patent documents included in the second patent document is not stored in association with the second output result. This makes it possible to control the conditions for determining whether the quality of the second processing result is satisfactory when the first and second processing are performed multiple times. For example, by making the conditions more lenient, the total processing time can be reduced and processing costs can be saved. On the other hand, by making the conditions stricter, processing quality can be improved.

[0139] (Note 9) The first condition is the program described in Appendix 8, including the second condition. This allows the quality requirements for the second process result to gradually become stricter when the first and second processes are executed multiple times, thereby improving the final processing quality.

[0140] (Note 10) The second condition is the program described in Appendix 8, including the first condition. This allows the quality requirements for the second process result to gradually loosen when the first and second processes are executed multiple times. This can reduce the total processing time and save on processing costs.

[0141] (Note 11) The program described in Appendix 4, wherein the processor executes a processing termination step (S110) in which it terminates processing if, among the population information received in the population reception step, there are no patent documents other than one or more patent documents stored in the first storage step in association with information based on the first output result, and one or more patent documents stored in the second storage step in association with the second output result. As a result, the first and second processes are repeatedly executed until the information based on the first processing result and the second processing result are associated and stored for all patent documents included in the population received in the population reception step.

[0142] (Note 12) The program as described in Appendix 1, wherein the first processing step (S104) is to acquire first classification information relating to the classification of the first patent document, which is output in response to inputting a prompt containing a first patent document and a classification definition document specifying classification rules to a large-scale language model, and the second processing step (S108) is to acquire second classification information relating to the classification of the second patent document, which is output in response to inputting a second patent document to a learning model, which is a classification model. This makes it possible to classify multiple patent documents included in a population containing multiple patent documents using a small amount of training data, or even without any training data at all, without having to prepare a large amount of training data. The first processing method, using a large-scale language model, tends to be more expensive and time-consuming than the second processing method. Since it is not necessary to apply the first processing method to all patent documents included in the population, processing time and costs can be saved.

[0143] (Note 13) The program as described in Appendix 12, wherein the processor performs a third patent extraction step (S103) in which it extracts a third patent document from among multiple patent documents included in a population received in a population reception step; a classification perspective acquisition step (S103) in which it acquires classification perspective information relating to the classification perspective of the third patent document, which is output in response to prompt input to a large-scale language model that includes the third patent document extracted in the third patent extraction step and a classification perspective document defining the classification perspective; and a classification definition acquisition step (S103) in which it acquires a classification definition document obtained by broadly grouping the classification perspective information relating to the third patent document, which is output in response to prompt input to a large-scale language model that includes the classification perspective information acquired in the classification perspective acquisition step, and the first processing step (S104) is a step of acquiring first classification information based on the classification definition document acquired in the classification definition acquisition step. This makes it possible to classify multiple patent documents included in a population containing multiple patent documents without having to prepare classification definitions.

[0144] (Note 14) The program according to Appendix 12, wherein the second processing step (S108) includes a step of acquiring second quality information indicating the quality of classification for one or more patent documents contained in the second patent document, which is output in response to inputting the second patent document to the learning model, and the processor performs a second classification storage step (S108) in which, if the second quality information acquired in the second processing step satisfies predetermined conditions, the processor stores at least a portion of the one or more patent documents contained in the second patent document in association with the second classification information. This allows the second patent document to be classified using information based on the first output result as training data. Only those documents with satisfactory classification quality according to the classification model can be associated with and stored in the second patent document.

[0145] (Note 15) The program as described in Appendix 1, wherein the processor performs a first presentation step (S105) in which it presents to the user a first output result obtained in the first processing step, and a training data creation step (S106) in which it receives an input operation from the user regarding the first output result presented in the first presentation step and creates training data relating to the first patent document in accordance with the input operation, and the second processing step (S108) is a step of obtaining a second output result that is output in response to inputting the second patent document into a learning model that has been trained based on the training data created in the training data creation step. This allows users to create training data with less workload by referring to the first output result. The second processing is performed based on the training data created and edited by the user, compared to when the first output result is used directly as training data, so higher quality processing results can be expected.

[0146] (Note 16) The program described in Appendix 1 includes a first patent extraction step (S102) which comprises dividing a plurality of patent documents included in the population received in the population reception step into a plurality of groups, and extracting one or more patent documents from each of the plurality of groups as the first patent document. This allows the first processing to be applied evenly across the entire representation space of the patent documents in the population, even if there is a bias in the content of the patent documents included in the population, by sampling a predetermined number of patent documents for each group. For example, the user can prepare training data that is well-balanced across the entire representation space of the patent documents included in the population. This enables the second processing to be performed with high quality, while also saving processing time and costs. The number of first patent documents extracted for each group may be a predetermined number, or it may be configured to extract a predetermined number of first patent documents according to the number of patent documents included in the group or the proportion of each group.

[0147] (Note 17) A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in Appendix 1 to Appendix 16. This makes it possible to apply suitable processing to a population containing multiple patent documents with a small amount of training data, or even without any training data at all, without having to prepare a large amount of training data. The first processing method, using a large-scale language model, tends to be more expensive and time-consuming than the second processing method. Since it is not necessary to apply the first processing method to all patent documents included in the population, processing time and costs can be saved.

[0148] (Note 18) An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in the invention according to any of the appendices 1 to 16. This makes it possible to apply suitable processing to a population containing multiple patent documents with a small amount of training data, or even without any training data at all, without having to prepare a large amount of training data. The first processing method, using a large-scale language model, tends to be more expensive and time-consuming than the second processing method. Since it is not necessary to apply the first processing method to all patent documents included in the population, processing time and costs can be saved.

[0149] (Note 19) A system comprising means for performing all steps performed in any of the inventions described in Appendix 1 to Appendix 16. This makes it possible to apply suitable processing to a population containing multiple patent documents with a small amount of training data, or even without any training data at all, without having to prepare a large amount of training data. The first processing method, using a large-scale language model, tends to be more expensive and time-consuming than the second processing method. Since it is not necessary to apply the first processing method to all patent documents included in the population, processing time and costs can be saved. [Explanation of symbols]

[0150] 1 System, 10 Servers, 101 Memory Unit, 104 Control Unit, 106 Input Device, 108 Output Device, 20 User Terminals, 201 Memory Unit, 204 Control Unit, 206 Input Device, 208 Output Device, 40 Artificial Intelligence System, 401 Memory Unit, 404 Control Unit, 406 Input Device, 408 Output Device

Claims

1. A program to be executed by a computer comprising a processor and a memory unit, wherein the processor A population reception step that receives population information relating to a population containing multiple patent documents, A first patent extraction step in which a first patent document is extracted from among the plurality of patent documents included in the population received in the population reception step, A first processing step to obtain a first output result that is output in response to inputting a prompt containing the first patent document extracted in the first patent extraction step to a large-scale language model, A second patent extraction step in which a second patent document is extracted from among the multiple patent documents included in the population received in the population reception step, A program that executes a second processing step which involves inputting the second patent document extracted in the second patent extraction step into a learning model that has been trained using the information based on the first output result obtained in the first processing step as training data, and in order to obtain a second output result, the second output result that is output in response to inputting the second patent document extracted in the second patent extraction step.

2. The second processing step includes a step of acquiring second quality information indicating the quality of processing for one or more patent documents included in the second patent document, which is output in response to inputting the second patent document into the learning model, The aforementioned processor If the second quality information obtained in the second processing step satisfies predetermined conditions, at least a portion of one or more patent documents included in the second patent document is stored in association with the second output result. If the second quality information obtained in the second processing step does not satisfy predetermined conditions, at least a portion of one or more patent documents included in the second patent document is not stored in association with the second output result. The second memory step, Execute The program according to claim 1.

3. The aforementioned population registration step is: A step to receive information about the first population regarding the first population, A step of receiving second population information relating to a second population that includes one or more patent documents, excluding one or more patent documents stored in association with the second output result in the second storage step from the first population information, including, The program according to claim 2.

4. The aforementioned processor A first storage step which stores at least part or all of the first patent documents extracted in the first patent extraction step in association with information based on the first output result, Execute, The aforementioned population registration step is: A step to receive information about the first population regarding the first population, From the above first population information, One or more patent documents stored in association with the information based on the first output result in the first storage step, In the second storage step, one or more patent documents stored in association with the second output result, A step of receiving second population information relating to a second population that includes one or more patent documents excluding the above, including, The program according to claim 2.

5. The first patent extraction step is, The steps include extracting a number of the first patent documents corresponding to a first proportion from the first population information, The steps include extracting the number of the first patent documents corresponding to the second proportion from the second population information, including, The program according to claim 3.

6. The second proportion is greater than the first proportion. The program according to claim 5.

7. The second proportion is smaller than the first proportion. The program according to claim 5.

8. The second storage step is, If the population information received in the population reception step is the first population information, If the second quality information satisfies the first condition, at least a portion of one or more patent documents included in the second patent document is stored in association with the second output result. If the second quality information does not satisfy the first condition, at least a portion of one or more patent documents included in the second patent document is not stored in association with the second output result. If the population information received in the population reception step is the second population information, If the second quality information satisfies the second condition, at least a portion of one or more patent documents included in the second patent document is stored in association with the second output result. If the second quality information does not satisfy the second condition, the step of not storing at least a portion of one or more patent documents included in the second patent document in association with the second output result, The program according to claim 3.

9. The first condition includes the second condition, The program according to claim 8.

10. The second condition includes the first condition, The program according to claim 8.

11. The aforementioned processor Of the population information received in the aforementioned population reception step, One or more patent documents stored in association with the information based on the first output result in the first storage step, In the second storage step, one or more patent documents stored in association with the second output result, A processing termination step that terminates the process if no patent documents other than those specified exist, Execute The program according to claim 4.

12. The first processing step is to obtain first classification information relating to the classification of the first patent document, which is output in response to inputting a prompt to the large-scale language model that includes the first patent document and a classification definition document that specifies the classification rules. The second processing step is to obtain second classification information relating to the classification of the second patent document, which is output in response to inputting the second patent document into the learning model, which is a classification model. The program according to claim 1.

13. The aforementioned processor A third patent extraction step in which a third patent document is extracted from among the plurality of patent documents included in the population received in the population reception step, A classification perspective acquisition step involves acquiring classification perspective information relating to the classification perspective of the third patent document, which is output in response to prompt input to a large-scale language model that includes the third patent document extracted in the third patent extraction step and a classification perspective document that defines the classification perspective. A classification definition acquisition step involves acquiring a classification definition document obtained by broadly grouping the classification perspective information relating to the third patent document, which is output in response to inputting a prompt containing the classification perspective information acquired in the classification perspective acquisition step to a large-scale language model, Execute, The first processing step is a step of obtaining the first classification information based on the classification definition document obtained in the classification definition acquisition step. The program according to claim 12.

14. The second processing step includes a step of acquiring second quality information indicating the quality of classification for one or more patent documents contained in the second patent document, which is output in response to inputting the second patent document into the learning model, The aforementioned processor If the second quality information obtained in the second processing step satisfies predetermined conditions, a second classification storage step is performed, in which at least a portion of one or more patent documents included in the second patent document is stored in association with the second classification information. Execute The program according to claim 12.

15. The aforementioned processor A first presentation step in which the first output result obtained in the first processing step is presented to the user, A training data creation step which receives an input operation from the user regarding the first output result presented in the first presentation step, and creates training data relating to the first patent document in accordance with the input operation, Execute, The second processing step is to obtain the second output result which is output in response to inputting the second patent document into the learning model which has been trained based on the training data created in the training data creation step. The program according to claim 1.

16. The first patent extraction step is, The steps include dividing the multiple patent documents included in the population received in the population reception step into multiple groups, A step of extracting one or more patent documents from each of the aforementioned groups as the first patent document, including, The program according to claim 1.

17. A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions according to claims 1 to 16.

18. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in the invention according to any one of claims 1 to 16.

19. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 16.

Images