Information processing system and information processing method

Abstract

[Problem] To realize a conclusive and flexible evaluation in a financial field. [Solution] Provided is an information processing system comprising: a search unit that, on the basis of data relating to a target document in a financial field, acquires data relating to a similar document that is similar to the target document; and an evaluation unit that inputs, to an LLM, a prompt generated dynamically on the basis of the data relating to the target document and the data relating to the similar document, and acquires an evaluation based on the target document. The data relating to the target document includes unstructured data and structured data pertaining to the target document. The data relating to the similar document includes unstructured data and structured data pertaining to the similar document. The unstructured data is written in a natural language, and the structured data is written in a non-natural language.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] For example, as disclosed in Patent Document 1, a technology for supporting evaluation in the financial field has been developed.

[0003] International Publication No. 2020 / 196058

[0004] Valuation in the financial sector is required to be conclusive, but it is also desirable for it to be flexible.

[0005] According to one aspect of the present disclosure, there is provided an information processing system including: a search unit that acquires data on similar documents that are similar to a target document based on data on the target document in the financial field; and an evaluation unit that inputs a prompt that is dynamically generated based on the data on the target document and the data on the similar documents into an LLM and acquires an evaluation based on the target document, wherein the data on the target document includes unstructured data and structured data related to the target document, the data on the similar documents includes unstructured data and structured data related to the similar documents, the unstructured data is data written in a natural language, and the structured data is data written in a non-natural language.

[0006] According to another aspect of the present disclosure, there is provided an information processing method including: a processor obtaining data on similar documents similar to a target document in the financial field based on data on the target document; and inputting dynamically generated prompts based on the data on the target document and the data on the similar documents into an LLM to obtain an evaluation based on the target document, wherein the data on the target document includes unstructured data and structured data related to the target document, the data on the similar documents includes unstructured data and structured data related to the similar documents, the unstructured data is data written in a natural language, and the structured data is data written in a non-natural language.

[0007] FIG. 1 is a block diagram illustrating an overview of the functional configuration of an information processing system 1 according to a first embodiment of the present disclosure. FIG. 2 is a block diagram illustrating details of the functional configuration of the information processing system 1 according to the same embodiment. FIG. 3 is a diagram for describing accumulation of data related to unstructured data and data related to structured data according to the same embodiment. FIG. 4 is a diagram for describing a first evaluation example according to the same embodiment. FIG. 5 is a diagram illustrating an example of a search result 62 in the first evaluation example according to the same embodiment. FIG. 6 is a diagram illustrating an example of an output 64 in the first evaluation example according to the same embodiment. FIG. 7 is a diagram for describing a second evaluation example according to the same embodiment. FIG. 8 is a diagram illustrating an example of a search result 62 in the second evaluation example according to the same embodiment. FIG. 9 is a diagram illustrating an example of an output 64 in the second evaluation example according to the same embodiment. FIG. 10 is a diagram for describing a third evaluation example according to the same embodiment. FIG. 11 is a diagram illustrating an example of a search result 62 in the third evaluation example according to the same embodiment. FIG. 12 is a diagram illustrating an example of an output 64 in the third evaluation example according to the same embodiment. FIG. 13 is a diagram for describing a second embodiment of the present disclosure. FIG. 14 is a diagram for describing an example of a similarity analysis by an analysis unit 250 according to the same embodiment. FIG. 15 is a diagram illustrating an example of a hardware configuration of an information processing device 90 according to an embodiment of the present disclosure.

[0008] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] In addition, in this specification and drawings, when multiple identical components are to be described separately, letters or the like may be added to the end of the reference numerals. On the other hand, when it is not necessary to distinguish between multiple identical components, the letters or the like may be omitted and a description common to all of the multiple identical components may be given.

[0010] The description will be given in the following order: 1. First embodiment 1.1. Overview 1.2. Functional configuration example 1.3. Evaluation example 1.3.1. First evaluation example 1.3.2. Second evaluation example 1.3.3. Third evaluation example 1.3.4. Fourth evaluation example 1.3.5. Fifth evaluation example 2. Second embodiment 3. Hardware configuration example 4. Summary

[0011] 1. First Embodiment 1.1. Overview First, a first embodiment of the present disclosure will be described.

[0012] A first embodiment of the present disclosure relates to document-based evaluation in the financial field.

[0013] In this embodiment, the document to be evaluated (hereinafter referred to as the target document) may be a document such as a disclosure made public by an organization such as a firm, fund, or company.

[0014] In the financial field, evaluations based on such target documents are required to be conclusive.

[0015] Here, for example, when evaluation based on a target document is performed based on past documents similar to the target document (hereinafter referred to as similar documents), it is required to search for similar documents with high accuracy.

[0016] Therefore, one of the features of the information processing system 1 (see Figure 1) of this embodiment is that it classifies and stores data related to documents that have been published in the past into unstructured data and structured data in order to ensure highly accurate searches and, ultimately, definitive evaluations based on highly accurate searches.

[0017] The unstructured data is data written in natural language.

[0018] On the other hand, the structured data is data written in a non-natural language such as a formal language.

[0019] Furthermore, it is desirable for evaluation in the financial sector to be decisive yet flexible.

[0020] Therefore, the information processing system 1 according to this embodiment employs LLM (Large Language Models) to enable flexible evaluation output according to various evaluation criteria.

[0021] <1.2. Functional Configuration Example> Hereinafter, a functional configuration example of the information processing system 1 according to this embodiment will be described in detail with reference to FIGS. 1 to 3. FIG.

[0022] FIG. 1 is a block diagram showing an outline of the functional configuration of an information processing system 1 according to this embodiment.

[0023] As shown in FIG. 1, the information processing system 1 according to this embodiment includes an input unit 110, a search unit 120, a storage unit 130, and an evaluation unit 140.

[0024] FIG. 2 is a block diagram showing the detailed functional configuration of the information processing system 1 according to this embodiment.

[0025] (Input Unit 110 ) The input unit 110 according to this embodiment inputs data related to the target document 50 to the search unit 120 .

[0026] The data relating to the target document 50 includes unstructured data and structured data relating to the target document 50 .

[0027] The unstructured data related to the target document 50 includes at least the unstructured data contained in the target document 50, that is, the sentences written in the target document 50 in a natural language.

[0028] However, the unstructured data related to the target document 50 is not limited to the unstructured data contained in the target document 50 .

[0029] The unstructured data associated with the target document 50 may include unstructured data contained in other documents related to the target document 50 .

[0030] A document related to the target document 50 may be, for example, a document different from the target document 50 that is published by the organization that published the target document 50 or an organization related to that organization.

[0031] Organizations related to the organization that published the target document 50 include firms, funds, and companies.

[0032] For example, if the organization that published the subject document 50 is a firm, documents related to the subject document 50 may be other documents published by funds or companies related to the firm.

[0033] Furthermore, other documents related to the target document 50 may be articles written by third parties about the organization that published the target document 50 or about organizations related to that organization.

[0034] The articles written by third parties include various news articles, articles posted on SNS (Social Networking Service), and the like.

[0035] The data relating to the target document 50 also includes values ​​(hereinafter also referred to as vector values) obtained by vectorizing the unstructured data included in the target document 50 .

[0036] For this purpose, the input unit 110 according to this embodiment includes a natural language processing model that vectorizes unstructured data.

[0037] The LLM 118 is an example of the natural language processing model. However, the natural language processing model is not limited to the LLM. The natural language processing model may be, for example, a Bidirectional Encoder Representations from Transformers (BERT) model.

[0038] The vectorization of unstructured data may be performed by the search unit 120.

[0039] Furthermore, the structured data related to the target document 50 includes at least one of the structured data included in the target document 50 and the structured data included in another document related to the target document 50 .

[0040] The structured data related to the target document 50 may include, for example, financial information of the organization that published the target document 50 or of organizations related to that organization.

[0041] The input unit 110 according to this embodiment inputs data relating to the target document 50, which may include the various types of data described above, to the search unit 120.

[0042] The input unit 110 may further include a configuration for the user to specify the target document 50, evaluation criteria, and the like.

[0043] (Search Unit 120 ) The search unit 120 according to this embodiment acquires data on similar documents that are similar to the target document 50 based on data on the target document 50 input from the input unit 110 .

[0044] More specifically, the search unit 120 according to this embodiment acquires data relating to similar documents based on the vector values ​​of the unstructured data included in the target document 50 .

[0045] The data relating to the similar documents includes unstructured data and structured data relating to the similar documents.

[0046] The unstructured data relating to the similar documents includes at least the unstructured data contained in the similar documents.

[0047] Furthermore, the unstructured data relating to the similar document may include unstructured data contained in other documents related to the similar document.

[0048] Other documents related to the similar document may be the same as other documents related to the target document 50, and therefore detailed description thereof will be omitted.

[0049] Furthermore, the structured data relating to the similar document includes at least one of the structured data included in the similar document and the structured data included in another document related to the similar document.

[0050] The search unit 120 according to this embodiment may acquire unstructured data and structured data associated with the vector values ​​of the unstructured data contained in the similar document based on the vector values ​​of the unstructured data contained in the target document 50.

[0051] Furthermore, the search unit 120 according to this embodiment inputs data on similar documents acquired as a result of the search to the evaluation unit 140 .

[0052] The search unit 120 according to this embodiment includes a search interface 122 for realizing the above.

[0053] (Storage Unit 130) The storage unit 130 according to this embodiment stores (accumulates) data relating to various documents that have been made public (hereinafter referred to as public documents).

[0054] The data about the published documents includes data about unstructured data and data about structured data.

[0055] The data related to the unstructured data includes the unstructured data itself, vector values ​​of the unstructured data, various metadata, and the like.

[0056] Data related to structured data includes the structured data itself, various calculated values ​​based on the structured data, various metadata, and the like.

[0057] The storage unit 130 according to this embodiment may include, for example, an unstructured data DB 132 that stores data related to unstructured data, and a structured data DB 134 that stores data related to structured data.

[0058] In this case, the data regarding unstructured data stored in unstructured data DB 132 and the data regarding structured data stored in structured data DB 134 are connected using identifiers assigned to firms, funds, companies, public documents, text written in public documents, etc. as keys.

[0059] On the other hand, the storage unit 130 may connect data related to unstructured data and data related to structured data in advance and manage them in a unified manner in a single DB.

[0060] Here, with reference to FIG. 3, the accumulation of data related to unstructured data and structured data according to this embodiment will be described.

[0061] As shown in FIG. 3, the unstructured data 1320 according to this embodiment may be categorized into, for example, firm-level information 1321, fund-level information 1322, and company-level information 1323.

[0062] Firm level information 1321 is information at the firm level that establishes the fund.

[0063] Firm-level information 1321 includes, for example, text written in natural language in disclosures made public by the firm.

[0064] Fund level information 1322 is information at the fund level for various types of funds, such as SDGs (Sustainable Development Goals) funds and ESG (Environment, Social, and Governance) funds.

[0065] The fund level information 1322 includes, for example, text written in natural language in disclosures made public by the fund.

[0066] Furthermore, company level information 1323 is company level information that is incorporated into the portfolio of the fund.

[0067] Company level information 1323 includes text written in natural language in disclosures made public by companies.

[0068] More specifically, the company-level information 1323 may include text written in natural language in an integrated report, a sustainability report, an annual report, a security report, and the like.

[0069] The unstructured data 1320 described above is vectorized by the LLM 138, for example.

[0070] Note that vectorization of the unstructured data 1320 is not limited to LLM, and may be performed by BERT or the like.

[0071] However, the same natural language processing model is used for vectorizing the unstructured data 1320 and the unstructured data related to the target document 50 .

[0072] Furthermore, it is desirable that the granularity of the vectorization of the unstructured data 1320 and the vectorization of the unstructured data related to the target document 50 be the same (for example, sentence level, paragraph level, etc.).

[0073] The vector values ​​of the unstructured data 1320 are stored in the unstructured data DB 132 together with other data relating to the unstructured data 1320 .

[0074] Next, the structured data 1340 will be described.

[0075] The structured data 1340 according to this embodiment may be classified into, for example, firm-level information 1341 , fund-level information 1342 , and company-level information 1343 .

[0076] Firm-level information 1341 includes, for example, stock prices and financial information published by the firm.

[0077] Fund level information 1342 includes, for example, historical performance, portfolio weights, etc., as published by the fund.

[0078] Furthermore, the company level information 1343 includes, for example, stock prices and financial information published by the company.

[0079] As shown in the above example, structured data 1340 may include financial information of an organization related to a public document.

[0080] The structured data 1340 is stored in the structured data DB 134 together with various types of metadata.

[0081] (Evaluation Unit 140) The evaluation unit 140 according to this embodiment will be described with reference to FIG. 2 again.

[0082] One of the features of the evaluation unit 140 according to this embodiment is that it dynamically generates prompts 52 to be input to the LLM 148 based on data related to the target document 50 and data related to similar documents.

[0083] The data on the similar documents is the search result by the search unit 120 .

[0084] The data relating to the target document 50 may be input to the evaluation unit 140 from the input unit 110 or from the search unit 120 .

[0085] The evaluation unit 140 according to this embodiment also inputs a prompt 52 to the LLM 148 to obtain an evaluation based on the target document 50 .

[0086] The functional configuration of the information processing system 1 according to the present embodiment has been described above. However, the functional configuration described above with reference to Figures 1 to 3 is merely an example, and the functional configuration of the information processing system 1 according to the present embodiment is not limited to this example.

[0087] 1.3. Evaluation Example Next, the evaluation according to this embodiment will be described in detail.

[0088] The evaluation unit 140 according to this embodiment performs an evaluation based on the target document 50 in relation to a predetermined evaluation criterion, based on data related to the target document 50 and data related to similar documents.

[0089] The evaluation based on the target document 50 may include, for example, an evaluation of the target document 50 itself.

[0090] More specifically, the evaluation unit 140 may evaluate whether the target document 50 satisfies a predetermined disclosure criterion, where the disclosure criterion is an example of the evaluation criterion.

[0091] Examples of the disclosure standards include various standards relating to the disclosure of financial documents, such as the Sustainable Finance Disclosure Regulation (SFDR) in the EU, the Sustainability Disclosure Requirements (SDR) in the UK, and the standards set out in the Task Force on Climate-related Financial Disclosure (TCFD).

[0092] Furthermore, the evaluation based on the target document 50 may include, for example, the evaluation of the organization that publishes the target document 50 or an organization related to that organization.

[0093] Additionally, the evaluation based on the subject document 50 may include, for example, an evaluation of financial products offered by the organization that publishes the subject document 50 or an organization related to that organization.

[0094] Examples of financial products include bonds and stocks.

[0095] Hereinafter, evaluation by the information processing system 1 according to this embodiment will be described in detail with specific examples.

[0096] <1.3.1. First Evaluation Example> First, the first evaluation example will be described.

[0097] In this example, we will discuss credit evaluation (company level) in the selection process.

[0098] In this example, the data related to the target document 50 input to the search unit 120 includes disclosures related to bonds (an example of unstructured data), performance information related to bonds (an example of structured data), and financial information related to the company issuing the bonds (an example of structured data).

[0099] The prompt 52 generated in this example also includes bond information and labels relating to valuation criteria such as TCFD as context to the LLM 148 .

[0100] Furthermore, the prompt 52 generated in this example includes, for example, an instruction to cause the LLM 148 to output the score of the target loan.

[0101] FIG. 4 is a diagram for explaining the first evaluation example.

[0102] The evaluator 140 dynamically generates the prompt 52 based on data 60 about the target document 50 and search results 62 by the searcher 120 based on the data 60 about the target document 50 .

[0103] As shown in FIG. 4, data 60 about target document 50 may include unstructured data, structured data, and metadata.

[0104] Specific examples of the data 60 relating to the target document 50 are as described above.

[0105] Furthermore, examples of metadata include various identifiers such as "Bond ID", "Company ID", and "Sentence ID".

[0106] Similarly, as shown in FIG. 5, search results 62 may also include unstructured data, structured data, and metadata.

[0107] Furthermore, the search results 62 may include labels in addition to the above data.

[0108] The labels may be data relating to evaluations of similar documents based on a certain evaluation criterion, and may be assigned manually in advance, for example.

[0109] The evaluator 140 dynamically generates the prompts 52 based on the data 60 about the target document 50 and the search results 62 as described above.

[0110] Typically, prompts entered into an LLM require five elements: clarity, context, precision, sensitivity, and creativity.

[0111] "Clarity," "context," and "precision" are factors that are sought to make the output of the LLM more definitive.

[0112] "Sensitivity" is a factor required to make the output of LLM less biased.

[0113] Furthermore, "creativity" is an element required to make the output of LLM more creative.

[0114] As described above, evaluations in the financial field are required to be decisive, and therefore the information processing system 1 according to this embodiment may place emphasis on "clarity," "context," and "precision."

[0115] A specific example of the prompt 52 in this example will be described below.

[0116] Prompt 52 in this example may include, for example, a system prompt such as:

[0117] (System prompt) You are an experienced analyst.

[0118] Prompts 52 in this example may include, for example, a main prompt such as:

[0119] (Main prompt) Using multiple examples of sentences that include and do not include information about the SDGs, judge whether the sentence you are evaluating is in line with the SDGs. Also, answer as thoroughly and in detail as possible so that even those without prior knowledge can understand the reason for your judgment. Evaluation target: Text: "XXX" Financial information: "123456" Judgment: "Included" or "Not included" Reason: "______" Reference example (1): Text (1): "YYY" Financial information (1): "987654" Judgment (1): "Included" Reference example (2): Text (2): "ZZZ" Financial information (2): "654987" Judgment (2): "Not included"

[0120] Let's say that the above sentence "XXX" is, for example, the following sentence: "Offshore wind power has great potential for the future expansion of renewable energy sources. We have been working on the technological development of floating offshore wind power generation and have been pursuing advanced initiatives. Japan's floating offshore wind power generation has the potential to generate more than twice the current domestic total annual power generation. We have been continuing commercial operation with a 2MW unit since 2016, and have steadily demonstrated safety, etc. We will work to expand our offshore wind power generation business in order to contribute to solving Japan's energy challenges."

[0121] Sentence (1) above: "YYY" is, for example, the following sentence: "Our company promotes environmental management through four approaches: environmental conservation, environmental business, environmental risk management, and environmental social contribution. In May 2019, we revised our Basic Environmental Policy to clarify our commitment to contributing to the achievement of the SDGs. Our basic stance throughout the entire group is to conduct business activities with minimal environmental impact and to create and restore the environment through each process of all business activities. Our efforts to address environmental issues are perpetual, and we will contribute to a sustainable global society by steadily promoting our medium- to long-term goal, "Ecology Mission 2030-2050," which aims to reduce CO2 emissions. In particular, we view climate change caused by global warming as one of our important management issues. We have expressed our support for the recommendations of the Task Force on Climate-related Financial Disclosures (TCFD). We are analyzing the risks and opportunities related to climate change and taking measures to address it." We will also contribute to the sustainability of the global environment by working on green infrastructure initiatives that develop infrastructure while wisely utilizing the functions of nature, and by working to recycle the earth's limited resources.

[0122] Furthermore, the above sentence (2): "ZZZ" could be, for example, the following sentence: "To ensure the sustainable growth of our domestic construction business, we are working to further strengthen our current production system and foundation, while also making every effort to implement various measures to improve productivity and strengthen our business competitiveness. We are promoting business process reforms based on the keywords of 'standardization,' 'ICT,' and 'cloud computing,' as well as the development and deployment of innovative technologies such as autonomous control robots and construction methods that contribute to the streamlining of construction, and various measures to build a production system based on BIM. Furthermore, in response to the diversifying needs of society and customers that accompany the advancement of cutting-edge technology, we are flexibly restructuring our sales strategies and organization, and are working to strengthen our proposal capabilities by utilizing computational design."

[0123] The evaluation unit 140 generates information about the above-mentioned “evaluation target” based on data 60 about the target document 50 and incorporates it into the prompt 52 .

[0124] Furthermore, the evaluation unit 140 generates information about the above-mentioned "reference examples" based on the search results 62 and includes the information in the prompt 52. In this way, the search results 62 can be used as labeled samples in the short-shot learning of the LLM 148.

[0125] The evaluation unit 140 inputs the generated prompt 52 to the LLM 148, and obtains an output 64 from the LLM 148. The output 64 can be said to be an evaluation result based on the target document 50.

[0126] Note that each "financial information" in the system prompts in this example is not essential and may be omitted.

[0127] FIG. 6 is a diagram for explaining an example of the output 64 in the first evaluation example.

[0128] As shown in FIG. 6, the output 64 may include metadata such as various identifiers, a "Predicted label" (which may be a score, etc.), an "Explanation of prediction", and the like.

[0129] Based on the output 64, the evaluation unit 140 performs control to present the following information to the user, for example:

[0130] (Example) Verdict: "Included" Reason: While the text being evaluated does not directly reference the Sustainable Development Goals (SDGs), it does include content that is aligned with two key SDG themes: "Ensure access to clean energy and sustainable energy for all" (SDG 7) and "Take urgent action to combat climate change" (SDG 13). The text mentions that the development and expansion of offshore wind power, particularly floating offshore wind power technology, will promote the use of renewable energy sources and address Japan's energy challenges. This directly contributes to the SDG goals of reducing greenhouse gas emissions by reducing fossil fuel use and promoting the widespread use of renewable energy. Furthermore, the development and expansion of renewable energy sources will contribute to the promotion of sustainable industry, innovation, and infrastructure (SDG 9) and the realization of sustainable cities and communities (SDG 11). These goals aim to improve the sustainability of society as a whole and promote balanced development in economic, environmental, and social aspects. The text mentions efforts to develop technology, to implement commercial operations and demonstrate their safety, and to expand the business, all of which can be interpreted as concrete actions toward achieving the SDGs. Therefore, even though the word "SDGs" is not directly used, it can be determined that the text describes efforts that are in line with the spirit and goals of the SDGs.

[0131] The “decision” corresponds to the “Predicted label” in the output 64 , and the “reason” corresponds to the “Explanation of prediction” in the output 64 .

[0132] <1.3.2. Second Evaluation Example> Next, a second evaluation example will be described.

[0133] In this example, we will discuss stock valuation (company level) in the selection process.

[0134] In this example, the data 60 relating to the target document 50 input to the search unit 120 includes company disclosures (an example of unstructured data), stock performance information (an example of structured data), financial information relating to the company issuing the stock (an example of structured data), etc.

[0135] FIG. 7 is a diagram for explaining the second evaluation example.

[0136] As in the first evaluation example, the evaluation unit 140 dynamically generates the prompt 52 based on data 60 related to the target document 50 and search results 62 based on the data 60 related to the target document 50 by the search unit 120 .

[0137] Also, as shown in FIG. 7, data 60 about target document 50 may include unstructured data, structured data, and metadata.

[0138] Specific examples of the data 60 relating to the target document 50 are as described above.

[0139] The metadata may include various identifiers such as "Stock ID", "Company ID", and "Sentence ID".

[0140] On the other hand, as shown in FIG. 8, search results 62 may include unstructured data, structured data, metadata, and labels.

[0141] The evaluator 140 dynamically generates the prompts 52 based on the data 60 about the target document 50 and the search results 62 as described above.

[0142] A specific example of the prompt 52 in this example will be described below.

[0143] Prompt 52 in this example may include, for example, a system prompt such as:

[0144] (System prompt) You are an experienced analyst.

[0145] Prompts 52 in this example may include, for example, a main prompt such as:

[0146] (Main prompt) Based on company reports, financial information, and multiple examples of stock price increases or decreases, determine whether to buy or sell the stock of the company being evaluated. Also, please provide as much detail and detail as possible so that even those without prior knowledge can understand the reason for your decision. Evaluation target: Text: "XXX" Financial information: "123456" Decision: "Sell" or "Buy" Reason: "______" Reference example (1): Text (1): "YYY" Financial information (1): "987654" Decision (1): "Sell" ...

[0147] The evaluation unit 140 generates information about the above-mentioned “evaluation target” based on data 60 about the target document 50 and incorporates it into the prompt 52 .

[0148] Furthermore, the evaluation unit 140 generates information about the above-mentioned "reference examples" based on the search results 62 and includes the information in the prompt 52. In this way, the search results 62 can be used as labeled samples in the short-shot learning of the LLM 148.

[0149] The evaluation unit 140 inputs the generated prompt 52 to the LLM 148, and obtains an output 64 from the LLM 148. The output 64 can be said to be an evaluation result based on the target document 50.

[0150] Note that each "financial information" in the system prompts in this example is not essential and may be omitted.

[0151] FIG. 9 is a diagram for explaining an example of the output 64 in the second evaluation example.

[0152] As shown in FIG. 9, the output 64 may include metadata such as various identifiers, a "Predicted label" (which may be a score, etc.), an "Explanation of prediction", and the like.

[0153] As in the first evaluation example, the evaluation unit 140 performs control to present an evaluation based on the target document 50 to the user based on the output 64 .

[0154] <1.3.3. Third Evaluation Example> Next, a third evaluation example will be described.

[0155] In this example, we will discuss monitoring sustainability activities (company level).

[0156] In this example, the data 60 related to the target document 50 input to the search unit 120 includes the company's sustainability report (an example of unstructured data), news articles and social media articles related to the company obtained from external sources (an example of unstructured data), and financial information related to the company (an example of structured data).

[0157] Thus, data 60 about the subject document 50 may include articles written by third parties about the organization that published the subject document 50 or organizations related to that organization.

[0158] FIG. 10 is a diagram for explaining the third evaluation example.

[0159] As in the first and second evaluation examples, the evaluation unit 140 dynamically generates a prompt 52 based on data 60 about the target document 50 and search results 62 by the search unit 120 based on the data 60 about the target document 50.

[0160] Also, as shown in FIG. 10, data 60 about target document 50 may include unstructured data, structured data, and metadata.

[0161] Specific examples of the data 60 relating to the target document 50 are as described above.

[0162] The metadata may include various identifiers such as "Source Type," "Company ID," and "Sentence ID."

[0163] On the other hand, as shown in FIG. 11, search results 62 may include unstructured data, structured data, metadata, and labels.

[0164] The search results 62 may include data 60 about the target document 50 as well as articles written by third parties about the organization that published the target document 50 or organizations related to that organization.

[0165] The evaluator 140 dynamically generates the prompts 52 based on the data 60 about the target document 50 and the search results 62 as described above.

[0166] A specific example of the prompt 52 in this example will be described below.

[0167] Prompt 52 in this example may include, for example, a system prompt such as:

[0168] (System prompt) You are an experienced analyst.

[0169] Prompts 52 in this example may include, for example, a main prompt such as:

[0170] (Main prompt) Using company reports, financial information, and articles about the company, judge whether the activities of the company being evaluated are good or bad in relation to the TCFD. Also, please provide as much detail and detail as possible so that even those without prior knowledge can understand the reason for your judgment. Evaluation target: Text: "XXX" Financial information: "123456" Judgment: "Good" or "Bad" Reason: "______" Reference example (1): Text (1): "YYY" Financial information (1): "987654" Judgment (1): "Bad" ...

[0171] The evaluation unit 140 generates information about the above-mentioned “evaluation target” based on data 60 about the target document 50 and incorporates it into the prompt 52 .

[0172] Furthermore, the evaluation unit 140 generates information about the above-mentioned "reference examples" based on the search results 62 and includes the information in the prompt 52. In this way, the search results 62 can be used as labeled samples in the short-shot learning of the LLM 148.

[0173] The evaluation unit 140 inputs the generated prompt 52 to the LLM 148, and obtains an output 64 from the LLM 148. The output 64 can be said to be an evaluation result based on the target document 50.

[0174] Note that each "financial information" in the system prompts in this example is not essential and may be omitted.

[0175] FIG. 12 is a diagram for explaining an example of the output 64 in the third evaluation example.

[0176] As shown in FIG. 12, the output 64 may include metadata such as various identifiers, a "Predicted label" (which may be a score, etc.), an "Explanation of prediction", and the like.

[0177] As in the first and second evaluation examples, the evaluation unit 140 performs control to present an evaluation based on the target document 50 to the user based on the output 64 .

[0178] Furthermore, the evaluation unit 140 may aggregate the evaluation results including the "Predicted label" for each organization such as a company, and store the results as time-series data. Such control makes it possible to continuously track the activities of the organization.

[0179] <1.3.4. Fourth Evaluation Example> Next, a fourth evaluation example will be described.

[0180] In this example, we will discuss fund evaluation in the selection process.

[0181] In this example, examples of unstructured data included in the data 60 relating to the target document 50 input to the search unit 120 include disclosures of funds, disclosures of bonds or stocks in which the funds invest, and the like.

[0182] In addition, in this example, examples of structured data contained in the data 60 regarding the target document 50 input to the search unit 120 include the fund's portfolio, performance information of the fund, performance information of bonds or stocks etc. in which the fund invests, financial information regarding the company issuing the bonds or stocks etc., etc.

[0183] The prompt 52 in this example includes, for example, fund information and scores (labels) relating to evaluation criteria such as SFDR and SDR as "reference examples."

[0184] Furthermore, the prompt 52 in this example includes an instruction to output the score (label) of the target fund with respect to the above evaluation criteria.

[0185] The prompt 52, data 60 relating to the target document 50, search results 62, and output 64 in this example may be generated by the same information processing methods as those in the first to third evaluation examples, and therefore detailed examples will be omitted.

[0186] <1.3.5. Fifth Evaluation Example> Next, a fifth evaluation example will be described.

[0187] In this example, we describe monitoring a fund's activities related to sustainability.

[0188] In this example, examples of unstructured data contained in the data 60 relating to the target document 50 input to the search unit 120 include fund disclosures, disclosures of bonds or stocks in which the fund invests, articles written by third parties about the fund or organizations related to the fund, etc.

[0189] In addition, in this example, examples of structured data contained in the data 60 regarding the target document 50 input to the search unit 120 include the fund's portfolio, performance information of the fund, performance information of bonds or stocks etc. in which the fund invests, financial information regarding the company issuing the bonds or stocks etc., etc.

[0190] The prompt 52 in this example includes, for example, fund information and scores (labels) relating to evaluation criteria such as SFDR and SDR as "reference examples."

[0191] Furthermore, the prompt 52 in this example includes an instruction to output the score (label) of the target fund with respect to the above evaluation criteria.

[0192] The prompt 52, data 60 relating to the target document 50, search results 62, and output 64 in this example may be generated by the same information processing methods as those in the first to third evaluation examples, and therefore detailed examples will be omitted.

[0193] The information processing method according to the present embodiment and the information processing system 1 that implements the information processing method have been described in detail above.

[0194] According to the technical concept of this embodiment, it is possible to realize decisive and flexible evaluation in the financial field.

[0195] 2. Second Embodiment Next, a second embodiment of the present disclosure will be described.

[0196] In the first embodiment, the information processing system 1 that supports various evaluations in the financial field has been described.

[0197] However, the technical concept described in the first embodiment can be widely applied to applications other than evaluation support, such as support for responding to inquiries at a customer center or the like.

[0198] For example, it is assumed that sentences relating to past inquiries and sentences relating to answers are stored in a database together with vector values ​​of the sentences relating to the past inquiries.

[0199] In this case, it is possible to search for sentences related to similar past inquiries and sentences related to answers based on the vector value of the new inquiry, and input a prompt generated based on the search results and the new inquiry into the LLM, thereby causing the LLM to output an answer to the new inquiry.

[0200] However, if there is insufficient data accumulated regarding past inquiries and answers (hereinafter also referred to as Q&A), situations may arise in which sufficient search accuracy cannot be ensured or hallucination occurs.

[0201] Therefore, in a second embodiment of the present disclosure, an information processing method is proposed in which, as a prelude to automatically generating answers using an LLM, a template for responding to an inquiry is generated from data related to past Q&As using an LLM.

[0202] In the following, we will mainly use the example of how to handle inquiries at a bank.

[0203] In the following, differences from the first embodiment will be mainly described, and detailed description of the configuration, functions, effects, etc. common to the first embodiment will be omitted.

[0204] FIG. 13 is a diagram for explaining the second embodiment of the present disclosure.

[0205] As shown in FIG. 13 , the information processing system 2 according to the second embodiment includes at least a Q&A DB 232 , an analysis unit 250 , and an LLM 248 .

[0206] (Q&A DB 232) The Q&A DB 232 records data related to past Q&As. The data related to past Q&As includes unstructured data and structured data.

[0207] The unstructured data includes, for example, text relating to past inquiries, vectorized values ​​of the text, and text relating to past answers.

[0208] The structured data includes, for example, attributes of the customer who made the inquiry.

[0209] (Analysis Unit 250) The analysis unit 250 performs a similarity analysis on data relating to past Q&As stored in the Q&A DB 232, and extracts past Q&As that are frequently asked and have a high degree of similarity.

[0210] The similarity analysis by the analysis unit 250 is performed to generate more useful templates.

[0211] FIG. 14 is a diagram for explaining an example of similarity analysis by the analysis unit 250. In FIG.

[0212] The left side of Fig. 14 illustrates the distribution of inquiries related to "opening an account" across all past Q&As. The center of Fig. 14 illustrates the distribution of inquiries related to "debit cards" across all past Q&As. The right side of Fig. 14 illustrates the distribution of inquiries related to "foreign currency deposits" across all past Q&As.

[0213] The analysis unit 250 can obtain the above distribution by, for example, vectorizing sentences related to past inquiries and projecting the vectorized values ​​into two dimensions.

[0214] Furthermore, the analysis unit 250 identifies inquiries with high density, i.e., inquiries with high frequency and high similarity, from the distribution obtained as described above, and extracts data related to the inquiries and their corresponding answers (data related to Q&As).

[0215] The description will continue with reference to FIG.

[0216] (LLM 248) The LLM 248 generates a template 74 based on data relating to high frequency and highly similar Q&As extracted by the analysis unit 250.

[0217] More specifically, the LLM 248 may generate a summary of the extracted high frequency and high similarity Q&As as a template 74 .

[0218] For this reason, the prompt input to the LLM 248 includes data relating to the extracted high frequency and high similarity Q&As and a sentence instructing the data to be summarized.

[0219] Below are examples of extracted high frequency and high similarity Q&As, and examples of templates 74 output by the LLM 248.

[0220] (Past inquiry) We purchase and sell recycling-related equipment from overseas companies, and would like to open an account to make payments to our suppliers.

[0221] (Previous answer) We are sorry, but ABC Bank only accepts individual customers for account openings. Please note that we cannot accept accounts for corporations, sole proprietors whose account name includes a trade name, or accounts for handling business funds. Thank you for your understanding.

[0222] (Example of Template 74) Key points of the answer: Key points: ABC Bank accounts can only be opened by individual customers. Accounts cannot be opened by corporations or sole proprietors whose account name is included in the account name.

[0223] Furthermore, when there are multiple extracted Q&As that are highly frequent and similar to each other, the LLM 248 can also combine multiple summaries into a template 74 as follows.

[0224] (Example of Template 74) Key points of the answer: Key point (1): Accounts can only be opened with ABC Bank by individual customers. We cannot open accounts for corporations or sole proprietors whose account name includes a trade name. Key point (2): If you have multiple countries of residence and are applying to open an account, you can notify us of your country of residence other than Japan by submitting a separate "Notification of Country of Residence." Key point (3): You can continue to use ABC Bank even if you change your address to a foreign country, but you will not be able to make foreign currency remittances (outbound remittances) from ABC Bank. If you change your address to a foreign country, there are some services that we will no longer be able to use and some transactions will be restricted.

[0225] The generation of the template 74 using the LLM 248 according to this embodiment has been described above.

[0226] The template 74 generated as described above enables an operator at a customer center or the like to accurately grasp summaries of past Q&As and to respond efficiently and effectively to new inquiries.

[0227] The template 74 may be used in business after being manually checked, corrected, etc.

[0228] The templates 74 may also be evaluated based on a similarity analysis.

[0229] For example, by analyzing the similarity between the generated template 74 and an actual Q&A, it is possible to evaluate the expected frequency of use of the template 74, etc.

[0230] Furthermore, for example, it is possible to evaluate the expected frequency of use of the template 74 based on whether the generated template 74 is included in a cluster that has a high frequency and a high degree of similarity and that has been extracted in advance.

[0231] The generated template 74 may be evaluated by a person based on the context of the text, etc.

[0232] The generated template 74 may also be used for short-shot learning of the LLM 248.

[0233] This is expected to enable the LLM 248 to output highly accurate answers to new inquiries by referencing summaries of past Q&As.

[0234] As described above, according to the information processing system 2 according to the second embodiment of the present disclosure, it is possible to automatically generate a large number of templates 74 that summarize past Q&As, thereby effectively supporting operators in responding to inquiries.

[0235] Furthermore, according to the information processing system 2 according to the second embodiment of the present disclosure, it is expected that the LLM 248 will automate the response to inquiries.

[0236] 3. Hardware Configuration Example Next, a hardware configuration example of the information processing device 90 according to an embodiment of the present disclosure will be described. Fig. 15 is a block diagram showing a hardware configuration example of the information processing device 90 according to an embodiment of the present disclosure.

[0237] The information processing device 90 may be a device having a hardware configuration equivalent to that of a device including the input unit 110, search unit 120, memory unit 130, evaluation unit 140, Q&A DB 232, analysis unit 250, and LLM 248.

[0238] 15 , the information processing device 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the information processing device 90 may include further components in addition to the components shown here.

[0239] (Processor 871) The processor 871 functions, for example, as an arithmetic processing device or control device, and controls the overall operation of each component or part of it based on various programs recorded in the ROM 872, RAM 873, storage 880, or removable storage medium 901.

[0240] (ROM 872, RAM 873) The ROM 872 is a means for storing programs to be read into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores, for example, the programs to be read into the processor 871 and various parameters that change as appropriate when the programs are executed.

[0241] (Host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 that is capable of high-speed data transmission. On the other hand, the host bus 874 is connected to, for example, an external bus 876 that has a relatively low data transmission speed via a bridge 875. Furthermore, the external bus 876 is connected to various components via an interface 877.

[0242] (Input Device 878) For example, a mouse, keyboard, touch panel, button, switch, lever, etc. are used as the input device 878. Furthermore, a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves may also be used as the input device 878. The input device 878 also includes an audio input device such as a microphone.

[0243] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimulation.

[0244] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.

[0245] (Drive 881) The drive 881 is a device that reads information recorded on a removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901.

[0246] (Removable storage medium 901) The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable storage medium 901 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0247] (Connection Port 882) The connection port 882 is a port for connecting an external device 902, such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal.

[0248] (Externally Connected Device 902) The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0249] (Communication device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0250] 4. Summary As described above, the information processing system 1 according to the first embodiment of the present disclosure includes a search unit 120 that acquires data on similar documents that are similar to the target document 50 based on data on the target document 50 in the financial field, and an evaluation unit 140 that inputs the prompt 52 that is dynamically generated based on the data 60 on the target document 50 and the data on the similar documents into the LLM 148 and acquires an evaluation based on the target document 50.

[0251] The data 60 relating to the target document 50 includes unstructured data and structured data relating to the target document 50, and the data relating to the similar documents includes unstructured data and structured data relating to the similar documents.

[0252] Unstructured data is data written in a natural language, and structured data is data written in a non-natural language.

[0253] According to the above-described configuration, it is possible to realize decisive and flexible evaluation in the financial field.

[0254] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0255] Furthermore, the steps of the processes described in this disclosure do not necessarily have to be processed in chronological order according to the order shown in the flowcharts or sequence diagrams. For example, the steps of the processes of each device may be processed in an order different from the order shown, or may be processed in parallel.

[0256] Furthermore, the series of processes performed by each device described in this disclosure may be realized by a program stored in a non-transitory computer-readable storage medium. Each program is, for example, loaded into RAM when executed by a computer and executed by a processor such as a CPU. The storage medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The program may also be distributed, for example, via a network, without using a storage medium.

[0257] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0258] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing system comprising: a search unit that acquires data on similar documents similar to a target document based on data on the target document in the financial field; and an evaluation unit that inputs prompts dynamically generated based on the data on the target document and the data on the similar documents into an LLM and acquires an evaluation based on the target document, wherein the data on the target document includes unstructured data and structured data on the target document, the data on the similar documents includes unstructured data and structured data on the similar documents, the unstructured data is data described in a natural language, and the structured data is data described in a non-natural language. (2) The information processing system described in (1), wherein the search unit acquires data on the similar documents based on a vectorized value of the unstructured data included in the target document. (3) The information processing system described in (2), wherein the search unit acquires unstructured data and structured data associated with a vectorized value of the unstructured data included in the similar documents based on the vectorized value of the unstructured data included in the target document. (4) The information processing system according to any one of (1) to (3), wherein the structured data includes financial information. (5) The information processing system according to any one of (1) to (4), wherein the prompt includes a reference example. (6) The information processing system according to (5), wherein the data regarding the similar documents further includes an evaluation based on the similar documents. (7) The information processing system according to any one of (1) to (6), wherein the prompt includes evaluation criteria. (8) The information processing system according to (7), wherein the evaluation criteria include disclosure criteria. (9) The information processing system according to any one of (1) to (8), wherein the target document and the similar documents include disclosure.(10) The information processing system according to any one of (1) to (9), wherein the data related to the target document includes an article written by a third party about an organization related to the target document. (11) The information processing system according to any one of (1) to (10), wherein the vectorization of structured data related to the target document and the vectorization of unstructured data related to the similar documents are performed using the same natural language processing model. (12) The information processing system according to (8), wherein the evaluation based on the target document includes whether the target document satisfies the disclosure standards. (13) The information processing system according to any one of (1) to (12), wherein the evaluation based on the target document includes an evaluation of a financial product. (14) The information processing system according to any one of (1) to (13), wherein the evaluation based on the target document includes an evaluation of an organization that publishes the target document. (15) An information processing method comprising: a processor, based on data about a target document in the financial field, acquiring data about similar documents that are similar to the target document; and inputting dynamically generated prompts based on the data about the target document and the data about the similar documents into an LLM to acquire an evaluation based on the target document, wherein the data about the target document includes unstructured data and structured data about the target document, the data about the similar documents includes unstructured data and structured data about the similar documents, the unstructured data is data described in a natural language, and the structured data is data described in a non-natural language. (16) The information processing method according to (15), wherein acquiring the data about the similar documents acquires the data about the similar documents based on vectorized values ​​of unstructured data included in the target document.(17) The information processing method according to (16), wherein acquiring the data related to the similar documents includes acquiring unstructured data and structured data associated with a vectorized value of unstructured data included in the similar documents, based on a vectorized value of unstructured data included in the target document. (18) The information processing method according to any one of (15) to (17), wherein the structured data includes financial information. (19) The information processing method according to any one of (15) to (18), wherein the prompt includes a reference example. (20) The information processing method according to (19), wherein the data related to the similar documents further includes an evaluation based on the similar documents.

[0259] REFERENCE SIGNS LIST 1 Information processing system 2 Information processing system 50 Target document 52 Prompt 60 Data related to target document 110 Input unit 120 Search unit 130 Storage unit 132 Unstructured DB 134 Structured DB 140 Evaluation unit 148 LLM

Claims

1. An information processing system comprising: a search unit that acquires data on similar documents that are similar to a target document based on data on the target document in the financial field; and an evaluation unit that inputs dynamically generated prompts based on the data on the target document and the data on the similar documents into an LLM and acquires an evaluation based on the target document, wherein the data on the target document includes unstructured data and structured data related to the target document, the data on the similar documents includes unstructured data and structured data related to the similar documents, the unstructured data is data written in a natural language, and the structured data is data written in a non-natural language.

2. The information processing system according to claim 1, wherein the search unit acquires data relating to the similar documents based on vectorized values ​​of unstructured data included in the target document.

3. The information processing system of claim 2, wherein the search unit acquires unstructured data and structured data associated with vectorized values ​​of unstructured data contained in the similar documents based on vectorized values ​​of unstructured data contained in the target document.

4. The information processing system of claim 1, wherein the structured data includes financial information.

5. The information processing system of claim 1, wherein the prompt includes a reference example.

6. The information processing system according to claim 5, wherein the data regarding the similar documents further includes ratings based on the similar documents.

7. The information processing system of claim 1, wherein the prompt includes evaluation criteria.

8. The information processing system according to claim 7, wherein the evaluation criteria include disclosure criteria.

9. The information processing system according to claim 1, wherein the target document and the similar documents include disclosures.

10. The information processing system of claim 1, wherein the data relating to the target document includes articles written by third parties about organizations related to the target document.

11. The information processing system according to claim 1, wherein the vectorization of the structured data related to the target document and the vectorization of the unstructured data related to the similar documents are performed using the same natural language processing model.

12. The information processing system according to claim 8, wherein the evaluation based on the target document includes whether the target document satisfies the disclosure criteria.

13. The information processing system according to claim 1, wherein the evaluation based on the target document includes an evaluation of a financial product.

14. The information processing system according to claim 1, wherein the reputation based on the target document includes a reputation of an organization that publishes the target document.

15. An information processing method comprising: a processor obtaining data on similar documents similar to a target document in the financial field based on data on the target document; and inputting dynamically generated prompts based on the data on the target document and the data on the similar documents into an LLM to obtain an evaluation based on the target document; wherein the data on the target document includes unstructured data and structured data related to the target document; the data on the similar documents includes unstructured data and structured data related to the similar documents; the unstructured data is data written in a natural language; and the structured data is data written in a non-natural language.

16. The information processing method according to claim 15, wherein acquiring data related to the similar documents comprises acquiring data related to the similar documents based on vectorized values ​​of unstructured data included in the target document.

17. The information processing method of claim 16, wherein acquiring data related to the similar documents comprises acquiring unstructured data and structured data associated with vectorized values ​​of unstructured data included in the similar documents based on vectorized values ​​of unstructured data included in the target document.

18. The information processing method of claim 15, wherein the structured data includes financial information.

19. The information processing method of claim 15, wherein the prompt includes a reference example.

20. The information processing method according to claim 19, wherein the data regarding the similar documents further includes ratings based on the similar documents.

Images