A Utility Testing Method and System for Generative AI in Reasoning for Disclosure Report Information

Abstract

This invention belongs to the field of natural language processing technology, specifically relating to a method and system for testing the utility of generative AI in information reasoning for disclosure reports. The method includes analyzing corporate disclosure reports using a generative AI model, determining the optimal number of topics and keywords through an LDA topic model, calculating topic word vectors and report sentence vectors using word2vec and BERT models respectively, and then determining the report's topic ratio vector. The Pearson chi-square test is used to calculate the statistics of the topic ratio vectors of two pairs of texts, and paired t-tests are used to calculate the significance of N statistics to evaluate the information reasoning effect of the AI ​​model. Finally, the KL divergence metric is used to quantify the degree of information reasoning effect. This invention improves the efficiency of disclosure report analysis, reduces labor costs, and ensures the reliability and accuracy of information reasoning results, providing strong decision support for investors, regulatory agencies, etc., and has broad application prospects and promotional value.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a method and system for testing the utility of generative AI for reasoning in disclosure reports. Background Technology

[0002] Current technologies primarily rely on manual reading and analysis of large volumes of corporate disclosure reports, a method that is inefficient and susceptible to subjectivity. With the development of information technology, traditional computational models such as database queries and keyword retrieval are relatively effective in processing structured data, but their ability to extract unstructured text and deeper information is limited. Statistical analysis methods mainly focus on quantitative data, lacking in-depth analysis of text content. In recent years, machine learning models, especially natural language processing (NLP) techniques, have been used for text analysis, but they still have limitations in understanding the deeper meaning and sentiment factors of text. Advanced generative AI models such as ChatGPT have shown great potential in the field of text analysis and processing, capable of improving the efficiency of text analysis through various tasks such as natural language generation, understanding, and processing.

[0003] While existing technologies have improved the analytical capabilities of corporate disclosure reports to some extent, several shortcomings remain. First, the problem of insufficient or inaccurate information disclosure persists, failing to fundamentally resolve information asymmetry. Second, traditional analytical methods often neglect the analysis of textual content when assessing disclosure quality, particularly key information and sentiment factors in interim announcements. Furthermore, existing methods still have limitations in handling unstructured text and complex information, failing to fully extract the informational value from the text. Although generative AI models such as ChatGPT possess powerful text processing capabilities, the effectiveness and accuracy of their information reasoning functions still require further testing. Summary of the Invention

[0004] The purpose of this invention is to provide a utility testing method and system for generative AI to reason about disclosure report information, in order to solve the problems of low efficiency in corporate disclosure report analysis, susceptibility to subjective influence, and limited ability of traditional computing models and statistical analysis methods to process unstructured text in the prior art.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] Firstly, this invention proposes a utility testing method for generative AI used in reasoning about disclosure report information, the method comprising the following steps:

[0007] S1. Based on preset prompt words, use a generative AI model to analyze N enterprise disclosure reports to obtain N analysis texts. Preprocess the analysis texts to generate a document-word matrix.

[0008] S2. Use the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determine the keywords and corresponding word vectors for each topic based on the word probability distribution;

[0009] S3. Based on the word vectors of keywords in each topic, determine the topic ratio vector in each company's disclosure report;

[0010] S4. Based on the sentence vector of each sentence in the N pre-determined analysis texts, determine the topic ratio vector of the analysis text for each generative AI model;

[0011] S5. Calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and perform a paired t-test. Based on the significance level of the test results, determine whether the generative AI model has the function of information reasoning.

[0012] S6. Utilize the topic distribution vector of each enterprise's disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model.

[0013] Furthermore, in step S1, the preprocessing includes: performing word segmentation, word frequency statistics, stop word removal, and low-frequency word processing on the document set of enterprise disclosure reports to obtain a document-word matrix. .

[0014] Furthermore, step S2 includes:

[0015] S201, Using LDA topic modeling to... Topic analysis was conducted, and the optimal number of topics k was determined using the perplexity index, resulting in k topics and the word probability distribution for each topic. ,in, Indicates the first One topic, express The total number of Chinese words, selecting the first... Each word serves as a keyword for each topic;

[0016] S202. Calculate the word vector for each keyword in each topic corresponding to the enterprise disclosure report using the word2vec model. The first topic The word vectors of each keyword are ,in, This represents the parameters of the word2vec model, including vector dimensions and window size.

[0017] Furthermore, step S3 includes:

[0018] S301. Decompose each company's disclosure report into sentences to obtain a set of sentences. , in, It is the number of sentences in the document;

[0019] S302, For each sentence The sentence vectors are calculated using the BERT model, and the vectors labeled [CLS] are used as sentence vectors. ,in, Sentence The sentence vector, This indicates that the BERT model represents the sentence. The encoded output, [CLS] represents the vector corresponding to the [CLS] tag extracted from the BERT model output;

[0020] S303. Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Sentence The sentence vector, Indicates the first The first topic Word vectors of each keyword, Represents the dot product of two vectors. Let represent the norms of the two vectors respectively;

[0021] S304. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic, where Indicates the topic The total number of keywords in the text;

[0022] S305. Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the company's disclosure report. ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. This indicates that the topic is discussed in the analysis text of N corporate disclosure reports. The proportion.

[0023] Furthermore, step S4 includes:

[0024] S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set. , in It is the number of sentences in the document;

[0025] S402, For each sentence The sentence vectors are calculated using the BERT model, and the vectors labeled [CLS] are used as sentence vectors. ,in, Sentence The sentence vector, This indicates that the BERT model represents the sentence. The encoded output, [CLS] represents the vector corresponding to the [CLS] tag extracted from the BERT model output;

[0026] S403, Calculate each sentence sentence vector Word vectors of each topic keyword in the corporate disclosure report text Cosine similarity between them; for each topic and keywords The cosine similarity of word vectors for each keyword in each topic. , Sentence The sentence vector, It is the first The word vector of a keyword in a topic. Represents the dot product of two vectors. Let represent the norms of the two vectors respectively;

[0027] S404. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic. The topic with the highest similarity is selected as the topic of the sentence. The ratio vector of each topic in the analyzed text of the generative AI model is calculated. ,in Indicates the first The ratio of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document. The representative discusses the topic in the analysis text of N generative AI models. The proportion.

[0028] Furthermore, step S5 includes:

[0029] The chi-square statistic for each pair of corporate disclosure report documents and the text document analyzed by the generative AI model was calculated, resulting in N pairs of chi-square statistics. A paired t-test was performed on the N pairs of chi-square statistics. If the p-value is <0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning capabilities.

[0030] Furthermore, step S6 includes:

[0031] S601. Analyze the topic distribution vector of each enterprise's disclosure report using the analysis text and the corresponding generative AI model. and ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. Indicates the first The ratio of the number of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document;

[0032] S602. Calculate the KL divergence between the analytical text of each enterprise disclosure report and the corresponding analytical text of the generative AI model. Then, the mean of the N KL divergences is calculated to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of the KL divergence, as shown in the following formula:

[0033] .

[0034] Secondly, the present invention proposes a utility testing system for generative AI used in reasoning about disclosure report information. The system is applied to perform any of the effect testing methods described above, and the system includes:

[0035] The analysis module is used to analyze N corporate disclosure reports based on preset prompt words using a generative AI model to obtain N analysis texts, and to preprocess the analysis texts to generate a document-word matrix;

[0036] The topic determination module is used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and to determine the keywords and corresponding word vectors for each topic based on the word probability distribution;

[0037] The first calculation module is used to determine the topic ratio vector of each company's disclosure report based on the word vectors of the keywords in each topic;

[0038] The second calculation module is used to determine the topic ratio vector of the analysis text of each generative AI model based on the sentence vector of each sentence in the N pre-determined analysis texts;

[0039] The effect testing module is used to calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and to perform a paired t-test. Based on the significance level of the test results, it is determined whether the generative AI model has the function of information reasoning.

[0040] The effect degree determination module is used to analyze the topic distribution vector of each enterprise disclosure report and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the effect degree of information inference of the generative AI model.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention introduces a scientific testing method to evaluate the information reasoning utility of generative AI models. By determining the optimal number of topics using LDA topic modeling, calculating word and sentence vectors using word2vec and BERT models, and combining statistical methods such as Pearson chi-square test and KL divergence, this invention can objectively and accurately measure the performance of generative AI models in information reasoning. This provides data support for model optimization and ensures the reliability and accuracy of the information reasoning results. Furthermore, this method can be applied to other text analysis fields, possessing certain universality and promotional value. Attached Figure Description

[0043] Figure 1 A flowchart illustrating a utility test method for generative AI used in reasoning for disclosure report information, provided in an embodiment of this application;

[0044] Figure 2 A technical roadmap for a utility test method of generative AI for reasoning in disclosure report information provided in this application embodiment;

[0045] Figure 3 This is a structural diagram of a generative AI utility testing system for disclosing report information reasoning, provided in an embodiment of this application. Detailed Implementation

[0046] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0047] Example 1

[0048] like Figure 1-2 As shown, this embodiment proposes a utility test method for using generative AI to reason about disclosure report information. The method includes the following steps:

[0049] S1. Based on preset prompts, use a generative AI model to analyze N corporate disclosure reports to obtain N analysis texts. Preprocess the analysis texts to generate a document-word matrix.

[0050] S2. Use the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determine the keywords and corresponding word vectors for each topic based on the word probability distribution;

[0051] S3. Based on the word vectors of keywords in each topic, determine the topic ratio vector in each company's disclosure report;

[0052] S4. Based on the sentence vector of each sentence in the N pre-determined analysis texts, determine the topic ratio vector of the analysis text for each generative AI model;

[0053] S5. Calculate the statistics of corporate disclosure reports and analysis texts based on Pearson chi-square test, and perform paired t-test. Based on the significance level of the test results, determine whether the generative AI model has the function of information reasoning.

[0054] S6. Utilize the topic distribution vector of each enterprise's disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model.

[0055] Specifically, in step S1, preprocessing includes: performing word segmentation, word frequency statistics, stop word removal, and low-frequency word processing on the document set of enterprise disclosure reports to obtain a document-word matrix. .

[0056] Specifically, step S2 includes:

[0057] S201, Using LDA topic modeling to... Topic analysis was conducted, and the optimal number of topics k was determined using the perplexity index, resulting in k topics and the word probability distribution for each topic. ,in, Indicates the first One topic, express The total number of Chinese words, selecting the first... Each word serves as a keyword for each topic;

[0058] S202. Calculate the word vector for each keyword in each topic corresponding to the enterprise disclosure report using the word2vec model. The first topic The word vectors of each keyword are ,in, This represents the parameters of the word2vec model, including vector dimensions and window size.

[0059] Specifically, step S3 includes:

[0060] S301. Decompose each company's disclosure report into sentences to obtain a set of sentences. , in, It is the number of sentences in the document;

[0061] S302, For each sentence The sentence vectors are calculated using the BERT model, and the vectors labeled [CLS] are used as sentence vectors. ,in, Sentence The sentence vector, This indicates that the BERT model represents the sentence. The encoded output, [CLS] represents the vector corresponding to the [CLS] tag extracted from the BERT model output;

[0062] S303. Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Sentence The sentence vector, Indicates the first The first topic Word vectors of each keyword, Represents the dot product of two vectors. Let represent the norms of the two vectors respectively;

[0063] S304. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic, where Indicates the topic The total number of keywords in the text;

[0064] S305. Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the company's disclosure report. ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. This indicates that the topic is discussed in the analysis text of N corporate disclosure reports. The proportion.

[0065] Specifically, step S4 includes:

[0066] S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set. , in It is the number of sentences in the document;

[0067] S402, For each sentence The sentence vectors are calculated using the BERT model, and the vectors labeled [CLS] are used as sentence vectors. ,in, Sentence The sentence vector, This indicates that the BERT model represents the sentence. The encoded output, [CLS] represents the vector corresponding to the [CLS] tag extracted from the BERT model output;

[0068] S403, Calculate each sentence sentence vector Word vectors of each topic keyword in the corporate disclosure report text Cosine similarity between them; for each topic and keywords The cosine similarity of word vectors for each keyword in each topic. , Sentence The sentence vector, It is the first The word vector of a keyword in a topic. Represents the dot product of two vectors. Let represent the norms of the two vectors respectively;

[0069] S404. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic. The topic with the highest similarity is selected as the topic of the sentence. The ratio vector of each topic in the analyzed text of this generative AI model is calculated. ,in Indicates the first The ratio of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document. The representative discusses the topic in the analysis text of N generative AI models. The proportion.

[0070] Specifically, step S5 includes:

[0071] The chi-square statistics for each pair of corporate disclosure report documents and generative AI model analysis text documents were calculated, resulting in N pairs of chi-square statistics. Paired t-tests were performed on the N pairs of chi-square statistics. If the p-value < 0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning capabilities.

[0072] Specifically, step S6 includes:

[0073] S601. Analyze the topic distribution vector of each enterprise's disclosure report using the analysis text and the corresponding generative AI model. and ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. Indicates the first The ratio of the number of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document;

[0074] S602. Calculate the KL divergence between the analytical text of each enterprise disclosure report and the corresponding analytical text of the generative AI model. Then, the mean of the N KL divergences is calculated to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of the KL divergence, as shown in the following formula:

[0075] ;

[0076] More specifically, the level of information reasoning utility of generative AI models is positively correlated with the value of KL divergence. The larger the KL divergence, the greater the difference between the analytical text of the generative AI model and the thematic distribution of the corporate disclosure report, and the higher the information reasoning utility.

[0077] According to the embodiments of the present invention described above, this method systematically evaluates the ability of generative AI models to understand and reason about corporate disclosure reports through a series of refined steps. Specifically, this is reflected in the following aspects:

[0078] Firstly, using pre-defined prompts, a generative AI model analyzes multiple corporate disclosure reports to generate analytical text, which is then pre-processed to construct a document-word matrix. This step provides foundational data for subsequent text analysis.

[0079] Secondly, the LDA topic model is used to analyze the document-word matrix to determine the optimal number of topics and the keywords and corresponding word vectors for each topic. This step helps to reveal the core themes and key information in the company's disclosure reports.

[0080] Thirdly, based on the word vectors of keywords within each topic, a topic ratio vector is determined for each company's disclosure report. Simultaneously, based on the pre-determined sentence vectors of each sentence, a topic ratio vector for the analyzed text is also determined. This step further refines the analysis of the text content, making the evaluation of information reasoning more accurate.

[0081] Fourthly, the Pearson chi-square test and paired t-test are used to evaluate the information reasoning ability of the generative AI model. By calculating the statistics between the corporate disclosure report and the analysis text and conducting significance tests, it can be determined whether the generative AI model has effective information reasoning ability.

[0082] Fifthly, the topic distribution vectors of the enterprise disclosure report analysis text and the generative AI model analysis text are defined, and the degree of information reasoning effect of the generative AI model is determined by combining the KL divergence value. The larger the KL divergence value, the better the generative AI model performs in information reasoning.

[0083] Combination Figure 3 Based on the same inventive concept, this embodiment also proposes a utility testing system for generative AI used in disclosure report information reasoning. The system is applied to perform the above-mentioned effect testing method and includes:

[0084] The analysis module is used to analyze N corporate disclosure reports based on preset prompts using a generative AI model, resulting in N analytical texts. The analytical texts are then preprocessed to generate a document-word matrix.

[0085] The topic determination module is used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and to determine the keywords and corresponding word vectors for each topic based on the word probability distribution;

[0086] The first calculation module is used to determine the topic ratio vector of each company's disclosure report based on the word vectors of the keywords in each topic;

[0087] The second calculation module is used to determine the topic ratio vector of the analysis text of each generative AI model based on the sentence vector of each sentence in the N pre-determined analysis texts;

[0088] The effect testing module is used to calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and to perform a paired t-test. Based on the significance level of the test results, it is determined whether the generative AI model has the function of information reasoning.

[0089] The effect degree determination module is used to analyze the topic distribution vector of each enterprise disclosure report and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the effect degree of information inference of the generative AI model.

[0090] It should be noted that each module in the above utility testing system corresponds to steps S1 to S6 in implementing the above utility testing method. The instances and application scenarios implemented by multiple modules and their corresponding steps are the same, but are not limited to the content disclosed in the above embodiment 1.

[0091] In its implementation, the aforementioned utility testing system effectively addresses the problems existing in the background technology. By integrating an analysis module, a topic identification module, a first calculation module, a second calculation module, and an effect testing module, the system achieves a comprehensive and in-depth analysis of the disclosure report. The analysis module utilizes a generative AI model to quickly generate analytical text, significantly improving processing speed. The topic identification module uses an LDA topic model to accurately capture the text's topic, ensuring the accuracy of information extraction. The first and second calculation modules meticulously depict the report's topic ratio vector based on word vectors and sentence vectors, respectively, further enhancing the analysis's detail. The effect testing module scientifically verifies the AI ​​model's information reasoning ability through Pearson chi-square tests and paired t-tests, and combines this with KL divergence quantification to evaluate its utility level, thereby ensuring the reliability and accuracy of the information reasoning results. This system not only reduces labor costs but also provides powerful decision support for investors, regulatory agencies, and others, demonstrating broad application prospects and promotional value.

[0092] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the steps of any of the above-described utility verification methods are implemented.

[0093] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the utility verification methods described in the above embodiments.

[0094] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0095] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0096] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A utility testing method for generative AI used in reasoning for disclosure report information, characterized in that, The method includes the following steps: S1. Based on preset prompt words, use a generative AI model to analyze N enterprise disclosure reports to obtain N analysis texts. Preprocess the analysis texts to generate a document-word matrix. S2. Use the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determine the keywords and corresponding word vectors for each topic based on the word probability distribution; S3. Based on the word vectors of keywords in each topic, determine the topic ratio vector in each company's disclosure report; S4. Based on the sentence vector of each sentence in the N pre-determined analysis texts, determine the topic ratio vector of the analysis text for each generative AI model; S5. Calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and perform a paired t-test. Based on the significance level of the test results, determine whether the generative AI model has the function of information reasoning. The chi-square statistic for each pair of corporate disclosure report documents and the text document analyzed by the generative AI model was calculated, resulting in N pairs of chi-square statistics. A paired t-test was performed on the N pairs of chi-square statistics. If the p-value is <0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning capabilities. S6. Utilize the topic distribution vector of each enterprise's disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model. S601. Analyze the topic distribution vector of each enterprise's disclosure report using the analysis text and the corresponding generative AI model. ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. Indicates the first The ratio of the number of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document; S602. Calculate the KL divergence between the analytical text of each enterprise disclosure report and the corresponding analytical text of the generative AI model. Then, the mean of the N KL divergences is calculated to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of the KL divergence, as shown in the following formula: 。 2. The utility testing method for generative AI used in disclosure report information reasoning according to claim 1, characterized in that, In step S1, the preprocessing includes: performing word segmentation, word frequency statistics, stop word removal, and low-frequency word processing on the document set of enterprise disclosure reports to obtain a document-word matrix. .

3. The utility testing method for generative AI used in disclosure report information reasoning according to claim 1, characterized in that, Step S2 includes: S201, Using LDA topic modeling to... Topic analysis was conducted, and the optimal number of topics k was determined using the perplexity index, resulting in k topics and the word probability distribution for each topic. ,in, Indicates the first One topic, express The total number of Chinese words, selecting the first... Each word serves as a keyword for each topic; S202. Calculate the word vector for each keyword in each topic corresponding to the enterprise disclosure report using the word2vec model. The first topic The word vectors of each keyword are ,in, express The model's parameters include vector dimensions and window size.

4. The utility test method for generative AI used in disclosure report information reasoning according to claim 3, characterized in that, Step S3 includes: S301. Decompose each company's disclosure report into sentences to obtain a set of sentences. ,in, It is the number of sentences in the document; S302, For each sentence ,use The model calculates its sentence vectors using The marked vector is used as a sentence vector. ,in, Sentence The sentence vector, express Model for sentences The encoded output, Indicates from Extracted from model output The corresponding vector is marked; S303. Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Sentence The sentence vector, Indicates the first The first topic Word vectors of each keyword, Represents the dot product of two vectors. Let represent the norms of the two vectors respectively; S304. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic, where, Indicates the topic The total number of keywords in the text; S305. Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the company's disclosure report. ,in, Indicates the first The percentage of sentences corresponding to each topic out of the total number of sentences in the company's disclosure reports. This indicates that the topic is discussed in the analysis text of N corporate disclosure reports. The proportion.

5. The utility testing method for generative AI used in disclosure report information reasoning according to claim 4, characterized in that, Step S4 includes: S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set. , in It is the number of sentences in the document; S402, For each sentence ,use The model calculates its sentence vectors using The marked vector is used as a sentence vector. ,in, Sentence The sentence vector, express Model for sentences The encoded output, Indicates from Extracted from model output The corresponding vector is marked; S403, Calculate each sentence sentence vector Word vectors of each topic keyword in the corporate disclosure report text Cosine similarity between them; for each topic and keywords The cosine similarity of word vectors for each keyword in each topic. , Sentence The sentence vector, It is the first The word vector of a keyword in a topic. Represents the dot product of two vectors. Let represent the norms of the two vectors respectively; S404. Take the mean of the cosine similarity of all keywords in each topic, as shown in the following formula: The mean value is used as the semantic similarity between the sentence and the topic. The topic with the highest similarity is selected as the topic of the sentence. The ratio vector of each topic in the analyzed text of the generative AI model is calculated. ,in Indicates the first The ratio of sentences corresponding to each topic to the total number of sentences in the generative AI-analyzed text document. The representative discusses the topic in the analysis text of N generative AI models. The proportion.

6. A utility testing system for generative AI used in reasoning about disclosure report information, characterized in that, The system is used to perform the utility testing method according to any one of claims 1-5, and the system comprises: The analysis module is used to analyze N corporate disclosure reports based on preset prompt words using a generative AI model to obtain N analysis texts, and to preprocess the analysis texts to generate a document-word matrix; The topic determination module is used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and to determine the keywords and corresponding word vectors for each topic based on the word probability distribution; The first calculation module is used to determine the topic ratio vector of each company's disclosure report based on the word vectors of the keywords in each topic; The second calculation module is used to determine the topic ratio vector of the analysis text of each generative AI model based on the sentence vector of each sentence in the N pre-determined analysis texts; The effect testing module is used to calculate the statistics of the corporate disclosure report and the analysis text based on the Pearson chi-square test, and to perform pairing. The test determines whether the generative AI model has the ability to perform information reasoning based on the significance level of the test results. The effect degree determination module is used to analyze the topic distribution vector of each enterprise disclosure report and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the effect degree of information inference of the generative AI model.

Images