Utility test method and system of generative AI for disclosure report information reasoning

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a utility test method and system of a generative AI for disclosure report information reasoning, and the method comprises the steps: analyzing an enterprise disclosure report by using a generative AI model, determining an optimal topic number and keywords through an LDA topic model, respectively calculating a topic word vector and a report sentence vector in combination with word2vec and BERT models, and calculating the effectiveness of the enterprise disclosure report information reasoning. And determining a subject ratio vector of the report. The statistical magnitude of the topic ratio vectors of the two pairs of texts is calculated through Pearson's chi-square test, the significance of N statistical magnitudes is calculated through pairing t test so as to evaluate the information reasoning effect of the AI model, and the KL divergence is utilized to quantify the information reasoning effect degree. According to the method, the analysis efficiency of the disclosed report is improved, the labor cost is reduced, meanwhile, the reliability and accuracy of an information reasoning result are ensured, powerful decision support is provided for investors, supervision organizations and the like, and the method has wide application prospects and popularization value.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a method and system for verifying the effectiveness of generative AI for information reasoning in disclosure reports. Background Art

[0002] Existing technologies mainly rely on manual reading and analysis of large amounts of corporate disclosure reports, which is inefficient and susceptible to subjectivity. With the development of information technology, traditional computing models such as database query and keyword retrieval are relatively effective in processing structured data, but have limited ability to extract unstructured text and deep information. Statistical analysis methods mainly focus on quantitative data and lack in-depth analysis of text content. In recent years, machine learning models, especially natural language processing technology, have been used for text analysis, but there are still limitations in understanding the deep meaning and emotional factors of the text. Advanced generative AI models such as ChatGPT have shown great potential in the field of text analysis and processing, and can improve the efficiency of text analysis through a variety of tasks such as natural language generation, understanding and processing.

[0003] Although existing technologies have improved the analytical capabilities of corporate disclosure reports to a certain extent, there are still many defects. First, the problem of insufficient or inaccurate information disclosure still exists, resulting in the failure to fundamentally solve the phenomenon of information asymmetry. Secondly, traditional analytical methods often ignore the analysis of text content when evaluating the quality of disclosure, especially the key information and emotional factors in interim announcements. In addition, existing methods still have limitations in processing unstructured text and complex information, and cannot fully explore the information value in the text. Although generative AI models such as ChatGPT have powerful text processing capabilities, the effectiveness and accuracy of their information reasoning function still need to be further tested. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for verifying the effectiveness of generative AI for information reasoning in disclosure reports, so as to solve the problems in the prior art of low efficiency of corporate disclosure report analysis, susceptibility to subjective influences, and limited capabilities of traditional computing models and statistical analysis methods in processing unstructured texts.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions: In the first aspect, the present invention proposes a method for testing the effectiveness of generative AI for information reasoning in disclosure reports, the method comprising the following steps: S1. Based on preset prompt words, use a generative AI model to analyze N corporate disclosure reports to obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; S2, using the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determining the keywords and corresponding word vectors of each topic based on word probability distribution; S3. Determine the topic ratio vector of each enterprise disclosure report based on the word vector of the keywords in each topic; S4. Determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; S5. Calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and perform a paired t-test to determine whether the generative AI model has information reasoning capabilities based on the significance level of the test results; S6. Use the topic distribution vectors of each company’s disclosure report analysis text and the corresponding generative AI model analysis text to calculate the mean of N KL divergences and determine the degree of information reasoning effect of the generative AI model.

[0006] Furthermore, in step S1, the preprocessing includes: performing word segmentation, word frequency statistics, and removing stop words and low-frequency words on the document set of the enterprise disclosure report to obtain a document-word matrix .

[0007] Further, step S2 includes: S201, using LDA topic model Perform topic analysis and use the perplexity index to determine the optimal number of topics k, and obtain k topics and the word probability distribution corresponding to each topic ,in, Indicates Themes, express The total number of words in the selection words as keywords for each topic; S202, using the word2vec model to calculate the word vector of each keyword of each topic corresponding to the enterprise disclosure report, Theme No. The word vector of a keyword is ,in, Represents the parameters of the word2vec model, including vector dimension and window size.

[0008] Further, step S3 includes: S301. Decompose each enterprise disclosure report into sentences to obtain a sentence set , in, is the number of sentences in the document; S302. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S303: Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Expressing sentences The sentence vector of Indicates Theme The word vectors of keywords, represents the dot product of two vectors, Respectively represent the norms of two vectors; S304, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, where Indicates the subject The total number of keywords in the S305: Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the enterprise disclosure report. ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates the topic discussed in the analysis text of N corporate disclosure reports proportion.

[0009] Further, step S4 includes: S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set , in is the number of sentences in the document; S402. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S403, calculate each sentence Sentence vector The word vector of each topic keyword in the corporate disclosure report text The cosine similarity between and keywords , the cosine similarity of the word vectors of each keyword in each topic , Expressing sentences The sentence vector of It is The word vector of a keyword in a topic, represents the dot product of two vectors, Respectively represent the norms of two vectors; S404, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, selecting the topic with the highest similarity as the topic of the sentence, and calculating the ratio vector of each topic of the analysis text of the generative AI model ,in Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI, Represents a topic discussed in the analysis text of N generative AI models proportion.

[0010] Further, step S5 includes: The chi-square statistic of each pair of corporate disclosure report documents and the generative AI model analysis text documents is calculated to obtain N pairs of chi-square statistics; a paired t-test is performed on the N pairs of chi-square statistics. If the P value is <0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning function.

[0011] Further, step S6 includes: S601. Analyze the text of each enterprise disclosure report and the corresponding generative AI model to analyze the topic distribution vector of the text and ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI; S602: Calculate the KL divergence of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text , and then find the average of N KL divergences to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of KL divergence; as shown in the following formula: .

[0012] In a second aspect, the present invention proposes a system for verifying the effectiveness of information reasoning using generative AI in disclosure reports, the system being applied to execute any of the above-mentioned effect verification methods, the system comprising: An analysis module, used to analyze N corporate disclosure reports based on preset prompt words using a generative AI model to obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; A topic determination module, used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and determine the keywords and corresponding word vectors of each topic based on word probability distribution; A first calculation module is used to determine a topic ratio vector in each enterprise disclosure report based on the word vectors of the keywords in each topic; A second calculation module, used to determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; An effect test module, which is used to calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and to perform a paired t-test to determine whether the generative AI model has information reasoning function according to the significance level of the test results; The effect degree determination module is used to use the topic distribution vector of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model.

[0013] The beneficial effects of the present invention are: The present invention introduces a scientific test method to evaluate the information reasoning utility of the generative AI model. By determining the optimal number of topics through the LDA topic model, calculating word vectors and sentence vectors using the word2vec and BERT models, and combining statistical methods such as the Pearson chi-square test and KL divergence, the present invention can objectively and accurately measure the performance of the generative AI model in information reasoning. This not only provides data support for the optimization of the model, but also ensures the reliability and accuracy of the information reasoning results. In addition, this method can also be applied to other text analysis fields, and has certain universality and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart of a method for verifying the effectiveness of using generative AI for information reasoning in disclosure reports provided in an embodiment of the present application; Figure 2 A technical roadmap for the utility testing method of the generative AI provided in the embodiment of the present application for information reasoning in disclosure reports; Figure 3 A structural diagram of a utility verification system for disclosure report information reasoning using the generative AI provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0016] Example 1

[0017] like Figure 1-2 As shown, this embodiment proposes a method for testing the effectiveness of generative AI for information reasoning in disclosure reports, and the method includes the following steps: S1. Based on the preset prompt words, use the generative AI model to analyze N corporate disclosure reports, obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; S2, using the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determine the keywords and corresponding word vectors of each topic based on the word probability distribution; S3. Determine the topic ratio vector of each enterprise disclosure report based on the word vector of the keywords in each topic; S4. Determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; S5. Calculate the statistics of the enterprise disclosure report and analysis text based on the Pearson chi-square test, and conduct a paired t-test. According to the significance level of the test results, determine whether the generative AI model has the function of information reasoning; S6. Use the topic distribution vectors of each company’s disclosure report analysis text and the corresponding generative AI model analysis text to calculate the mean of N KL divergences and determine the degree of information reasoning effect of the generative AI model.

[0018] Specifically, in step S1, the preprocessing includes: performing word segmentation, word frequency statistics, and removing stop words and low-frequency words on the document set of the enterprise disclosure report to obtain a document-word matrix .

[0019] Specifically, step S2 includes: S201, using LDA topic model Perform topic analysis and use the perplexity index to determine the optimal number of topics k, and obtain k topics and the word probability distribution corresponding to each topic ,in, Indicates Themes, express The total number of words in the selection words as keywords for each topic; S202, using the word2vec model to calculate the word vector of each keyword of each topic corresponding to the enterprise disclosure report, Theme No. The word vector of a keyword is ,in, Represents the parameters of the word2vec model, including vector dimension and window size.

[0020] Specifically, step S3 includes: S301. Decompose each enterprise disclosure report into sentences to obtain a sentence set , in, is the number of sentences in the document; S302. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S303: Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Expressing sentences The sentence vector of Indicates Theme The word vectors of keywords, represents the dot product of two vectors, Respectively represent the norms of two vectors; S304, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, where Indicates the subject The total number of keywords in the S305: Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the enterprise disclosure report. ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates the topic discussed in the analysis text of N corporate disclosure reports proportion.

[0021] Specifically, step S4 includes: S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set , in is the number of sentences in the document; S402. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S403, calculate each sentence Sentence vector The word vector of each topic keyword in the corporate disclosure report text The cosine similarity between and keywords , the cosine similarity of the word vectors of each keyword in each topic , Expressing sentences The sentence vector of It is The word vector of a keyword in a topic, represents the dot product of two vectors, Respectively represent the norms of two vectors; S404, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, selecting the topic with the highest similarity as the topic of the sentence, and calculating the ratio vector of each topic of the analysis text of the generative AI model ,in Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI, Represents a topic discussed in the analysis text of N generative AI models proportion.

[0022] Specifically, step S5 includes: The chi-square statistic of each pair of corporate disclosure report documents and generative AI model analysis text documents was calculated to obtain N pairs of chi-square statistics. A paired t-test was performed on the N pairs of chi-square statistics. If the P value is <0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning function.

[0023] Specifically, step S6 includes: S601. Analyze the text of each enterprise disclosure report and the corresponding generative AI model to analyze the topic distribution vector of the text and ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI; S602: Calculate the KL divergence of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text , and then find the average of N KL divergences to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of KL divergence; as shown in the following formula: ; More specifically, the level of information reasoning utility of the generative AI model is positively correlated with the value of KL divergence. The larger the KL divergence, the greater the difference in the subject distribution between the analysis text of the generative AI model and the corporate disclosure report, and the higher the information reasoning utility.

[0024] According to the above embodiment of the present invention, the method systematically evaluates the ability of the generative AI model in understanding and reasoning about the information disclosed in the enterprise report through a series of detailed steps. Specifically, it is reflected in the following aspects: First, using preset prompt words, the generative AI model analyzes multiple corporate disclosure reports, generates analysis text, and performs preprocessing to build a document-word matrix. This step provides basic data for subsequent text analysis.

[0025] Secondly, the LDA topic model is used to analyze the document-word matrix to determine the optimal number of topics and the keywords and corresponding word vectors for each topic. This step helps to reveal the core topics and key information in the company's disclosure report.

[0026] Third, based on the word vectors of the keywords in each topic, the topic ratio vectors in each company's disclosure report are determined. At the same time, based on the pre-determined sentence vectors of each sentence, the topic ratio vectors of the analyzed text are also determined. This step further refines the analysis of the text content and makes the evaluation of information reasoning more accurate.

[0027] Fourthly, the Pearson chi-square test and paired t-test are used to evaluate the information reasoning function of the generative AI model. By calculating the statistics between the corporate disclosure report and the analysis text and conducting a significance test, it can be determined whether the generative AI model has effective information reasoning capabilities.

[0028] The fifth aspect is to define the topic distribution vectors of the analysis text of the corporate disclosure report and the analysis text of the generative AI model, and combine the value of KL divergence to determine the degree of information reasoning effect of the generative AI model. The larger the value of KL divergence, the better the performance of the generative AI model in information reasoning.

[0029] Combination Figure 3 Based on the same inventive concept, this embodiment also proposes a system for verifying the effectiveness of information reasoning using generative AI in disclosure reports. The system is used to execute the above-mentioned effect verification method, and the system includes: An analysis module is used to analyze N corporate disclosure reports based on preset prompt words using a generative AI model to obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; The topic determination module is used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and determine the keywords and corresponding word vectors of each topic based on the word probability distribution; A first calculation module is used to determine a topic ratio vector in each enterprise disclosure report based on the word vectors of the keywords in each topic; A second calculation module, used to determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; An effect test module, which is used to calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and to perform a paired t-test to determine whether the generative AI model has information reasoning function according to the significance level of the test results; The effect degree determination module is used to use the topic distribution vector of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model.

[0030] It should be noted here that each module in the above-mentioned utility verification system corresponds to steps S1 to S6 in implementing the above-mentioned utility verification method, and the instances and application scenarios implemented by multiple modules are the same as those of the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0031] In the specific implementation, the above utility verification system effectively solves the problems existing in the background technology. The system realizes a comprehensive and in-depth analysis of the disclosure report by integrating the analysis module, the theme determination module, the first calculation module, the second calculation module and the effect verification module. The analysis module uses the generative AI model to quickly generate analysis text, which greatly improves the processing speed; the theme determination module uses the LDA theme model to accurately capture the text theme, ensuring the accuracy of information extraction; the first and second calculation modules are based on word vectors and sentence vectors, respectively, to carefully depict the theme ratio vector of the report, further enhancing the meticulousness of the analysis. The effect verification module scientifically verifies the information reasoning ability of the AI ​​model through the Pearson chi-square test and paired t-test, and quantitatively evaluates its utility level in combination with the KL divergence, thereby ensuring the reliability and accuracy of the information reasoning results. The system not only reduces labor costs, but also provides strong decision-making support for investors, regulators, etc., showing broad application prospects and promotion value.

[0032] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above utility verification methods are implemented.

[0033] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute the steps of any utility verification method in the above embodiments.

[0034] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0035] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0036] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for testing the effectiveness of generative AI for information reasoning in disclosure reports, characterized in that: The method comprises the following steps: S1. Based on preset prompt words, use a generative AI model to analyze N corporate disclosure reports to obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; S2, using the LDA topic model to determine the optimal number of topics k in the document-word matrix, and determining the keywords and corresponding word vectors of each topic based on word probability distribution; S3. Determine the topic ratio vector of each enterprise disclosure report based on the word vector of the keywords in each topic; S4. Determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; S5. Calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and perform a paired t-test to determine whether the generative AI model has information reasoning capabilities based on the significance level of the test results; S6. Use the topic distribution vectors of each company’s disclosure report analysis text and the corresponding generative AI model analysis text to calculate the mean of N KL divergences and determine the degree of information reasoning effect of the generative AI model.

2. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 1 is characterized in that: In step S1, the preprocessing includes: performing word segmentation, word frequency statistics, and removing stop words and low-frequency words on the document set of the enterprise disclosure report to obtain a document-word matrix .

3. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 1 is characterized in that: Step S2 includes: S201, using LDA topic model Perform topic analysis and use the perplexity index to determine the optimal number of topics k, and obtain k topics and the word probability distribution corresponding to each topic ,in, Indicates Themes, express The total number of words in the selection words as keywords for each topic; S202, using the word2vec model to calculate the word vector of each keyword of each topic corresponding to the enterprise disclosure report, Theme No. The word vector of a keyword is ,in, Represents the parameters of the word2vec model, including vector dimension and window size.

4. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 3 is characterized in that: Step S3 includes: S301. Decompose each enterprise disclosure report into sentences to obtain a sentence set , in, is the number of sentences in the document; S302. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S303: Calculate the cosine similarity between the sentence vector of each sentence and the word vector of each keyword in each topic. ,in, Expressing sentences The sentence vector of Indicates Theme The word vectors of keywords, represents the dot product of two vectors, Respectively represent the norms of two vectors; S304, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, where Indicates the subject The total number of keywords in the S305: Select the topic with the highest similarity as the topic of the sentence, and calculate the ratio vector of each topic in the enterprise disclosure report. ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates the topic discussed in the analysis text of N corporate disclosure reports proportion.

5. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 4 is characterized in that: Step S4 includes: S401. Decompose the analysis text of each generative AI model into sentences to obtain a sentence set , in is the number of sentences in the document; S402. For each sentence , use the BERT model to calculate its sentence vector, and use the vector of the [CLS] tag as the sentence vector ,in, Expressing sentences The sentence vector of Represents the BERT model for the sentence The encoded output of [CLS] represents the vector corresponding to the [CLS] token extracted from the BERT model output; S403, calculate each sentence Sentence vector The word vector of each topic keyword in the corporate disclosure report text The cosine similarity between and keywords , the cosine similarity of the word vectors of each keyword in each topic , Expressing sentences The sentence vector of It is The word vector of a keyword in a topic, represents the dot product of two vectors, Respectively represent the norms of two vectors; S404, taking the mean of the cosine similarities of all keywords in each topic, as shown in the following formula: , taking the mean as the semantic similarity between the sentence and the topic, selecting the topic with the highest similarity as the topic of the sentence, and calculating the ratio vector of each topic of the analysis text of the generative AI model ,in Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI, Represents a topic discussed in the analysis text of N generative AI models proportion.

6. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 1 is characterized in that: Step S5 includes: The chi-square statistic of each pair of corporate disclosure report documents and the generative AI model analysis text documents is calculated to obtain N pairs of chi-square statistics; a paired t-test is performed on the N pairs of chi-square statistics. If the P value is <0.05, the difference is significant at the 5% level, indicating that the generative AI model has information reasoning function.

7. The utility test method of using generative AI for information reasoning in disclosure reports according to claim 5 is characterized in that: Step S6 includes: S601. Analyze the text of each enterprise disclosure report and the corresponding generative AI model to analyze the topic distribution vector of the text and ,in, Indicates The ratio of the number of sentences corresponding to the topic to the total number of sentences in the company's disclosure report, Indicates The ratio of the number of sentences corresponding to each topic to the total number of sentences in the text document analyzed by the generative AI; S602: Calculate the KL divergence of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text , and then find the average of N KL divergences to determine the degree of information reasoning effect of the generative AI model; among them, the level of information reasoning utility of the generative AI model is positively correlated with the value of KL divergence; as shown in the following formula: 。 8. A utility verification system for generative AI for information reasoning in disclosure reports, characterized in that: The system is used to perform the effect inspection method according to any one of claims 1 to 7, and the system comprises: An analysis module, used to analyze N corporate disclosure reports based on preset prompt words using a generative AI model to obtain N analysis texts, pre-process the analysis texts, and generate a document-word matrix; A topic determination module, used to determine the optimal number of topics k in the document-word matrix using the LDA topic model, and determine the keywords and corresponding word vectors of each topic based on word probability distribution; A first calculation module is used to determine a topic ratio vector in each enterprise disclosure report based on the word vectors of the keywords in each topic; A second calculation module, used to determine a topic ratio vector of the analysis text of each generative AI model according to the predetermined sentence vector of each sentence in the N analysis texts; An effect test module, which is used to calculate the statistics of the enterprise disclosure report and the analysis text based on the Pearson chi-square test, and to perform a paired t-test to determine whether the generative AI model has information reasoning function according to the significance level of the test results; The effect degree determination module is used to use the topic distribution vector of each enterprise disclosure report analysis text and the corresponding generative AI model analysis text, calculate the mean of N KL divergences, and determine the degree of information reasoning effect of the generative AI model.

Images