Test method and device for investment research large model, computing equipment and storage medium

Abstract

The invention provides a testing method and device for a large investment research model, computing equipment and a storage medium. The method comprises the following steps: generating a plurality of pairs consisting of expected questions and expected answer results based on an investment research corpus; based on the multiple pairs, the recall ability of the investment and research big model is determined, and the recall ability represents the accuracy of an answer result returned by the investment and research big model for the expected question relative to the expected answer result; inputting a query related to at least one theme in the plurality of different themes into the investment research large model to obtain a plurality of generated answer results; on the basis of the multiple generated answer results, the generation capacity of the large investment research model is determined, the generation capacity depends on the capacity on multiple generation dimensions, and the multiple generation dimensions at least comprise the data accuracy and the data logicality of the generated answer results; and based on the recall capability of the investment research large-scale model and the generation capability of the investment research large-scale model, obtaining a test result of the investment research large-scale model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to testing methods and apparatus for large-scale investment research models, computing devices, and computer storage media. Background Technology

[0002] With the rapid development of internet technology, the securities industry is increasingly using various large language models (LLMs) to assist in completing various tasks, such as investment research. Due to the complexity, timeliness, high compliance, and accuracy requirements of its business, the securities industry imposes stringent standards on the application quality of large language models (LLMs). However, large language models suffer from overgeneralization and data bias, easily leading to "illusion" phenomena and unreliable output information, posing significant risks to their application in securities investment research scenarios. Therefore, before deploying a large model, it must be rigorously tested to ensure the accuracy and reliability of the output information and mitigate potential risks.

[0003] Currently, testing large language models in the investment research field faces unique challenges. For example, evaluating the semantics of long texts in actual test designs within the investment research industry is difficult, and there is a lack of timely reference answers as test benchmarks. Traditional testing methods based on labeled data are ill-suited to the rapidly changing information environment in investment research scenarios, resulting in low testing efficiency and accuracy for large language models in the investment research field. Summary of the Invention

[0004] In view of this, this disclosure provides testing methods and apparatus, computing devices and computer storage media for large-scale investment research models, in order to overcome some or all of the defects mentioned above and other possible defects.

[0005] According to a first aspect of this disclosure, a testing method for a large-scale investment research model is provided, comprising: generating multiple pairs consisting of expected questions and expected answer results based on an investment research corpus related to the investment research field, wherein the investment research corpus includes corpus segments related to multiple different topics; determining the recall capability of the large-scale investment research model based on the multiple pairs, wherein the recall capability characterizes the accuracy of the answer results returned by the large-scale investment research model for the expected questions relative to the expected answer results; inputting a query related to at least one of the multiple different topics into the large-scale investment research model to obtain multiple generated answer results; determining the generation capability of the large-scale investment research model based on the multiple generated answer results, wherein the generation capability depends on capabilities in multiple generation dimensions, wherein the multiple generation dimensions include at least the data accuracy and data logicality of the generated answer results, wherein the data accuracy characterizes the accuracy of the multiple generated answer results, and the data logicality characterizes the logical correctness of the multiple generated answer results; and obtaining the test results of the large-scale investment research model based on the recall capability and the generation capability of the large-scale investment research model.

[0006] In some embodiments, the answer result includes the answer to the question and the corresponding citation, wherein the citation indicates the source of the corresponding answer. Based on an investment research corpus related to the investment research field, multiple pairs consisting of expected questions and expected answer results are generated, including: selecting a target topic, wherein the target topic is one of multiple different topics; selecting multiple core corpus segments related to the target topic from the investment research corpus; and generating multiple pairs consisting of expected questions and expected answer results related to the target topic based on the multiple core corpus segments, wherein the expected citation corresponding to the expected answer in the expected answer result indicates the core corpus segment from which the expected answer originates.

[0007] In some embodiments, determining the recall capability of the investment research big data model based on the plurality of pairs includes: inputting expected questions related to the target topic from the plurality of pairs into the investment research big data model to obtain a plurality of true first answer results, each true first answer result including a first true answer and a first citation corresponding to the first true answer; determining the recall capability of the investment research big data model based on the expected answer results corresponding to the expected questions related to the target topic and the plurality of true first answer results, wherein the recall capability depends on the accuracy of the citations in the first answer results relative to the citations in the expected answer results.

[0008] In some embodiments, selecting multiple core corpus segments related to the target topic from the investment research corpus includes: selecting one core corpus segment related to the target topic from the investment research corpus; and, based on the selected core corpus segment, selecting corpus segments from the investment research corpus whose semantic similarity to the selected core corpus segment is greater than a similarity threshold as other core corpus segments among the multiple core corpus segments.

[0009] In some embodiments, a query related to at least one of the plurality of different topics is input into an investment research big data model to obtain a plurality of generated answer results, including: obtaining a query related to the at least one topic; using the investment research big data model to obtain a plurality of generated answer results for the query, each generated answer result including at least one second true answer and a second reference corresponding to each second true answer, the second reference indicating the source of the corresponding second true answer.

[0010] In some embodiments, obtaining multiple generated answer results for the query using the investment research big data model includes: determining a target topic; extracting a target topic query related to the target topic from the query based on the target topic; and inputting the target topic query into the investment research big data model to obtain multiple generated answer results for the query.

[0011] In some embodiments, determining the generation capability of the investment research big data model based on the plurality of generated answer results includes: for each generated answer result, obtaining the content data from which the second true answer corresponding to the second reference is obtained based on the second reference included in each generated answer result; determining the content summary data related to the target topic query based on the content data, the content summary data including data obtained by summarizing the content data; determining the generation capability of the investment research big data model based on the content summary data and each second true answer, wherein the data accuracy of the answer results indicates the degree of accuracy of each second true answer compared to the content summary data, and the data logic of the answer results indicates the degree of logical correctness of each second true answer compared to the content summary data.

[0012] In some embodiments, the plurality of generation dimensions further include one or more of data integrity, data timeliness, and data readability of the generated answer results, wherein data integrity indicates the completeness of each second true answer relative to the content summary data; data timeliness indicates the temporal consistency of each second true answer relative to the content summary data; and data readability indicates the consistency of the representational structure of each second true answer relative to the content summary data.

[0013] In some embodiments, the plurality of generation dimensions further include the data robustness of the generated response results, wherein data robustness indicates the degree of consistency of the generated response results when multiple response results are generated for the same query.

[0014] In some embodiments, determining the generation capability of the investment research big model based on the multiple generated answer results includes: for each of the multiple generation dimensions, using multiple judgment models to determine multiple scores of the generated answer results on each generation dimension, wherein the multiple judgment models have different weights; and determining the generation capability of the investment research big model on each generation dimension based on the weights of the multiple judgment models and the multiple scores.

[0015] In some embodiments, the recall capability of the investment research model is represented by a recall capability score, and the generation capability of the investment research model is represented by a generation capability score. The test result of the investment research model is obtained based on its recall capability and generation capability, including: determining a weighted sum of the recall capability score and the generation capability score; and determining the weighted sum as the test result of the investment research model.

[0016] According to a second aspect of this disclosure, a testing apparatus for a large-scale investment research model is provided, comprising: an expected result generation module configured to generate multiple pairs consisting of expected questions and expected answer results based on an investment research corpus related to the investment research field, wherein the investment research corpus includes corpus segments related to multiple different topics; a recall capability determination module configured to determine the recall capability of the large-scale investment research model based on the multiple pairs, wherein the recall capability characterizes the accuracy of the answer results returned by the large-scale investment research model for expected questions relative to expected answer results; and an answer result generation module configured to input a query related to at least one of the multiple different topics into the investment research corpus. In the large model, multiple generated answer results are obtained; the generation capability determination module is configured to determine the generation capability of the investment research large model based on the multiple generated answer results, wherein the generation capability depends on the capability in multiple generation dimensions, the multiple generation dimensions including at least the data accuracy and data logic of the generated answer results, the data accuracy characterizing the accuracy of the multiple generated answer results, and the data logic characterizing the logical correctness of the multiple generated answer results; the test result determination module is configured to obtain the test result of the investment research large model based on the recall capability and the generation capability of the investment research large model.

[0017] According to a third aspect of this disclosure, a computing device is provided, including a processor; and a memory configured to store computer-executable instructions thereon, which, when executed by the processor, perform any of the methods described above.

[0018] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed, perform any of the methods described above.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, the computer program product including computer-executable instructions that, when executed, implement any of the methods described above.

[0020] In the testing method and apparatus for the large-scale investment research model claimed in this disclosure, multiple pairs consisting of expected questions and expected answers are generated based on an investment research corpus related to the investment research field. The recall capability of the large-scale investment research model is determined based on these pairs, thus providing a highly timely reference answer as a testing benchmark. Furthermore, by determining the generative capability of the large-scale investment research model across multiple generative dimensions, the semantics of long texts designed for actual testing in the investment research industry can be effectively evaluated. By determining the recall and generative capabilities of the large-scale investment research model, the application capability of the model in the investment research field can be comprehensively determined, improving the testing efficiency and accuracy of large-scale language models in the investment research field.

[0021] These and other advantages of this disclosure will become clear from the embodiments described below, and will be illustrated with reference to the embodiments described below. Attached Figure Description

[0022] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings, in which:

[0023] Figure 1 Exemplary application scenarios in which the technical solutions according to embodiments of this disclosure can be implemented are illustrated;

[0024] Figure 2 The illustration shows a schematic flowchart of a testing method for a large-scale investment research model according to an embodiment of the present disclosure;

[0025] Figure 3 The illustration shows an example of obtaining generated answer results using a large-scale investment research model;

[0026] Figure 4 The diagram illustrates how three judgment models are used to determine the generation capability of a large model. Figure 5 The figure shows an exemplary structural block diagram of a testing apparatus for a large-scale investment research model according to an embodiment of the present disclosure; Figure 6 An example system is illustrated, which includes an example computing device representing one or more systems and / or devices that can implement the various technologies described herein. Detailed Implementation

[0027] The following description provides specific details of various embodiments of this disclosure to enable those skilled in the art to fully understand and implement the various embodiments of this disclosure. It should be understood that the technical solutions of this disclosure can be implemented without some of these details. In some cases, this disclosure does not show or describe in detail some well-known structures or functions to avoid such unnecessary descriptions that would obscure the description of the embodiments of this disclosure. The terminology used in this disclosure should be understood in its broadest and most reasonable manner, even when used in conjunction with specific embodiments of this disclosure.

[0028] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0029] Investment Research Corpus: The investment research corpus occupies a crucial position in the investment research process within the financial field. Its sources are broad and diverse, including corpus segments related to multiple different themes. For example, these corpus segments may originate from news released by domestic and international financial media and information platforms, reflecting timely market dynamics and industry trends; from listed companies' financial reports showcasing their financial status and operating results, such as balance sheets and profit statements; from industry research reports written by professional institutions to analyze the current state of industry development, competitive landscape, and future trends; from macroeconomic data and other indicators reflecting the macroeconomic environment; and so on.

[0030] Figure 1 The illustration shows an exemplary application scenario 100 in which the technical solutions according to embodiments of this disclosure can be implemented. For example... Figure 1 As shown, the application scenario includes a terminal 110 and a server 120, with the terminal 110 and the server 120 being communicatively coupled together via a network 130.

[0031] As an example, the terminal 110 may send a test request for a large-scale investment research model to the server 120, wherein the test request may specify the large-scale investment research model to be tested. The large-scale investment research model is a large language model developed specifically for the field of investment research based on deep learning technology. By learning from massive amounts of financial data, information texts, research reports, etc., it has the ability to understand, analyze, and generate relevant content, providing strong support for investment decisions. After receiving a test request, server 120 can generate multiple pairs consisting of expected questions and expected answers based on an investment research corpus related to the investment research field. The investment research corpus includes corpus segments related to multiple different topics. Based on these multiple pairs, the recall capability of the investment research model is determined, whereby the recall capability characterizes the accuracy of the answers returned by the investment research model for the expected questions relative to the expected answers. A query related to at least one of the multiple different topics is input into the investment research model to obtain multiple generated answer results. Based on the multiple generated answer results, the generation capability of the investment research model is determined, where the generation capability depends on capabilities in multiple generation dimensions. These multiple generation dimensions include at least the data accuracy and data logic of the generated answer results. The data accuracy characterizes the accuracy of the multiple generated answer results, and the data logic characterizes the logical correctness of the multiple generated answer results. Based on the recall capability and the generation capability of the investment research model, the test result of the investment research model is obtained. Finally, server 120 can return the test result to client 110 for presentation.

[0032] The scenario described above is merely one example in which embodiments of this disclosure can be implemented and is not restrictive. For example, in some exemplary scenarios, the task test scheduling method described above may also be implemented independently on terminal 110 or server 120.

[0033] It should be noted that terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations. The network 130 can be, for example, a wide area network (WAN), a local area network (LAN), a wireless network, a public telephone network, an intranet, and any other type of network well known to those skilled in the art.

[0034] Figure 2 The illustration shows a schematic flowchart of a testing method 200 for a large-scale investment research model according to an embodiment of the present disclosure. The testing method for the large-scale investment research model can, for example, be... Figure 1 The terminal 110 or server 120 shown is used for implementation. The large-scale investment research model here is a pre-trained large language model. For example... Figure 2 As shown, the method 200 includes the following steps.

[0035] In step 210, based on an investment research corpus related to the investment research field, multiple pairs consisting of expected questions and expected answers are generated. The investment research corpus includes corpus segments related to multiple different topics. A corpus segment is a relatively independent and complete content fragment within the investment research corpus. The multiple topics mentioned here may include content related to stock analysis, industry research, macroeconomics, company financial reports, etc. As an example, the topic could be "the performance of the liquor industry." The question here can be a question related to one of the multiple topics, such as "What will the performance of the liquor industry be like in 2025?" The expected question refers to a pre-set or generated question, which can be obtained, for example, based on relevant corpus segments in the investment research corpus.

[0036] The response results include the answer to the question and the corresponding citation, whereby the citation indicates the source of the answer. Correspondingly, the expected response results include the answer to the expected question and the corresponding citation. The citation corresponding to the answer to the expected question is typically represented as an identifier of a corpus segment in an investment research corpus from which the answer to the expected question originates. This identifier can be various symbols, such as the title, address, or representative number of the article from which the corpus segment originates. In some embodiments, when generating multiple pairs of expected questions and expected response results based on an investment research corpus related to the investment research field, a target topic can be selected first, which is one of the multiple different topics. Then, multiple core corpus segments related to the target topic are selected from the investment research corpus. The core corpus segments are at least a portion of the corpus segments related to the target topic selected from the investment research corpus. Finally, based on the multiple core corpus segments, multiple pairs of expected questions and expected response results related to the target topic are generated, where the expected citation corresponding to the expected answer in the expected response indicates the core corpus segment from which the expected answer originates.

[0037] As an example, tools such as RAGAS and LlamaIndex can be combined to generate multiple pairs of expected questions and expected responses related to the target topic. It should be noted that when the corpus selected from the investment research corpus includes charts and graphs, these can be converted into text form (e.g., extracting the text) before using these tools to generate multiple pairs. These generated pairs can then be used as reference answers for subsequent recall determination. Reference answers typically refer to highly authoritative reference data considered "true" or "correct," used to verify the accuracy of model predictions, algorithm outputs, or system results.

[0038] In some embodiments, when selecting multiple core corpus segments related to a target topic from the investment research corpus, one core corpus segment related to the target topic can be selected first from the investment research corpus, for example, a core corpus segment related to the target topic can be randomly selected from the investment research corpus; then, based on the selected core corpus segment, corpus segments from the investment research corpus with a semantic similarity greater than a similarity threshold are selected as other core corpus segments among the multiple core corpus segments. As an example, the core corpus segments can be vectorized to facilitate the calculation of semantic similarity. As an example, ROUGE technology can be used to select corpus segments from the investment research corpus with a semantic similarity greater than a similarity threshold as other core corpus segments among the multiple core corpus segments. ROUGE technology is an indicator system used to evaluate the quality of text summaries (such as automatically generated summaries). It mainly measures the similarity between candidate summaries and reference summaries by calculating the overlapping units (such as words, n-grams, sentences, etc.).

[0039] In step 220, based on the plurality of pairs, the recall capability of the investment research big model is determined, wherein the recall capability characterizes the accuracy of the response returned by the investment research big model for the expected question relative to the expected response.

[0040] In some embodiments, expected questions related to the target topic from the plurality of pairs can be input into the investment research big data model to obtain multiple true first answer results. Each true first answer result includes a first true answer and a first citation corresponding to the first true answer. The multiple true first answer results are output by the investment research big data model and are not directly taken from the investment research corpus. Accordingly, the first citation indicates the source of the corresponding first true answer. Then, based on the expected answer results corresponding to the expected questions related to the target topic and the multiple true first answer results, the recall capability of the investment research big data model is determined. The recall capability depends on the accuracy of the citation in the first answer result relative to the citation in the expected answer result. Here, the citation in the first answer result is compared with the citation in the expected answer result. If the citation in the first answer result and the citation in the expected answer result are the same, it indicates that the recalled first answer result is a correct result. As mentioned above, the expected answer result is generated based on core corpus segments in the investment research corpus.

[0041] As an example, the recall capability of the investment research model can be determined by utilizing the mean reciprocal rank. Mean reciprocal rank is a metric used to evaluate the ranking quality of the results output by an investment research model. Its core idea is to measure the position of the correct result (i.e., the expected answer) in the list of answers retrieved by the investment research model (the list of first-response results), emphasizing the order and user visibility of the results; the higher the correct result appears, the higher the algorithm score. The formula for mean reciprocal rank (MRR) can be expressed as:

[0042]

[0043] Where N represents the total number of queries or recalls. For the i-th query, p i This refers to the position of the correct result in the list of answers retrieved for each query. For each query, the position p of the correct result is found. i And calculate its reciprocal 1 / p i If the correct result does not appear in the list, then p i If the value is infinity, then 1 / p i =0. 1 / p for all queries i The average value is taken to obtain the final MRR value. If the correct result for a query appears as the second result in the answer list, then 1 / p i =1 / 2=0.5. The higher the average MRR across multiple queries, the better the recall capability of the aforementioned investment research model.

[0044] In step 230, a query related to at least one of the multiple different topics is input into the investment research big data model to obtain multiple generated answer results. The query is a user-designed or defined query related to at least one of the multiple different topics. For example, the query could be a query related to the topic "Performance of the Baijiu Industry" such as "How was the performance of the Baijiu industry in 2025?"

[0045] In some embodiments, a query related to the at least one topic may be obtained first, and then multiple generated answer results for the query may be obtained using the investment research big data model. Each generated answer result includes at least one second true answer and a second reference corresponding to each second true answer, wherein the second reference indicates the source of the corresponding second true answer. Figure 3 The illustration shows an example of obtaining generated response results using the aforementioned investment research model. For example... Figure 3 As shown, the query 310 obtained that is related to the at least one topic is "What is the development trend of Tesla this year?", and the generated answer obtained using the investment research big data model is as follows. Figure 3 As shown in 330. Each item or sentence in the answer result 330 can be regarded as a second true answer, and each second true answer has a corresponding second reference, as shown in the data [1], [2], [3] etc.

[0046] In some embodiments, when obtaining multiple generated answer results for a query using the investment research big data model, a target topic can be determined first. Then, based on the target topic, target topic queries related to the target topic can be extracted from the query. For example, when querying "What is Tesla's development trend this year?", the target topic can be determined as "Tesla," rather than a meaningless word like "what is." Then, based on the target topic, target topic queries 320 related to the target topic can be extracted from the query, namely "Tesla's development trend in 2025." This allows the core topic targeted by the query to be obtained, facilitating the generation of invalid answer information during the response. Finally, the target topic query is input into the investment research big data model to obtain multiple generated answer results for the query. For example, various suitable large language models (e.g., various suitable agents) can be used to extract target topic queries related to the target topic.

[0047] In step 240, based on the multiple generated response results, the generation capability of the investment research big data model is determined. This generation capability depends on capabilities across multiple generation dimensions, which include at least the data accuracy and data logic of the generated response results. Data accuracy characterizes the degree of accuracy of the multiple generated response results, and data logic characterizes the logical correctness of the multiple generated response results. In some embodiments, the multiple generation dimensions also include the data robustness of the generated response results. Data robustness indicates the consistency of the generated response results when multiple response results are generated for the same query. Higher consistency in the generated response results when multiple response results are generated for the same query indicates higher data robustness.

[0048] In some embodiments, determining the generation capability of the investment research big data model based on the plurality of generated answer results includes: for each generated answer result, obtaining the content data from which the second true answer corresponding to the second reference is obtained based on the second reference included in each generated answer result; determining the content summary data related to the target topic query based on the content data, the content summary data including data obtained by summarizing the content data; determining the generation capability of the investment research big data model based on the content summary data and each second true answer, wherein the data accuracy of the answer results indicates the degree of accuracy of each second true answer compared to the content summary data, and the data logic of the answer results indicates the degree of logical correctness of each second true answer compared to the content summary data.

[0049] like Figure 3 As shown, each generated answer result is as follows: Figure 3 As shown in Figure 330. Each generated answer includes multiple second true answers and a second reference corresponding to each second true answer, whereby the second reference indicates the source of the second true answer. For each generated answer, the content data from which the second true answer corresponding to the second reference originates can be obtained based on the second reference included in each generated answer. For example... Figure 3 In this process, the research report from which the corresponding second true answer originates can be obtained based on each second reference. Then, based on the content data (i.e., the research report), summary content data related to the target topic query is determined. This summary content data includes data obtained by summarizing the content data. For example... Figure 3As shown, other trained large language models (e.g., various suitable agents) can be used in conjunction with target topic queries to summarize each research report to determine the corresponding content summary data, which is typically a full summary of the research report around the target topic. Then, based on the content summary data and each second true answer, the generation capability of the investment research model is determined. The data accuracy of the response indicates the accuracy of each second true answer relative to the content summary data, and the data logicality of the response indicates the logical correctness of each second true answer relative to the content summary data. The generation capability of the investment research model can be determined by comparing each second true answer with the content summary data of the corresponding research report. As an example, when determining generation capability, multiple generated response results obtained by inputting multiple different queries related to multiple different topics into the investment research model can be used to determine the generation capability.

[0050] In some embodiments, the plurality of generation dimensions further include one or more of data integrity, data timeliness, and data readability of the generated answer results, wherein data integrity indicates the completeness of each second true answer relative to the content summary data; data timeliness indicates the temporal consistency of each second true answer relative to the content summary data; and data readability indicates the consistency of the representational structure of each second true answer relative to the content summary data.

[0051] In some embodiments, when determining the generation capability of the investment research big model based on the multiple generated answer results, multiple judgment models can be used to determine multiple scores of the generated answer results on each generation dimension, with the multiple judgment models having different weights. As an example, various large language models (e.g., various suitable agents) can be used to determine the scores on each generation dimension, and each of the multiple judgment models can correspond to a different large language model or agent. Then, based on the weights of the multiple judgment models and the multiple scores, the capability of the investment research big model on each generation dimension is determined. The weights here can be pre-specified. As an example, the capability of the investment research big model on each generation dimension can be represented by scores, or directly by 1 (good) and 0 (bad); neither is limited.

[0052] As an example, a voting agent based on soft voting can be used to determine the capabilities of the large-scale investment research model in each generation dimension. Commonly used soft voting methods include, for example, logistic regression, k-nearest neighbor, decision tree analysis, random forest, and some ensemble learning algorithms (e.g., Adaboost and GBDT algorithms based on boosting algorithms).

[0053] As an example, Figure 4 The diagram illustrates how three judgment models are used to determine the generative capacity of a large model. For example... Figure 4 As shown, judgment model 1, judgment model 2, and judgment model 3 can be used to determine three scores for the generated answer results on the data logicality dimension. The three judgment models can have different weights. Then, based on the weights of the three judgment models and the three scores obtained, the capability of the investment research big model on the data logicality dimension can be determined. As an example, the score determined by the judgment model with the highest weight on the data logicality dimension can be directly used as the score reflecting the capability of the investment research big model on the data logicality dimension, or the weighted sum of the scores obtained by the three judgment models can be used as the score reflecting the capability of the investment research big model on the data logicality dimension; neither is restrictive. As an example, for each generation dimension, namely data logicality, data accuracy, data robustness, etc., multiple judgment models are used to determine multiple scores for the generated answer results on each generation dimension.

[0054] In step 250, based on the recall capability and generation capability of the investment research model, the test result of the investment research model is obtained. The test result can comprehensively reflect the recall capability and generation capability of the investment research model. In some embodiments, the recall capability of the investment research model is represented by a recall capability score, and the generation capability of the investment research model is represented by a generation capability score. In this case, obtaining the test result of the investment research model based on its recall capability and generation capability includes: determining the weighted sum of the recall capability score and the generation capability score; and determining the weighted sum as the test result of the investment research model. Here, the weights of the recall capability score and the generation capability score can also be pre-specified.

[0055] In some embodiments, the multiple generation dimensions also include the security of the generated response results. When determining the security of the response results generated by the investment research model, a pre-defined list of restrictions can be established. This list specifies query expressions, query topics, or query keywords that the investment research model is not allowed to generate response results for. During testing, various queries related to the restrictions can be designed (e.g., by making various changes to existing query expressions, topics, or keywords), and the security of the investment research model can be determined by testing whether it generates response results for these queries. This security can also be represented by a security score. For example, the number of tests that do not generate response results out of one hundred tests can be used as a score on the security dimension; however, this is not a limitation.

[0056] In the testing method of the investment research big data model claimed in this disclosure, multiple pairs consisting of expected questions and expected answers are generated based on an investment research corpus related to the investment research field. The recall capability of the investment research big data model is determined based on these pairs, thus providing a highly timely reference answer as a testing benchmark. Furthermore, by determining the generative capability of the investment research big data model across multiple generative dimensions, the semantics of long texts designed for actual testing in the investment research industry can be effectively evaluated. By determining the recall and generative capabilities of the investment research big data model, the application capability of the big data model in the investment research field can be comprehensively determined, improving the testing efficiency and accuracy of large language models in the investment research field.

[0057] Figure 5 The illustration shows an exemplary structural block diagram of a testing apparatus 500 for a large-scale investment research model according to an embodiment of the present disclosure. Figure 5 As shown, the testing device for the large-scale investment research model includes an expected result generation module 510, a recall capability determination module 520, an answer result generation module 530, a generation capability determination module 540, and a test result determination module 550.

[0058] The expected result generation module 510 is configured to generate multiple pairs consisting of expected questions and expected answers based on an investment research corpus related to the investment research field. The investment research corpus includes corpus segments related to multiple different topics.

[0059] The recall capability determination module 520 is configured to determine the recall capability of the investment research big model based on the plurality of pairs, wherein the recall capability characterizes the accuracy of the response returned by the investment research big model for the expected question relative to the expected response.

[0060] The answer result generation module 530 is configured to input queries related to at least one of the multiple different topics into the investment research big model to obtain multiple generated answer results.

[0061] The generation capability determination module 540 is configured to determine the generation capability of the investment research big model based on the multiple generated answer results, wherein the generation capability depends on the capability in multiple generation dimensions, the multiple generation dimensions including at least the data accuracy and data logic of the generated answer results, the data accuracy characterizing the accuracy of the multiple generated answer results, and the data logic characterizing the logical correctness of the multiple generated answer results.

[0062] The test result determination module 550 is configured to obtain the test results of the investment research model based on the recall capability and generation capability of the investment research model.

[0063] Figure 6 The illustration depicts an example system 600, which includes an example computing device 610 representing one or more systems and / or devices that can implement the various technologies described herein. The computing device 610 may be, for example, a server of a service provider, a device associated with a server, a system-on-a-chip, and / or any other suitable computing device or computing system. (Refer to above) Figure 5 The test apparatus 500 for the large-scale investment research model described can take the form of a computing device 610. Alternatively, the test apparatus 500 for the large-scale investment research model can be implemented as a computer program in the form of an application 616.

[0064] The example computing device 610 shown includes a processing system 611 communicatively coupled to each other, one or more computer-readable media 612, and one or more I / O interfaces 613. Although not shown, the computing device 610 may also include a system bus or other data and command transfer system that couples the various components to each other. The system bus may include any or a combination of different bus architectures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of the various bus architectures. Various other examples, such as control and data lines, are also conceived.

[0065] Processing system 611 represents the functionality of performing one or more operations using hardware. Therefore, processing system 611 is illustrated as including hardware elements 614 that can be configured as processors, function blocks, etc. This may include other logic devices implemented in hardware as application-specific integrated circuits (ASICs) or formed using one or more semiconductors. Hardware element 614 is not limited by the materials in which it is formed or the processing mechanism employed therein. For example, a processor may consist of semiconductors and / or transistors (e.g., integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically executable instructions.

[0066] Computer-readable medium 612 is illustrated as including memory / storage device 615. Memory / storage device 615 represents a memory / storage capacity associated with one or more computer-readable media. Memory / storage device 615 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage device 615 may include fixed media (e.g., RAM, ROM, fixed hard disk drive, etc.) and removable media (e.g., flash memory, removable hard disk drive, optical disk, etc.). Computer-readable medium 612 may be configured in various other ways as further described below.

[0067] One or more I / O interfaces 613 represent the functionality to allow users to input commands and information to computing device 610 using various input devices and optionally also to present information to the user and / or other components or devices using various output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones (e.g., for voice input), scanners, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., capable of detecting non-touch-related motion as gestures using visible or invisible wavelengths (such as infrared frequencies), and so on. Examples of output devices include display devices (e.g., monitors or projectors), speakers, printers, network interface cards, haptic-responsive devices, and so on. Therefore, computing device 610 can be configured in various ways, as further described below, to support user interaction.

[0068] The computing device 610 also includes an application 616. The application 616 may be, for example, a software instance of the task test scheduling device 300, and implements the techniques described herein in combination with other elements in the computing device 610.

[0069] This document describes various technologies within the general context of software and hardware components or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc., that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," and "component" generally refer to software, firmware, hardware, or a combination thereof. The technologies described herein are characterized as platform-independent, meaning that these technologies can be implemented on a variety of computing platforms with various processors.

[0070] Implementations of the described modules and technologies may be stored on or transmitted across some form of computer-readable medium. The computer-readable medium may include various media accessible by the computing device 610. By way of example and not limitation, the computer-readable medium may include "computer-readable storage media" and "computer-readable signal media".

[0071] In contrast to simple signal transmission, carrier waves, or signals themselves, a "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented using methods or techniques suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD or other optical storage devices, hard disks, magnetic tape cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of art suitable for storing desired information and accessible by a computer.

[0072] "Computer-readable signal medium" refers to a signal-bearing medium configured to transmit instructions, such as via a network, to computing device 610. A signal medium typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transmission mechanism. Signal media also include any information transmission medium. The term "modulated data signal" refers to a signal in which one or more of its characteristics are set or altered to encode information. By way of example and not limitation, communication media include wired media such as wired networks or direct connections, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0073] As previously described, hardware element 614 and computer-readable medium 612 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware, which in some embodiments can be used to implement at least some aspects of the techniques described herein. Hardware elements may include components of integrated circuits or systems-on-a-chip, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations or other hardware devices in silicon. In this context, hardware elements can serve as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by the hardware element, and as hardware devices for storing instructions for execution, such as the previously described computer-readable storage medium.

[0074] The foregoing combinations can also be used to implement the various techniques and modules described herein. Therefore, software, hardware, or program modules and other program modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 614. The computing device 610 can be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Thus, for example, by using the computer-readable storage medium and / or hardware elements 614 of a processing system, modules can be implemented at least partially in hardware as modules executable as software by the computing device 610. Instructions and / or functions can be executable / operable by one or more articles of art (e.g., one or more computing devices 610 and / or processing systems 611) to implement the techniques, modules, and examples described herein.

[0075] In various embodiments, the computing device 610 can be configured in various ways. For example, the computing device 610 can be implemented as a computer-type device, including personal computers, desktop computers, multi-screen computers, laptop computers, netbooks, etc. The computing device 610 can also be implemented as a mobile device, including mobile devices such as mobile phones, portable music players, portable gaming devices, tablet computers, multi-screen computers, etc. The computing device 610 can also be implemented as a television-type device, including devices with or connected to a generally large screen in a leisure viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0076] The techniques described herein can be supported by these various configurations of computing device 610, and are not limited to specific examples of the techniques described herein. Functionality can also be implemented, wholly or partially, on “cloud” 620 using distributed systems, such as platform 622 as described below.

[0077] Cloud 620 includes and / or represents platform 622 for resource 624. Platform 622 abstracts the underlying functionality of the hardware (e.g., server) and software resources of cloud 620. Resource 624 may include applications and / or data that can be used when performing computer processing on a server remote from computing device 610. Resource 624 may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks.

[0078] Platform 622 can abstract resources and functions to connect computing device 610 to other computing devices. Platform 622 can also be used to abstract resource hierarchy to provide a tiered hierarchy of the appropriate levels of demand for resource 624 implemented via platform 622. Therefore, in interconnect device embodiments, the implementation of the functions described herein can be distributed throughout system 600. For example, functions can be implemented partly on computing device 610 and partly through platform 622, which abstracts the functions of cloud 620.

[0079] This disclosure provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed, implement any of the methods described above.

[0080] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform any of the methods provided in the various alternative implementations described above.

[0081] It should be understood that, for clarity, embodiments of this disclosure have been described with reference to different functional units. However, it will be apparent that, without departing from this disclosure, the functionality of each functional unit may be implemented in a single unit, in multiple units, or as part of other functional units. For example, functionality described as being performed by a single unit may be performed by multiple different units. Therefore, references to a particular functional unit are considered merely as references to the appropriate unit used to provide the described functionality, and not as indicating a strict logical or physical structure or organization. Thus, this disclosure may be implemented in a single unit, or may be physically and functionally distributed among different units and circuits.

[0082] It will be understood that although the terms first, second, third, etc., may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these terms. These terms are used only to distinguish one device, element, component, or part from another device, element, component, or part.

[0083] Although this disclosure has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Rather, the scope of this disclosure is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may be advantageously combined, and inclusion in different claims does not imply that such a combination of features is not feasible and / or advantageous. The order of features in the claims does not imply that the features must be in any particular order of their operation. Furthermore, in the claims, the word "comprising" does not exclude other elements, and the terms "a" or "an" do not exclude a plurality. Reference numerals in the claims are provided only by way of explicit example and should not be construed as limiting the scope of the claims in any way.

Claims

1. A testing method for a large-scale investment research model, comprising: Based on an investment research corpus related to the investment research field, multiple pairs consisting of expected questions and expected answers are generated. The investment research corpus includes corpus segments related to multiple different topics. Based on the multiple pairs, the recall capability of the investment research model is determined, and the recall capability characterizes the accuracy of the response returned by the investment research model for the expected question relative to the expected response. Input queries related to at least one of the multiple different topics into the investment research big model to obtain multiple generated answer results; Based on the multiple generated answer results, the generation capability of the investment research big model is determined, wherein the generation capability depends on the capability in multiple generation dimensions, and the multiple generation dimensions include at least the data accuracy and data logic of the generated answer results. The data accuracy characterizes the accuracy of the multiple generated answer results, and the data logic characterizes the logical correctness of the multiple generated answer results. Based on the recall capability and generation capability of the aforementioned investment research model, the test results of the investment research model are obtained.

2. The method according to claim 1, wherein, The response results include the answer to the question and the corresponding citation, whereby the citation indicates the source of the answer. Furthermore, based on an investment research corpus related to the investment research field, multiple pairs consisting of expected questions and expected response results are generated, including: Select a target topic, where the target topic is one of the plurality of different topics; Select multiple core corpus segments related to the target topic from the aforementioned investment research corpus; Based on the multiple core corpus segments, multiple pairs consisting of expected questions and expected answers related to the target topic are generated, wherein the expected citations corresponding to the expected answers in the expected answers indicate the core corpus segments from which the expected answers originate.

3. The method according to claim 2, wherein, Based on the aforementioned multiple pairs, the recall capability of the investment research model is determined, including: The expected questions related to the target topic from the multiple pairs are input into the investment research model to obtain multiple real first answer results. Each real first answer result includes a first real answer and a first reference corresponding to the first real answer. Based on the expected answers to the expected questions related to the target topic and multiple actual first answers, the recall capability of the investment research model is determined. The recall capability depends on the accuracy of the references in the first answers relative to the references in the expected answers.

4. The method according to claim 2, wherein, Several core corpus segments related to the target topic were selected from the aforementioned investment research corpus, including: Select a core corpus segment related to the target topic from the aforementioned investment research corpus; Based on the selected core corpus segment, corpus segments from the investment research corpus that have a semantic similarity greater than the similarity threshold with the selected core corpus segment are selected as other core corpus segments among the multiple core corpus segments.

5. The method according to claim 1, wherein, Queries related to at least one of the multiple different topics are input into the investment research big data model to obtain multiple generated answer results, including: Retrieve queries related to at least one of the topics; Using the aforementioned investment research model, multiple generated response results are obtained for the query. Each generated response result includes at least one second true answer and a second reference corresponding to each second true answer. The second reference indicates the source of the corresponding second true answer.

6. The method according to claim 5, wherein, Using the aforementioned investment research model, multiple generated answer results are obtained for the query, including: Define the target theme; Based on the target topic, extract target topic queries related to the target topic from the query; The target topic query is input into the investment research big data model to obtain multiple generated answer results for the query.

7. The method according to claim 6, wherein, Based on the multiple generated responses, the generation capability of the investment research big data model is determined, including: For each generated answer, the content data from which the second true answer originates is obtained based on the second reference included in each generated answer; Based on the content data, determine the content summary data related to the target topic query, wherein the content summary data includes data obtained by summarizing the content data; Based on the content summary data and each of the second true answers, the generation capability of the investment research model is determined. The data accuracy of the answer results indicates the accuracy of each of the second true answers compared to the content summary data, and the data logic of the answer results indicates the logical correctness of each of the second true answers compared to the content summary data.

8. The method according to claim 7, wherein, The multiple generation dimensions also include one or more of the following: data integrity, data timeliness, and data readability of the generated answer results. Data integrity indicates the completeness of each second true answer relative to the representation of the content summary data; Data timeliness indicates the degree of consistency between each of the second true answers and the content summary data over time; Data readability indicates the degree of consistency between each of the second true answers and the representational structure of the content summary data.

9. The method according to claim 1, wherein, The multiple generation dimensions also include the data robustness of the generated answer results, where data robustness indicates the consistency of the generated answer results when multiple answer results are generated for the same query.

10. The method according to claim 1, wherein, Based on the multiple generated responses, the generation capability of the investment research big data model is determined, including: For each of the multiple generation dimensions, multiple judgment models are used to determine multiple scores of the generated answer on each generation dimension, and the multiple judgment models have different weights; Based on the weights of the multiple judgment models and the multiple scores, the generation capability of the investment research big model is determined in each generation dimension.

11. The method according to claim 1, wherein, The recall capability of the investment research model is represented by a recall capability score, and the generation capability of the investment research model is represented by a generation capability score. The test results of the investment research model are obtained based on its recall capability and generation capability, including: Determine the weighted sum of the recall capability score and the generation capability score; The weighted sum is determined as the test result of the investment research model.

12. A testing device for a large-scale investment research model, comprising: The expected result generation module is configured to generate multiple pairs consisting of expected questions and expected answers based on an investment research corpus related to the investment research field. The investment research corpus includes corpus segments related to multiple different topics. The recall capability determination module is configured to determine the recall capability of the investment research big model based on the plurality of pairs, wherein the recall capability characterizes the accuracy of the response returned by the investment research big model for the expected question relative to the expected response. The answer result generation module is configured to input queries related to at least one of the multiple different topics into the investment research big model to obtain multiple generated answer results; The generation capability determination module is configured to determine the generation capability of the investment research big model based on the multiple generated answer results, wherein the generation capability depends on the capability in multiple generation dimensions, the multiple generation dimensions including at least the data accuracy and data logic of the generated answer results, the data accuracy characterizing the accuracy of the multiple generated answer results, and the data logic characterizing the logical correctness of the multiple generated answer results; The test result determination module is configured to obtain the test results of the investment research model based on the recall capability and generation capability of the investment research model.

13. A computing device, comprising Memory, which is configured to store computer-executable instructions; A processor configured to perform the method as described in any one of claims 1-11 when the computer-executable instructions are executed by the processor.

14. A computer-readable storage medium storing computer-executable instructions that, when executed, perform the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes computer-executable instructions that, when executed, implement the method according to any one of claims 1 to 11.

Images