Financial model evaluation method and device, equipment and storage medium

Abstract

The invention discloses a financial model evaluation method and device, equipment and a storage medium, and relates to the technical field of model evaluation, and the method comprises the steps: predicting a test sample through a to-be-evaluated financial model, and obtaining a prediction result; the test sample is financial related image-text question and answer type sample data, and comprises a to-be-tested picture and a to-be-tested text; performing feature extraction on the prediction result and a preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and performing feature extraction on the to-be-tested picture to obtain an image feature; and evaluating the to-be-evaluated financial model based on a relationship among the first text feature, the second text feature and the image feature. Therefore, the financial ability of the financial model can be evaluated, the picture of the test sample and the text generated by the financial model are taken as reference factors, and the credibility of the evaluation process is improved; and an accurate model capability evaluation result can be obtained by combining a standard answer corresponding to the test sample.

Description

Technical Field

[0001] The present invention relates to the field of model evaluation technology, and in particular to a financial model evaluation method, device, equipment and storage medium. Background Art

[0002] Multimodal benchmarks in general fields (including text modality, voice modality, image modality, video modality) guide the rapid development of multimodal models on general tasks. However, the financial field has its own particularity. It has unique graphic images (e.g., K-line charts (a kind of charting tool), technical indicator charts) and has rich professional financial knowledge (e.g., futures, turnover rate). Therefore, benchmarks in general fields often cannot measure the performance of multimodal models in the financial field, and thus cannot effectively guide the rapid development of large financial models.

[0003] It can be seen that how to effectively evaluate the capabilities of financial models is a problem that needs to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a financial model evaluation method, device, equipment and storage medium, which uses the test sample image and the text generated by the financial model as reference factors to improve the credibility of the evaluation process; combined with the standard answer corresponding to the test sample, an accurate model capability evaluation result can be obtained. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a financial model evaluation method, comprising:

[0006] The test sample is predicted by the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the financial-related picture and text question and answer type, including the picture to be tested and the corresponding text to be tested;

[0007] Performing feature extraction on the prediction result and the preset standard answer corresponding to the test sample respectively to obtain a first text feature and a second text feature, and performing feature extraction on the image to be tested to obtain a corresponding image feature;

[0008] The financial model to be evaluated is evaluated based on the relationship between the first text feature, the second text feature, and the image feature.

[0009] Optionally, predicting the test sample by using the financial model to be evaluated to obtain corresponding prediction results includes:

[0010] The financial model to be evaluated is used to predict the test images representing the same content in several preset image formats and the corresponding test texts, and obtain several prediction results corresponding to the several image formats.

[0011] Optionally, the performing feature extraction on the prediction result and the preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and performing feature extraction on the image to be tested to obtain corresponding image features, includes:

[0012] Performing feature extraction on the prediction results corresponding to the tested images of each image format to obtain a corresponding number of first text features, and performing feature extraction on the preset standard answers corresponding to the test samples to obtain corresponding second text features;

[0013] Extracting features of the tested pictures with the same representation content in each picture format to obtain a corresponding number of image features;

[0014] Accordingly, the evaluating the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature includes:

[0015] Evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature, and the image feature corresponding to each image format, and obtain a corresponding number of initial evaluation results;

[0016] An average value is calculated for each of the initial evaluation results, and the obtained average value is determined as the target evaluation result of the financial model to be evaluated.

[0017] Optionally, the extracting features of the image to be tested to obtain corresponding image features includes:

[0018] Extracting features of the test pictures in different picture formats respectively by using a preset picture encoder to obtain a corresponding number of image features;

[0019] The preset image encoder is an image encoder based on a visual Transformer architecture.

[0020] Optionally, the extracting features of the prediction result and the preset standard answer corresponding to the test sample to obtain the first text feature and the second text feature respectively includes:

[0021] The prediction result and the preset standard answer corresponding to the test sample are respectively extracted by using a large language model preset for processing text to obtain a first text feature and a second text feature.

[0022] Optionally, the evaluating the financial model to be evaluated based on the relationship between the first text feature, the second text feature, and the image feature includes:

[0023] Comparing the first text feature and the second text feature based on a Compare function to obtain a corresponding first evaluation score;

[0024] A cosine similarity calculation is performed on the second text feature and the image feature to obtain a corresponding second evaluation score.

[0025] Optionally, the evaluating the financial model to be evaluated includes:

[0026] Determine a first weight parameter and a second weight parameter corresponding to the first evaluation score and the second evaluation score respectively; the sum of the first weight parameter and the second weight parameter satisfies a preset condition;

[0027] performing a multiplication operation on the first weight parameter and the first evaluation score to obtain a first result, and performing a multiplication operation on the second weight parameter and the second evaluation score to obtain a second result;

[0028] The sum of the first result and the second result is determined as an evaluation result of the financial model to be evaluated.

[0029] In a second aspect, the present application provides a financial model evaluation device, comprising:

[0030] A prediction module is used to predict the test sample through the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the financial-related picture and text question and answer type, including the picture to be tested and the corresponding text to be tested;

[0031] A feature extraction module is used to extract features from the prediction result and the preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and to extract features from the image to be tested to obtain corresponding image features;

[0032] An evaluation module is used to evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature.

[0033] In a third aspect, the present application provides an electronic device, including:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the financial model evaluation method as described above.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the financial model evaluation method as described above.

[0037] It can be seen that the present application first predicts the test sample through the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the financial-related picture-text question-and-answer type, including the picture to be tested and the corresponding text to be tested; then, the prediction result and the preset standard answer corresponding to the test sample are respectively subjected to feature extraction to obtain the first text feature and the second text feature, and the picture to be tested is subjected to feature extraction to obtain the corresponding image feature; then, the financial model to be evaluated is evaluated based on the relationship between the first text feature, the second text feature and the image feature. In this way, the present application can evaluate the financial capability of the financial model, using the picture of the test sample and the text generated by the financial model as reference factors to improve the credibility of the evaluation process; and then, accurate model capability evaluation results can be obtained in combination with the standard answer corresponding to the test sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a financial model evaluation method disclosed in this application;

[0040] Figure 2 A flowchart of a specific financial model evaluation method disclosed in this application;

[0041] Figure 3 A specific test sample statistical graph disclosed in this application;

[0042] Figure 4 This is a schematic diagram of the structure of a financial model evaluation device disclosed in this application;

[0043] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1As shown, the embodiment of the present invention discloses a financial model evaluation method, including:

[0046] Step S11, predicting the test sample through the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the finance-related picture and text question and answer type, including the picture to be tested and the corresponding text to be tested.

[0047] In the embodiment of the present application, in order to evaluate the model performance of the financial model to be evaluated, it is necessary to use the model for complete prediction. Specifically, the pre-prepared test samples are predicted by the financial model to be evaluated, and the corresponding prediction results can be obtained. It should be pointed out that the pre-prepared test samples are usually financial-related picture-text question-and-answer type data, that is, including financial image data (pictures to be tested) and corresponding texts to be tested. The text can represent financial issues related to the consulting pictures, or guide the mining of financial information contained in the pictures.

[0048] In a specific embodiment, the prediction of the test sample by the financial model to be evaluated to obtain the corresponding prediction results may include: predicting the test pictures and corresponding texts to be tested in several preset picture formats representing the same content by the financial model to be evaluated, and obtaining several prediction results corresponding to the several picture formats. Specifically, the test pictures in the prepared test sample can be in different picture formats, such as obtaining financial related image data by taking screenshots of financial pages in a computer, or by shooting with a mobile device; and the same content can be collected to obtain multiple pictures of different types, such as K-line charts, technical indicator charts, statistical charts, tables, documents, mixed charts, etc. In a specific embodiment, a bilingual multimodal financial benchmark data set can be selected to classify the collected financial related pictures; further, the test pictures and corresponding texts to be tested in different picture formats representing the same content need to be predicted by the financial model to be evaluated to obtain a corresponding number of prediction results; since the content represented by the test pictures is the same, the corresponding texts to be tested can be selected.

[0049] Step S12: performing feature extraction on the prediction result and the preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and performing feature extraction on the image to be tested to obtain corresponding image features.

[0050] In the embodiment of the present application, the financial model to be evaluated can be used to predict the test sample through the above steps, that is, to predict the test picture and the corresponding test text to be tested, and obtain the corresponding prediction result; further, the text feature of the prediction result can be extracted to obtain the first text feature, and the text feature of the text to be tested in the test sample can be extracted to obtain the second text feature; and the feature can also be extracted for the test picture to be tested to obtain the image feature. It can be understood that the first text feature corresponds to the prediction result obtained by the financial model to be evaluated for predicting the test sample, and the second text feature corresponds to the preset standard answer corresponding to the test sample. Therefore, the relationship between the first text feature and the second text feature can reflect the prediction ability of the financial model to be evaluated for the test sample. It should be pointed out that in order to ensure the reliability of the standard answer, at least two rounds of manual evaluation can be carried out in the process of preparing the standard answer corresponding to the test sample; specifically, several sample questions are designed for each task type. Subsequently, based on these sample questions, the large language model can be used to generate candidate questions for each image, and all questions can be manually reviewed to correct inappropriate questions. In the answer generation stage, the large language model can also be used to generate preliminary answers based on the questions and images. All answers are then manually checked to correct the wrong content. For complex and subjective questions, it is necessary to combine with experienced financial researchers to conduct evaluation. The reference answer will be finalized only after the review experts reach a consensus.

[0051] In a specific embodiment, the feature extraction of the prediction result and the preset standard answer corresponding to the test sample is respectively performed to obtain the first text feature and the second text feature, and the feature extraction of the picture to be tested is performed to obtain the corresponding image feature, which may include: feature extraction of the prediction result corresponding to the picture to be tested in each picture format to obtain a corresponding number of first text features, and feature extraction of the preset standard answer corresponding to the test sample to obtain a corresponding second text feature; feature extraction of the pictures to be tested with the same representation content in each picture format to obtain a corresponding number of image features; specifically, for pictures to be tested with the same content in different picture formats, feature extraction of the prediction result corresponding to the pictures to be tested in each picture format can be performed respectively, so that a corresponding number of first text features can be obtained, that is, the first text features corresponding to the prediction result corresponding to the picture to be tested in each picture format are obtained; accordingly, since the pictures to be tested in each picture format have the same representation content, the text to be tested with the same content can be adapted, that is, the pictures to be tested with the same content in multiple picture formats correspond to a single text to be tested; feature extraction of the single text to be tested can be performed to obtain a corresponding single second text feature. Furthermore, due to different image formats, even if the representation content is the same, it is necessary to perform feature extraction on the image to be tested in each image format to obtain a corresponding number of image features.

[0052] Accordingly, in another specific embodiment, the evaluation of the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature may include: evaluating the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature corresponding to each image format, and obtaining a corresponding number of initial evaluation results; averaging each of the initial evaluation results, and determining the obtained average value as the target evaluation result of the financial model to be evaluated. Specifically, through the above embodiment, the same number of first text features and image features as the number of image formats and a single second text feature can be obtained; it can be seen that one image format corresponds to a corresponding single first text feature, a single second text feature and a corresponding single image feature, and a single image format corresponds to these three features as a group of features; subsequently, the financial model to be evaluated can be evaluated for each group of features, so that the same number of initial evaluation results as the number of image formats can be obtained. Further, since the content corresponding to the text to be tested in each image format is the same, the final evaluation result corresponding to the financial model to be evaluated should be the average of each initial evaluation result.

[0053] In another specific embodiment, the feature extraction of the test picture to obtain the corresponding image features may include: extracting features of the test pictures of several different picture formats respectively through a preset picture encoder to obtain a corresponding number of image features; wherein the preset picture encoder is a picture encoder based on a visual Transformer architecture. Specifically, when extracting features of the test picture, image features may be extracted through a pre-prepared picture encoder, such as a picture encoder based on a visual Transformer architecture.

[0054] In a specific embodiment, the feature extraction of the preset standard answer corresponding to the prediction result and the test sample to obtain the first text feature and the second text feature may include: extracting the features of the preset standard answer corresponding to the prediction result and the test sample by a large language model pre-set for processing text to obtain the first text feature and the second text feature. Specifically, when extracting text features, the large language model pre-prepared for processing text may be used to extract text features of the preset standard answer corresponding to the prediction result and the test sample to obtain the corresponding first text feature and the second text feature.

[0055] Step S13: Evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature.

[0056] In an embodiment of the present application, the above steps can be used to use the financial model to be evaluated to predict the test sample, and extract the corresponding prediction results, preset standard answers and features of the image to be tested, to obtain the first text feature, the second text feature and the corresponding image feature; further, based on the relationship between the first text feature and the second text feature, and the relationship between the second text feature and the image feature, the performance of the financial model to be evaluated can be evaluated.

[0057] In a specific embodiment, the evaluation of the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature may include: comparing the first text feature and the second text feature based on the Compare function to obtain a corresponding first evaluation score; calculating the cosine similarity between the second text feature and the image feature to obtain a corresponding second evaluation score. Specifically, the first text feature and the second text feature may be compared by a comparison function, and the corresponding first evaluation score may be calculated based on the comparison result; correspondingly, for the second text feature and the image feature, the corresponding second evaluation score may be obtained by calculating the cosine similarity. It is understandable that the final evaluation result of the financial model to be evaluated should be calculated from the first evaluation score and the second evaluation score.

[0058] In a specific embodiment, the evaluation of the financial model to be evaluated may include: determining the first weight parameter and the second weight parameter corresponding to the first evaluation score and the second evaluation score respectively; the sum of the first weight parameter and the second weight parameter meets a preset condition; multiplying the first weight parameter and the first evaluation score to obtain a first result, and multiplying the second weight parameter and the second evaluation score to obtain a second result; determining the sum of the first result and the second result as the evaluation result of the financial model to be evaluated. Specifically, for the first evaluation score calculated by the first text feature and the second text feature, and the second evaluation score calculated by the second text feature and the image feature, the first evaluation score and the second evaluation score may be weighted by the weight parameter; the first evaluation score and the first weight parameter are multiplied, the second evaluation score and the second weight parameter are multiplied, and the results of the two multiplications are summed, and the sum obtained is used as the final evaluation result of the financial model to be evaluated, which can characterize the financial performance ability of the model.

[0059] It can be seen that the present application can prepare in advance the test images in various image formats and the corresponding test texts, and then predict the relevant test samples through the financial model to be evaluated; the financial capabilities of the financial model can be evaluated by combining the relationship between the prediction results, the standard answers corresponding to the test samples, and the image features of the images to be tested. The present application uses the images of the test samples and the texts generated by the financial model as reference factors, which can improve the credibility of the evaluation process; combined with the standard answers corresponding to the test samples, accurate model capability evaluation results can be obtained.

[0060] like Figure 2 As shown, the embodiment of the present application discloses a financial model evaluation method, which specifically includes:

[0061] In this embodiment, for three types (picture formats) of test pictures A with the same content and the corresponding test texts, predictions are made through the multimodal model to be evaluated, and three corresponding prediction results can be obtained; the prediction results are combined with the standard answers corresponding to the pre-prepared test samples, and features can be extracted through the text model to obtain three corresponding text features; the text features here represent three groups of text features, each group of text features includes a first text feature corresponding to a single prediction result and a second text feature corresponding to the standard answer. Accordingly, for the three types of test pictures, features can be extracted through the pre-prepared visual encoder to obtain the corresponding three image features.

[0062] Furthermore, the obtained three groups of text features and the corresponding three image features can be weighted by the multimodal evaluation model to obtain the final evaluation result of the multimodal model to be evaluated. Specifically, in the process of weighted calculation by the multimodal evaluation model, the formula for calculating the first evaluation score is as follows:

[0063] ;

[0064] Among them, pred i is the i-th prediction result, w i is the weight of each prediction result, and Compare() is the comparison function, satisfying the weight constraint: .

[0065] Correspondingly, the image feature extraction formula is as follows:

[0066] ;

[0067] Among them, Fi represents the i-th feature, is the feature weight, is the activation function and m is the number of features.

[0068] Furthermore, the formula for calculating the second evaluation score by cosine similarity is as follows:

[0069] ;

[0070] in, The text features of the standard answer.

[0071] The final evaluation score weighted calculation formula is as follows:

[0072] ;

[0073] in, and is the weight parameter.

[0074] In a specific embodiment, the test sample is Figure 3As shown in the figure, it covers different types of tasks, which are divided into different levels according to difficulty, and shows the proportion of test samples under each evaluation dimension. There are 1,103 Chinese test samples and 1,171 English test samples in the test set. The number of samples for each task ranges from 42 to 395. When using multimodal models for prediction, it can be a question and answer of the cause identification type. For example, the question related to the candlestick chart to be tested can be "Why did stock A experience a significant decline on August 21, 2024?", and the corresponding background information can be given "2024-08-21|Reason for the decline in A's stock price A's stock price fell today because of news that B sold its shares in A. Founded in 1993, C is a financial services company dedicated to making the world smarter, happier and richer. "A Foundation" reaches millions of people every month through free guidance and market analysis on our premium investment solutions, high-rated podcasts, and non-profit organization "D". B sold approximately $3.6 billion of A's stock, all of its shares in the company. According to a document filed by E this morning, B sold A's shares, causing the company's stock price to fall. In a certain country, A and Walmart were once considered partners, but the sale seems to indicate that Walmart has lost interest in the relationship, especially after A's stock fell in the past few years. The model combines the picture under test with the question and background information to predict that "A experiences a significant decline on August 21, 2024, mainly due to B's decision to sell its entire stake in F. The move was seen as a significant vote of no confidence in A, especially given B's long-term investment in the company since 2016. The sale raised approximately $3.6 billion for B and marked the end of an eight-year investment partnership. The market reacted negatively to the news, causing A's share price to fall sharply. Investors were concerned about the impact of B's ​​exit, seeing it as a sign that A was struggling in the competitive F and a sign of its slow sales growth since the pandemic. Overall sentiment towards emerging markets also contributed to the decline in the share price, as B's sale exacerbated investors' cautious sentiment. In summary, the sharp decline in A's share price on August 21, 2024 was driven by B's divestiture of its stake in the company, reflecting broader concerns about A's performance and F's competitive landscape. ". Correspondingly, when using a multimodal model for prediction, it can be a character recognition type question and answer. For example, a question related to the K-line chart to be tested can be "What is the 5 / 10 / 15 / 30 / 60-day moving average value (SMA) in the figure?" The result predicted by the model can be "The values ​​of the 5 / 10 / 20 / 30 and 60-day moving averages of stock M are as follows: SMA 5: 448.088S; MA 10: 444.670; SMA 20: 433.292; SMA 30: 430.082; SMA 60: 420.903".

[0075] It can be seen that the present application can prepare in advance the test images in various image formats and the corresponding test texts, and then predict the relevant test samples through the financial model to be evaluated; the financial capabilities of the financial model can be evaluated by combining the relationship between the prediction results, the standard answers corresponding to the test samples, and the image features of the images to be tested. The present application uses the images of the test samples and the texts generated by the financial model as reference factors, which can improve the credibility of the evaluation process; combined with the standard answers corresponding to the test samples, accurate model capability evaluation results can be obtained.

[0076] like Figure 4 As shown, the embodiment of the present application discloses a financial model evaluation device, including:

[0077] The prediction module 11 is used to predict the test sample through the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the financial-related picture-text question-and-answer type, including the picture to be tested and the corresponding text to be tested;

[0078] A feature extraction module 12 is used to perform feature extraction on the prediction result and the preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and perform feature extraction on the image to be tested to obtain a corresponding image feature;

[0079] The evaluation module 13 is used to evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature.

[0080] It can be seen that the present application can evaluate the financial capabilities of financial models, using the images of test samples and the text generated by the financial model as reference factors to improve the credibility of the evaluation process; combined with the standard answers corresponding to the test samples, accurate model capability evaluation results can be obtained.

[0081] In a specific embodiment, the prediction module 11 may include:

[0082] The prediction unit is used to predict the test pictures representing the same content in several preset picture formats and the corresponding test texts through the financial model to be evaluated, and obtain several prediction results corresponding to the several picture formats respectively.

[0083] In a specific embodiment, the feature extraction module 12 may include:

[0084] A first feature extraction unit is used to extract features from the prediction results corresponding to the test images of each image format to obtain a corresponding number of first text features, and to extract features from the preset standard answers corresponding to the test samples to obtain corresponding second text features;

[0085] A second feature extraction unit is used to extract features from the test pictures with the same representation content in each picture format to obtain a corresponding number of image features;

[0086] Accordingly, the evaluation module 13 may include:

[0087] A first evaluation unit, configured to evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature, and the image feature corresponding to each image format, and obtain a corresponding number of initial evaluation results;

[0088] The second evaluation unit is used to calculate an average value of each of the initial evaluation results, and determine the obtained average value as the target evaluation result of the financial model to be evaluated.

[0089] In a specific embodiment, the feature extraction module 12 may include:

[0090] The third feature extraction unit is used to perform feature extraction on the test pictures of several different picture formats respectively through a preset picture encoder to obtain a corresponding number of image features; wherein the preset picture encoder is a picture encoder based on a visual Transformer architecture.

[0091] In another specific embodiment, the feature extraction module 12 may include:

[0092] The fourth feature extraction unit is used to extract features of the prediction result and the preset standard answer corresponding to the test sample respectively through a preset large language model for processing text, so as to obtain a first text feature and a second text feature.

[0093] In a specific embodiment, the evaluation module 13 may include:

[0094] a third evaluation unit, configured to compare the first text feature and the second text feature based on a Compare function to obtain a corresponding first evaluation score;

[0095] The fourth evaluation unit is used to calculate the cosine similarity between the second text feature and the image feature to obtain a corresponding second evaluation score.

[0096] In another specific embodiment, the evaluation module 13 may include:

[0097] A weight determination unit, configured to determine a first weight parameter and a second weight parameter corresponding to the first evaluation score and the second evaluation score, respectively; a sum of the first weight parameter and the second weight parameter satisfies a preset condition;

[0098] an operation unit, configured to perform a multiplication operation on the first weight parameter and the first evaluation score to obtain a first result, and to perform a multiplication operation on the second weight parameter and the second evaluation score to obtain a second result;

[0099] An evaluation result determining unit is used to determine the sum of the first result and the second result as the evaluation result of the financial model to be evaluated.

[0100] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0101] Figure 5 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the financial model evaluation method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0102] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the financial model evaluation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0105] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed financial model evaluation method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0106] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0108] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0110] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A financial model evaluation method, characterized in that: include: Predict the test samples through the financial model to be evaluated and obtain the corresponding prediction results; The test sample is sample data of a finance-related picture-text question-and-answer type, including a picture to be tested and a corresponding text to be tested; Performing feature extraction on the prediction result and the preset standard answer corresponding to the test sample respectively to obtain a first text feature and a second text feature, and performing feature extraction on the image to be tested to obtain a corresponding image feature; The financial model to be evaluated is evaluated based on the relationship between the first text feature, the second text feature, and the image feature.

2. The financial model evaluation method according to claim 1, characterized in that: The prediction of the test sample by the financial model to be evaluated to obtain the corresponding prediction results includes: The financial model to be evaluated is used to predict the test images representing the same content in several preset image formats and the corresponding test texts, and obtain several prediction results corresponding to the several image formats.

3. The financial model evaluation method according to claim 2, characterized in that: The feature extraction is performed on the prediction result and the preset standard answer corresponding to the test sample to obtain the first text feature and the second text feature, and the feature extraction is performed on the image to be tested to obtain the corresponding image feature, including: Performing feature extraction on the prediction results corresponding to the tested images of each image format to obtain a corresponding number of first text features, and performing feature extraction on the preset standard answers corresponding to the test samples to obtain corresponding second text features; Extracting features of the tested pictures with the same representation content in each picture format to obtain a corresponding number of image features; Accordingly, the evaluating the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature includes: Evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature, and the image feature corresponding to each image format, and obtain a corresponding number of initial evaluation results; An average value is calculated for each of the initial evaluation results, and the obtained average value is determined as the target evaluation result of the financial model to be evaluated.

4. The financial model evaluation method according to claim 2, characterized in that: The step of extracting features from the image to be tested to obtain corresponding image features includes: Extracting features of the test pictures in different picture formats respectively by using a preset picture encoder to obtain a corresponding number of image features; The preset image encoder is an image encoder based on a visual Transformer architecture.

5. The financial model evaluation method according to claim 1, characterized in that: The feature extraction of the prediction result and the preset standard answer corresponding to the test sample is performed to obtain the first text feature and the second text feature, including: The prediction result and the preset standard answer corresponding to the test sample are respectively extracted by using a large language model preset for processing text to obtain a first text feature and a second text feature.

6. The financial model evaluation method according to any one of claims 1 to 5, characterized in that: The evaluating the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature includes: Comparing the first text feature and the second text feature based on a Compare function to obtain a corresponding first evaluation score; A cosine similarity calculation is performed on the second text feature and the image feature to obtain a corresponding second evaluation score.

7. The financial model evaluation method according to claim 6, characterized in that: The evaluating the financial model to be evaluated includes: Determine a first weight parameter and a second weight parameter corresponding to the first evaluation score and the second evaluation score respectively; the sum of the first weight parameter and the second weight parameter satisfies a preset condition; performing a multiplication operation on the first weight parameter and the first evaluation score to obtain a first result, and performing a multiplication operation on the second weight parameter and the second evaluation score to obtain a second result; The sum of the first result and the second result is determined as an evaluation result of the financial model to be evaluated.

8. A financial model evaluation device, characterized in that: include: A prediction module is used to predict the test sample through the financial model to be evaluated to obtain the corresponding prediction result; the test sample is sample data of the financial-related picture and text question and answer type, including the picture to be tested and the corresponding text to be tested; A feature extraction module is used to extract features from the prediction result and the preset standard answer corresponding to the test sample to obtain a first text feature and a second text feature, and to extract features from the image to be tested to obtain corresponding image features; An evaluation module is used to evaluate the financial model to be evaluated based on the relationship between the first text feature, the second text feature and the image feature.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the financial model evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the financial model evaluation method according to any one of claims 1 to 7.

Images