A method, system and device for constructing and evaluating a bilingual financial large model for Chinese and English

Abstract

The application discloses a method, system and device for constructing and evaluating a bilingual financial large model. By collecting original financial corpus and combining high-quality prompts annotated by experts, a large-scale and diversified financial instruction dataset ICE-FIND containing 8 specific tasks, 36 datasets and 604000 samples is constructed. On this basis, the InternLM bilingual base model is fine-tuned using 80% as the training set and combining the QLora fine-tuning technology to construct the bilingual financial large model ICE-INTENT. In order to comprehensively evaluate the performance of different financial models, 20% is used as the test set to establish the bilingual evaluation benchmark ICE-FLARE, and the financial task without fine-tuning is introduced. The experimental results show that the performance of the large model proposed in the application on the bilingual financial task is better than that of various baseline models and financial large models, and even surpasses the optimal GPT-4, showing its advantages in diversified language resources, strategy instruction optimization and cross-language generalization.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and specifically designs a method, system and device for constructing and evaluating a bilingual financial large model. BACKGROUND

[0002] Nowadays, artificial intelligence technology is developing at an unprecedented speed, and large language models (LLMs) as an important technology in the field of artificial intelligence are leading the innovation of human-computer interaction. Large language models are a kind of artificial intelligence systems with massive parameters that can automatically learn language rules and generate text. Their appearance not only changes people's cognition of intelligent interaction, but also brings great changes to the fields of language processing and natural language understanding. The purpose of large language models is to understand and generate natural language, and the core technology is based on deep learning and neural networks. Through large-scale text data training, they can understand the grammar, semantics, logic and other aspects of human language, and predict the next word or generate content related to the given text by learning a large amount of text data. This model can realistically simulate human language expression, not only can it have a conversation, answer questions, but also can creatively generate articles, poems, stories and other text content. The application field of large language models is very wide, and they have been applied to intelligent customer service, intelligent assistants, intelligent creation, knowledge Q&A and many other fields. For example, in the field of education, large language models can provide personalized learning assistance for students, help them solve problems and understand knowledge; in the medical field, they can assist doctors in analyzing cases and developing treatment plans; in the creative field, large language models can cooperate with humans to create literary works, film scripts, etc.

[0003] Despite the great potential and application value of Large Language Models (LLMs) in various fields, their development in the financial sector has not been fully explored and utilized. However, with the increasing digitalization and informatization of the financial industry, the application prospects of LLMs in the financial sector are becoming increasingly attractive. The financial sector is an industry with a large amount of information and rapid changes, which contains a large amount of text data such as financial reports, market analysis, news reports, etc. LLMs have the ability to process large-scale text data, and their application potential in the financial sector is self-evident. Especially with the expansion of China's investment and consumption influence, cross-language communication has become increasingly important in global business. The interaction between Chinese and American financial markets has strengthened the need to bridge the language and cultural gap in the financial sector. To bridge the language and cultural gap in the complex financial sector, specialized tools and models need to be tailored for cross-language business scenarios, and financial LLMs have gradually become the direction of exploration for cross-language solutions. From widely functional models such as ChatGPT to BloombergGPT, FinGPT, InvestLM, and FinTral, which focus on finance, the diversification of instruction data for FinMA marks a significant progress. Chinese FinLLM has also made breakthroughs, such as Baidu's "Xuan Yuan", DISC-FinLLM, CFGPT, and Huawei's PanGu-π. These models provide diverse solutions to address the challenges of the financial market.

[0004] Despite the progress made by existing financial large model technologies, there are still many challenges in bilingual applications and evaluations. First, most existing models are designed for a single language, lacking bilingual support, which limits their application scope. Second, most models are not open source, and key fine-tuning instruction datasets are often closed, which poses obstacles to research. In addition, evaluation methods are mostly focused on monolingual evaluation, failing to fully measure bilingual capabilities, and lacking validation of out-of-training tasks, which may lead to exaggerated evaluation indicators. At the same time, existing instruction datasets lack diversity in task categories, especially in the Chinese part, and the development of original and corresponding translation datasets is often overlooked, hindering the evaluation of model cross-language generalization capabilities.

[0005] Therefore, the present application inventors found that the prior art method at least has the following problems in the process of implementing the present application: (1) all models are specially tailored for English or Chinese, which highlights the lack of models proficient in both languages. (2) Most models do not have open source code, especially almost all key fine-tuning teaching data sets are closed, which will seriously hinder research in this field. (3) Current evaluation methods tend to focus on monolingual evaluation, failing to comprehensively evaluate bilingual capabilities. More worrying, these methods lack performance verification on training task data, resulting in evaluation indicators that may be exaggerated. (4) Although existing instruction data sets include financial NLP tasks such as classification, extraction, prediction, and reasoning, these data sets, especially Chinese data sets, lack diversity in each task category. (5) When developing a comprehensive bilingual teaching data set, the original data set and the corresponding translation data set are often ignored, making it challenging to evaluate the cross-language generalization capabilities of the model. SUMMARY

[0006] To address the challenges in bilingual financial tasks, the first specially designed open-source Chinese-English bilingual large model and evaluation method, system and device aims to fill the existing research gap. The disclosed 40 carefully planned datasets contain a total of 1,185,076 original data points, 603,940 fine-tuning instruction data and 95,091 evaluation data. This extensive collection covers a variety of bilingual financial tasks, highlighting our commitment to language diversity and task-specific depth, ensuring the effectiveness of ICE-INTENT in various financial scenarios. The invention includes a bilingual open-source multi-task instruction dataset (ICE-FIND) containing 36 datasets covering 18 specific tasks, and a pioneering bilingual evaluation benchmark (ICE-FLARE) containing 40 datasets covering 20 specific tasks. In addition, it also includes an open-source Chinese-English bilingual financial large language model ICE-INTENT. To establish robust multi-task and bilingual instruction data and evaluation resources, we integrated Chinese and English financial tasks from four data categories: (A) Chinese financial datasets, (B) English financial datasets, (C) Chinese translation datasets, and (D) out-of-domain financial datasets, combining expert annotated prompts with task-specific data samples. This led to the creation of ICE-FIND, a Chinese-English bilingual financial instruction data. To enhance bilingual financial analysis capabilities, we introduced the bilingual financial large language model ICE-INTENT, which was fine-tuned on the InternLM-7B base model to adapt to ICE-FIND. To evaluate model performance, we further constructed the bilingual ICE-FLARE benchmark, including test sets from ICE-FIND, as well as introducing out-of-domain data (DOT) tasks and 3 unseen specific tasks and 4 unseen datasets. The addition of DOT tasks aims to evaluate the model's generalization ability in various financial contexts and tasks.

[0007] The invention integrates Chinese and English financial specific tasks from four categories of data by creating multi-task bilingual instruction data and evaluation benchmarks, combining expert annotated prompts with specific task data samples to generate Chinese-English bilingual financial structured data; improves bilingual financial analysis capabilities through fine-tuning base and instruction dataset fine-tuning; further constructs a bilingual financial large model benchmark for evaluation by introducing tasks related to training data and 3 unseen specific tasks and 4 unseen datasets, which includes test sets from the instruction dataset, and introduces tasks related to training data and 3 unseen specific task datasets, which can evaluate the model's generalization ability in various financial environments and tasks.

[0008] To solve the above technical problems, the first aspect of the invention provides a method for constructing and evaluating a Chinese-English bilingual financial large model, comprising the following steps:

[0009] S1. Collect text descriptions through public channels such as market reports, news articles, financial forums, and social media, and organize them into various types of English financial data with corresponding labels. This step covers a wide range of real data collection from different countries and language regions, combining different language descriptions and cultural backgrounds;

[0010] The specific steps for collecting English financial data are as follows:

[0011] S1.1. Clearly define the target and scope of data collection, determine the required data types and sources; for example, for stock market data, historical prices, trading volume and company financial report information are needed, which can be obtained through public financial APIs, exchange websites or data providers;

[0012] S1.2. Write scripts in Python to obtain data through API interfaces, and store the collected data as CSV and JSON files in the database;

[0013] S2. Build a bilingual financial instruction data set by processing the collected data;

[0014] After data collection, enter the data processing phase, which ensures the quality and applicability of the data, the specific steps are as follows:

[0015] S2.1. Preliminary inspection and processing of missing values, repeated values and outliers in the collected data;

[0016] Preliminary inspection of data includes: verifying whether the data types meet the expectations (for example, date fields should be date types, numerical fields should be numerical types);

[0017] Fill in, delete or interpolate missing values in the data through manual correction;

[0018] Remove repeated values in the data through text matching to avoid interference with the analysis results;

[0019] Deal with outliers in the data through statistical analysis, data standardization or anomaly detection algorithm to ensure the accuracy and integrity of the data;

[0020] S2.2. Clean and convert the processed data, and construct the initial financial data set through feature engineering, i.e. data cleaning and data conversion;

[0021] Data cleaning includes: converting date strings into standard date formats, ensuring data consistency and operability; encoding is required for category data representing different categories or category labels, such as converting categorical variables to numerical types so that models can be processed; for numerical data such as age, date and income, standardization or normalization is required to eliminate the influence of different scales on the analysis results;

[0022] Data conversion includes: format checking and logical checking of data, removing format abnormal values and logical error values in data, ensuring data accuracy;

[0023] Further, the steps of constructing a bilingual financial instruction data set are as follows:

[0024] S2.3, generate high-quality English-Chinese cross-language translation data set by ChatGPT;

[0025] Specifically, use ChatGPT to translate English multi-task data into Chinese, and keep stock code information in complex stock classification tasks. Through manual review of language structure and professional term translation differences, ensure data integrity, and finally through multiple rounds of manual correction and evaluation, eliminate text data that is prone to error or cannot be translated;

[0026] S2.4, according to the difference of different data in different financial tasks (such as Table 1, including sentiment analysis, semantic matching, news classification, negative message judgment, question and answer selection, stock prediction, relationship extraction, title classification, credit evaluation, hawk-pigeon classification, question and answer, named entity recognition, event detection, text summary), manually annotate high-quality prompts for different financial tasks;

[0027] The specific steps of manually annotating high-quality prompts include the following:

[0028] S2.4.1, clearly define the goals and needs of each financial task, write 20 unique prompts for each data set in different financial tasks, and use different prompt structures and formats;

[0029] S2.4.2, manually evaluate and filter the prompts for financial tasks;

[0030] The artificial evaluation team consists of 10 students proficient in Chinese and English and with a background in the financial field; Each person writes 20 to 30 unique prompts for each data set, manually evaluates the corresponding prompts in various financial specific tasks through accuracy, naturalness, and information quantity indicators (0, 1, 2, 3 points), tests different prompt variants, and retains prompts with an average value score of more than 2 by calculating the average value of the three indicators, and filters other prompts;

[0031] S2.5, combined with artificial prompts and financial task description, build financial instruction data set; The specific steps are as follows:

[0032] S2.5.1, by defining the task type of finance and input and output target, design the instruction data format suitable for the task type, define the JSON format of input and output as needed;

[0033] Input part: define all the necessary information required by the task, including raw data, problem description, context;

[0034] Output part: define the expected result format, which is usually the prediction result of the model, classification label, generated text;

[0035] S2.5.2, according to the type of financial instruction data set, divide different types of financial tasks according to the ratio of 7:1:2, and build ICE-FIND bilingual financial instruction data set;

[0036] S3, through the constructed Chinese-English bilingual financial instruction data set, build Chinese-English bilingual financial large model; The steps are as follows:

[0037] S3.1, select the base model with the best performance;

[0038] Specifically, by evaluating the bilingual performance of 5 kinds of backbone large models (InternLM-7B, Bloomz-7B1, ChatGLM3-6B, FinMA-7B, DISCFin-13B), InternLM-7B is selected as the bilingual financial base model;

[0039] S3.2, by using the divided training set to fine-tune the model, optimize the model parameters to adapt to specific tasks, which includes the following steps:

[0040] S3.2.1, by introducing 4 types of data sets (including classification, extraction, translation and English data) in turn during fine-tuning, create 4 variants of ICE-INTENT model to evaluate the influence of various data types on fine-tuned bilingual model;

[0041] The creation of 4 model variants is to fine-tune the InternLM-7B base model from the classification data set, then introduce the extraction, translation and English data set in turn each time, observe the change of evaluation index on the model after introducing different types of data sets, and create 4 model variants according to the combination of different data sets;

[0042] 4 model variants include: model ICEdlc-7B fine-tuned only with classification task dataset, model ICEdle-7B fine-tuned with classification + extraction task dataset, model ICEdtt-7B fine-tuned with classification + extraction + translation dataset, model ICEfull-7B fine-tuned with all 4 types of dataset together;

[0043] S3.2.2, optimize memory usage and reduce training time by introducing advanced fine-tuning techniques (QLora);

[0044] Specifically, only the necessary low-rank matrices need to be updated, while all weights are quantized to 4-bit, reducing the storage and computational requirements of the model, making large-scale model fine-tuning more efficient and convenient;

[0045] S3.2.3, fine-tune the model on the platform with a batch size of 24 using the QLora technique;

[0046] Specifically, by cutting the training sequence (text data input into the model for training) into 2048 tokens, using the AdamW optimizer, the initial learning rate is 5e-5, and the weight decay is 1e-5; In addition, the warm-up step (the process of gradually increasing the learning rate at the beginning of training) is set to 1% of the total training steps (total number of training iterations), and the learning rate will gradually increase from 0 to the set initial learning rate in the first 1% of training steps;

[0047] During training, the training task is divided into 24 batches for GPU parallel processing, while monitoring memory usage and training speed; by merging the pre-training weights of the base model InternLM with the low-rank matrices, the specific steps are as follows: decompose the base model weights into low-rank matrices A and B, then update the weights in each layer of the Transformer as follows:

[0048] W' = W + (A x B)

[0049] In the formula, W is the original pre-training weight matrix of the base model, which contains the knowledge and parameters learned by the model during training; A is part of the low-rank matrix, which is usually a small parameter matrix that will be updated during training to adapt to the needs of a specific task; B is another low-rank matrix related to A, which is also usually a small parameter matrix, matching the dimensions of A, used to reduce the number of model parameters; A x B represents the product of low-rank matrices, generating a new matrix with the same dimensions as the original weight matrix W, representing new parameters added to the original weights after low-rank decomposition adjustment; W' represents the final model weight matrix after merging;

[0050] After training is complete, the final model's weights are composed of the original weights and a low-rank matrix, preserving the knowledge of the base model and achieving optimization for specific tasks through adjustments to the low-rank matrix.

[0051] S3.3. Ensure effective model learning by observing changes in loss functions (such as training loss and validation loss), accuracy (such as training accuracy and validation accuracy), F1 score, precision, recall, and area under the curve (AUC-ROC) during the process.

[0052] S3.4. Evaluate model performance using a test set at the end of each training epoch to adjust hyperparameters (deep learning hyperparameter tuning) and avoid overfitting during model training.

[0053] S4. Construct the evaluation benchmark ICE-FLARE using the pre-defined 20% instruction test set, and formulate different evaluation metrics for different tasks (as shown in Table 1, including accuracy, F1 score, entity F1 score, recall, Rouge, BERTScore, and BARTScore). Each metric helps to comprehensively analyze the model performance.

[0054] Table 1. Detailed information on specific tasks, language types, dataset names, and evaluation metrics in this invention.

[0055]

[0056] Based on the same inventive concept, a second aspect of the present invention provides a system for constructing and evaluating large-scale financial models in both Chinese and English, comprising the following modules:

[0057] (1) Financial raw data collection module, used to extract relevant content from financial market reports, news articles, financial forums and public social media channels;

[0058] Specifically, the raw financial data collection module is used to extract relevant content from financial news websites, market analysis reports, and financial forums. Each data entry needs to contain clear market analysis instructions and extract corresponding responses from actual market analysis articles to obtain preliminary data for processing.

[0059] (2) Financial instruction dataset construction module, which is used to process the collected data to construct the Chinese-English bilingual financial instruction dataset ICE-FIND containing translated data and to divide the dataset;

[0060] Specifically, according to the instruction data set format and English and Chinese cross-language data requirements required by the framework, the cross-language Chinese data set is constructed by translating the data set of the ChatGPT translation query question and answer, stock classification, sentiment analysis and other tasks, 20-30 high-quality prompts Prompt written by financial experts are used for artificial evaluation and correction, and the financial instruction data set is constructed based on the JSON format standard in Figure 4

[0061] (3) The financial large model fine-tuning module is used for fine-tuning 80% of the training set in the constructed financial instruction data set based on the selected English and Chinese base model with superior bilingual performance to construct an English and Chinese bilingual financial large model ICE-INTENT.

[0062] Specifically, the financial large model fine-tuning module refers to constructing an English and Chinese bilingual financial large model by fine-tuning based on the QLora technology through the financial instruction fine-tuning data set. The effective learning of the model is ensured by observing the loss function, accuracy and other indicators in the model fine-tuning process.

[0063] (4) The financial evaluation benchmark evaluation module is used for evaluating the model performance by using 20% of the independent test set in the constructed financial instruction data set, different evaluation indexes are formulated for different financial tasks, and a comprehensive evaluation benchmark ICE-FLARE is constructed.

[0064] Specifically, the financial evaluation benchmark evaluation module refers to evaluating the model performance by using 20% of the independent test set in the constructed financial instruction data set, different evaluation indexes are formulated for different financial tasks, and each index can help to comprehensively analyze the model performance, and a comprehensive evaluation benchmark is constructed.

[0065] Based on the same inventive concept, the third aspect of the present application provides a system for constructing and evaluating an English and Chinese bilingual financial large model, which is a computer readable storage medium having a computer program stored thereon, and the program is executed to realize the method of the first aspect.

[0066] Based on the same inventive concept, the fourth aspect of the present application provides a system device for constructing and evaluating an English and Chinese bilingual financial large model, which is a computer device, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the program to realize the method of the first aspect.

[0067] The practical value of the present application

[0068] ​1、The present application constructs a comprehensive Chinese-English financial large model that utilizes knowledge in finance, natural language processing (NLP), machine learning (ML), and other fields by in-depth analysis and research of existing large language model algorithms and models, which can process and understand financial data in different languages. It effectively solves the obstacles of cross-language information processing in the financial field and provides global financial market participants with a wider range of information sources and more accurate analysis capabilities.

[0069] 2、The present application establishes an effective Chinese-English bilingual financial large model evaluation benchmark to address the challenges of multi-task and bilingual data in the financial field, solving the problem of evaluating the performance of different financial multi-language models. With its convenient and fast characteristics, it continuously optimizes the model to adapt to changes in the financial market, enhancing the value of long-term application.

[0070] Advantages of the present application

[0071] The present application has significant advantages over existing technology, especially in cross-language, multi-task processing, and financial-specific tasks:

[0072] (1) The financial framework disclosed in the present application can process Chinese and English financial text simultaneously, breaking the limitation of existing models mainly supporting single language and improving the breadth and accuracy of cross-language analysis.

[0073] (2) The large model disclosed in the present application combines knowledge specific to the financial field, enabling better understanding and analysis of complex financial terminology and context, and exhibiting higher task-specific accuracy. In core tasks such as sentiment analysis, risk judgment, news classification, and stock prediction, it exhibits stronger performance.

[0074] (3) The large model disclosed in the present application integrates multi-source financial data and NLP tasks, achieving higher accuracy and robustness in information extraction, reasoning, and prediction, reducing data misinterpretation and analysis errors.

[0075] (4) The evaluation benchmark disclosed in the present application can effectively evaluate the consistency of current financial large models in bilingual performance, continuously measuring and optimizing the performance of financial models in different languages. BRIEF DESCRIPTION OF DRAWINGS

[0076] To more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiment or prior art description. Obviously, the drawings in the following description only show part of the embodiments of the present application, and those skilled in the art can derive other related drawings from these drawings without creative labor.

[0077] Figure 1An implementation flowchart of a method for constructing and evaluating a Chinese-English bilingual financial large model is provided in the present application.

[0078] Figure 2 An overview diagram of the present application is shown, which demonstrates unique data types, financial tasks, and bilingual data sources, and provides services for different user groups in various financial scenarios.

[0079] Figure 3 An example of translating English BigData22 into Chinese CBigData22 using specific translation prompts in the stock classification financial task in the present application;

[0080] Figure 4 A customized format of instruction data set used for fine-tuning of the Chinese-English bilingual financial large model in the embodiment of the present application;

[0081] Figure 5 An evaluation format used in the evaluation framework of the Chinese-English bilingual financial large model in the embodiment of the present application;

[0082] Figure 6 A histogram of the best result count of the cornerstone model on Chinese and English tasks in the embodiment of the present application;

[0083] Figure 7 A structural block diagram of the method for constructing and evaluating a Chinese-English bilingual financial large model in the embodiment of the present application;

[0084] Figure 8 A structural block diagram of a computer readable storage medium in the embodiment of the present application;

[0085] Figure 9 A structural diagram of a computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0086] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0087] The present application will be further described in detail below with reference to specific embodiments.

[0088] Embodiment One

[0089] Referring to Figures 1-2 The present embodiment provides a method for constructing and evaluating a Chinese-English financial large model, including the following steps:

[0090] S1. Collect text descriptions through public channels such as market reports, news articles, financial forums, and social media, and organize the text descriptions and corresponding labels into various types of English financial data. This step covers financial information from different countries and language regions, combining different language descriptions and cultural backgrounds;

[0091] The specific steps for collecting English financial data are as follows:

[0092] S1.1. Clearly define the target and scope of data collection, determine the required data types and sources; for example, for stock market data, historical prices, trading volume, and company financial report information are needed, which can be obtained through public financial APIs, exchange websites, or data providers;

[0093] S1.2. Write scripts using Python to obtain data through API interfaces, and save the collected data as CSV and JSON files to the database;

[0094] S2. Build a bilingual financial instruction data set by processing the collected data;

[0095] After data collection, enter the data processing phase, which ensures the quality and applicability of the data, the specific steps are as follows:

[0096] S2.1. Preliminary inspection and processing of missing values, duplicate values and outliers in the collected data;

[0097] Preliminary inspection of data includes: verifying whether the data types meet the expectations (for example, date fields should be date types, numerical fields should be numerical types);

[0098] Fill in, delete or interpolate missing values in the data through manual correction;

[0099] Remove duplicate values in the data through text matching to avoid interference with the analysis results;

[0100] Deal with outliers in the data through statistical analysis, data standardization or anomaly detection algorithm to ensure the accuracy and integrity of the data;

[0101] S2.2. Clean and convert the processed data, and construct the initial financial data set through feature engineering, i.e. data cleaning and data conversion;

[0102] Data cleaning includes: converting date strings into standard date formats to ensure data consistency and operability; encoding is required for categorical data representing different categories or category labels, such as converting categorical variables to numerical types so that the model can process them; for numerical data such as age, date and income, standardization or normalization is required to eliminate the influence of different scales on the analysis results;

[0103] Data conversion includes: format checking and logical checking of data, removing format outliers and logical error values in data to ensure data accuracy;

[0104] Further, the steps of constructing the Chinese-English bilingual financial instruction data set are as follows:

[0105] S2.3, generate high-quality Chinese-English cross-language translation data set by ChatGPT;

[0106] In order to improve the task processing and cross-language generalization ability of the Chinese-English bilingual financial large model, it is necessary to supplement translation data, which accounts for 4% of the entire data scale.(1) The process of translating English data into Chinese data can increase the amount and diversity of model training data, which helps the model to be robust when processing data in different languages, thereby improving its performance in various financial tasks, such as classification, extraction and prediction;(2) After translating English data into Chinese data, the model can better understand and process financial terms, trends and behaviors related to the Chinese market; In multi-task learning, cross-language knowledge transfer can improve the comprehensive ability of the model, and by translating English data into Chinese data, the model can learn more knowledge on Chinese data and apply these knowledge to other financial tasks, thereby improving the performance of the bilingual model;(3) For tasks that need to process cross-language data, translated data can make the bilingual model better align between languages, which helps to improve the accuracy of the bilingual model in cross-language tasks, such as Chinese-English contrast financial intelligence extraction or multi-language risk assessment;

[0107] The present application takes the stock prediction task as an example, specifically, uses ChatGPT to translate English multi-task data into Chinese, and retains stock information in complex stock classification tasks, manually reviews the translation differences of language structure and professional terms by personnel with financial background, ensures the integrity of the data, and finally removes the Chinese and English text data that is prone to error or cannot be translated through multiple rounds of manual correction and translation quality evaluation;

[0108] For example, Figure 3As shown, in ChatGPT, the embodiment requires to retain detailed information about stocks, stock codes and historical stock prices by using translation prompts, only translate the text of task prompts and stock descriptions, and retain the original time series data of stocks; for stock classification financial tasks with complex data, regular expressions are used to extract the text of task prompts and stock descriptions respectively; specifically, regular expression matching query p_query, matching content p_context and matching tweet p_stock perform specific matching operations in the translation task, which are used to extract questions, contexts and tweet contents respectively, for processing different parts separately to process input text in the translation task; by querying the context in each piece of data to find the division, the time series data in the stock and the tweet are divided, ensuring the complete translation of stock information, and efficiently translating the tweet;

[0109] The matching query p_query matches a line of text that encounters Context: at a certain position in the string and captures all content before Context: (i.e. the question part); specifically, the query part before Context: is extracted and the content after Context: is ignored; the question or query (which may be the content to be translated) is captured, and the part after Context: is not captured, the expression of matching query p_query is as follows:

[0110] p_query = re.compile(r'(.*)\nContext:.*')

[0111] In the formula,'re.compile()' is a function provided by the regular expression module re in Python; the prefix r in 'r'...'indicates a raw string; the round brackets in '(.*)' indicate a capture group, '.' indicates any character, ‘*’ '.' indicates that the preceding character (i.e. any character) can appear 0 times or more, '.*' matches any character sequence from the beginning to the newline character before it; '\n' indicates a newline character; The string 'Context:' is a literal match ;

[0112] The matching content p_context regular expression extracts the content from Context: to Answer: in the text; that is, it matches all characters after Context: until it encounters the Answer: part; the context content (text between Context: and Answer:) is extracted for the context part in the translation task, the specific expression of matching content p_context is as follows:

[0113] p_context = re.compile(r'.*\nContext:([\s\S]*)\nAnswer:')

[0114] In the formula, '.*' matches any character sequence from the beginning to the line break before; '\n' represents a line break; 'Context:' matches the fixed text Context: In '([\s\S]*)', all the characters contained in it are matched; '\s' matches any white space character, including space, tab, line break, etc.; '\S' matches any non-white space character; '[\s\S]' this combination means matching any character (including line break), because \s and \S combined represent all possible characters; '*' indicates that the preceding character is matched 0 times or multiple times. Combined with [\s\S]*, it means matching a string of any length; The '()'parentheses are used For grouping, to save the matched content for later use 'Answer:' matches the fixed text Answer;

[0115] The regular expression matching the tweet p_stock matches the tweet part in the text, followed by two line breaks and followed by a date (format YYYY-MM-DD, where YYYY represents the year; MM represents the month; DD represents the date); extract the tweet content, ignore the date and subsequent content; capture the tweet content, i.e. all text before the date, the specific expression matching the tweet p_stock is as follows:

[0116] p_stock = re.compile(r'([\s\S]*)\n\n\d{4}-\d{2}-\d{2}:[\s\S]*')

[0117] In the formula, 'r"' indicates that this is a raw string, where the backslash \ is not interpreted as an escape character, but treated literally; In '([\s\S]*)', any one of the characters in the parentheses is matched; '\s' matches any white space character, including space, tab, line break, etc.; '\S' matches any non-white space character;'\s\S' combines \s and \S, which can match any character, including white space and non-white space characters; '*' indicates that the character set [\s\S] before it can match zero or more characters ； '\n\n' represents two line breaks; '\d{4}-\d{2}-\d{2}' in'\d' matches a digit (0-9); '\d{4}' means matching 4 consecutive digits, i.e. the year part (such as 2024); '-' matches the connector, i.e. the dash (-) in the date; '\d{2}' means matching 2 digits, used to match the month or date part;'\d{4}-\d{2}-\d{2}' thus matches a date format like 2024-10-11; ':' matches a colon character; '[\s\S]*' and the previous explanation are the same, matching any character (including white space and non-white space characters);

[0118] The text description of the tweet is segmented, and the tweet text is processed by character replacement and space division operations to better translate;

[0119] Specifically, first, the specific characters in the tweet are replaced, a space is added before the dollar sign $ in the tweet to ensure that it can be segmented alone instead of being connected with other words; spaces are added on both sides of the vertical bar | to treat the vertical bar as a separate word; ellipsis... and... are separated from the surrounding content to make the ellipsis an independent word; spaces are added for the colon : and question mark? to ensure that the colon and question mark are processed as separate symbols, and through these replacement operations, the symbols are separated from the words to ensure that the symbols can be segmented independently; finally, the entire string is split by spaces through split(”) to make each word and symbol an independent element in the list, and in this way, the symbols and words in the tweet are separated to facilitate subsequent translation processing, and the specific regular expression is as follows:

[0120] p_stock = re.compile(r'([\s\S]*)\n\n\d{4}-\d{2}-\d{2}:[\s\S]*')

[0121] In the formula, 'r" indicates that this is a raw string, where the backslash \ is not interpreted as an escape character, but is processed literally; in '([\s\S]*)', any character in the parentheses is matched; '\s' matches any white space character, including spaces, tabs, line breaks, etc.; '\S' matches any non-white space character; '\s\S' combines \s and \S and can match any character, including white space and non-white space characters; '*' indicates that the character set [\s\S] before it can match zero or more characters; '\n\n' indicates two line breaks; '\d{4}-\d{2}-\d{2}' in 'd' matches a digit (0-9); '\d{4}' means matching 4 consecutive digits, i.e., the year part (such as 2024); '-' matches the connector, i.e., the dash (-) in the date; '\d{2}' means matching 2 digits, used to match the month or date part; '\d{4}-\d{2}-\d{2}' thus matches a date format like 2024-10-11; ':' matches a colon character. Usually, the date part in the tweet is followed by a colon; '[\s\S]*' and the previous explanation are the same, matching any character (including white space and non-white space characters);

[0122] The extracted tweets are input into the API implementation function of the invoked ChatGPT, combined with the translation prompt "Extract the following financial description into two Chinese sentences that can be accurately understood", with fluent expression and concise language, the word limit is about 400 characters, and the word with a dollar sign must not be translated because it refers to the stock code. After multiple rounds of translation, the translated data is obtained; through multiple rounds of manual correction and evaluation, the data that is prone to error or cannot be translated is removed to generate a high-quality English-Chinese cross-language translation dataset; when screening the translated data, the dataset is strictly reviewed according to multiple standards, and the content that is prone to error or cannot be accurately translated is removed; first, sentences with serious grammar and spelling errors are excluded to avoid errors in translation; second, similar expressions containing idioms, slang, and other phrases that cannot be one-to-one translated in the target language are selected to ensure accurate translation; sentences that lack context or cannot be understood from a single context are also removed, and multi-meaning sentences are reviewed to remove content that may cause confusion after translation; incomplete or obviously missing sentences are removed to ensure that the data has complete semantics; in addition, sentences that rely too much on cultural background or historical allusions are also removed because they lack corresponding expressions in the target language; finally, punctuation and format are correct to avoid translation errors caused by format problems; through strict standards, a high-quality English-Chinese cross-language translation dataset is generated;

[0123] S2.4, according to the differences of different financial tasks (such as shown in Table 1, including sentiment analysis, semantic matching, news classification, negative message judgment, question and answer selection, stock prediction, relationship extraction, title classification, credit evaluation, hawk-dove classification, question and answer, named entity recognition, event detection, text summary) and different data, manually annotate high-quality prompts for different financial tasks;

[0124] Table 1 Detailed information of specific tasks, language types, dataset names and evaluation indicators in the present application

[0125]

[0126] Specifically, the processed data is combined with the prompts designed in S2.4.1, combined with the data structure as shown in Figure 5 , and 20 to 30 unique prompts are generated for each dataset, the purpose is to improve the adaptability and generalization ability of the bilingual model, through this method, more context information can be provided for the bilingual model, making it more flexible and accurate when processing different tasks;

[0127] Each unique prompt (different types of prompts or questions input to the model, which can be requests for specific tasks, scenarios or data, and mainly focus on guiding the model to pay attention to specific financial aspects) can guide the model to pay attention to different aspects of the data, so that the model can better understand and generate relevant output when facing diversified inputs; through diversified prompt design, the model can learn and adapt to various input scenarios more comprehensively, which helps to reduce the model's dependence on specific prompts, while improving the model's robustness and performance in various input scenarios, ultimately improving its practicality and accuracy in actual applications;

[0128] The manual labeling of high-quality prompts specifically includes the following steps:

[0129] S2.4.1, clearly define the goals and needs of each financial task, and write 20 unique prompts for each data set in different financial tasks, using different prompt structures and formats;

[0130] The performance difference of the prompt language guiding the model to pay attention to specific financial aspects in the large model and the data size problem, by analyzing the reply effect presented by different prompt languages, adjusting the expression way of prompt description, selecting more targeted examples. For this purpose, two strategies are adopted, respectively: for English tasks, each data except the two datasets involved in the financial question and answer task of FinQA and ConvFinQA is paired with all the prompts written; for Chinese task data, a prompt is randomly selected for each data to pair.

[0131] As shown in Figure 4 , the initial data set is converted into instruction adjustment samples by reorganizing the prompt, input text and answer;

[0132] S2.4.2, manual evaluation and filtering of financial task prompts;

[0133] Specifically, by manually evaluating and feeding back different types of prompts or questions input to the model, an evaluation team consisting of 10 students proficient in Chinese and English and with a background in the financial field is formed, and the randomly selected LLM test prompts are used on the ChatALL platform (a conversational application integrating various artificial intelligence models and tools, providing intelligent question and answer, information retrieval and multi-field task support) to score each prompt on all indicators and evaluate the response effect; effectively guide the model to pay attention to specific financial field content and ensure the effectiveness of the prompt;

[0134] From the task, write 20 to 30 unique prompts for each dataset. Through accuracy, naturalness, and information quantity indicators (0, 1, 2, 3 points), manually evaluate the corresponding prompts in various financial specific tasks, test different prompt variants, calculate the average value AVG of the three indicators, and retain prompts with AVG scores above 2, filter other prompts;

[0135] S2.5, combined with artificial prompts and financial task descriptions, build financial instruction datasets;

[0136] According to Figure 5 the given JSON format, namely {"Id": [data_id], "Instruction": [prompt], "Input": [text], "Output": [response]}, on the basis of initial data, combined with expert prompts, the framework builds bilingual financial instruction datasets. Financial instruction datasets cover 20 specific tasks and 40 datasets, totaling 604000 data samples;

[0137] The method for building financial instruction datasets includes the following steps:

[0138] S2.5.1, by clearly defining the task type and input and output targets, design instruction data formats suitable for the task type, and define the JSON format of input and output as needed;

[0139] Input section: define all necessary information required for the task, including raw data, question description, context;

[0140] Output section: define the expected result format, usually the model's prediction result, classification label, generated text;

[0141] Assuming the task is to predict stock performance based on company announcements, this task involves multiple input data elements (such as company announcements, market data) and multiple prediction labels (such as good performance, neutral, poor performance), ensure that the output result structure can clearly express each part of the prediction;

[0142] S2.5.2, according to the type of dataset, divide different financial task types according to the ratio of 7:1:2, and build bilingual financial instruction datasets;

[0143] S3, through the constructed bilingual financial instruction datasets, build bilingual financial large models;

[0144] Specifically, by using 70% of the training set and 10% of the validation set of the Chinese-English bilingual financial instruction data set, based on the comparison of the preliminary model evaluation results, the Chinese-English performance superior base model is selected for low-rank adaptive fine-tuning, thereby constructing a Chinese-English bilingual financial large model;

[0145] Further, fine-tuning the instruction data set for the Chinese-English financial large model can enable the model to better understand and perform complex financial tasks, and improve the performance of the model in the Chinese-English financial field; the instruction data set can help the model learn how to generate accurate and clear responses according to user needs, thereby enhancing the practicality and reliability of the model in actual application; the fine-tuned model can not only handle more complex financial problems, but also overcome language barriers to provide high-quality financial analysis and prediction for Chinese and English users;

[0146] The specific steps of constructing the Chinese-English bilingual financial large model are as follows:

[0147] S3.1, select the base model with the best performance;

[0148] As shown in Figure 6 , by evaluating the bilingual performance of the five backbone large models (Bloomz-7B1, ChatGLM3-6B, FinMA-7B, DISCFin-13B), InternLM-7B is selected as the bilingual financial base model;

[0149] The divided evaluation data set is used to preliminarily evaluate the selected general model Bloomz-7B1, ChatGLM3-6B and the financial model FinMA-7B, DISCFin-13B, and the performance of the two types of models in classification, extraction and reasoning tasks is compared; in order to ensure fair evaluation, the experiment is carried out under the same hardware environment and the same hyperparameter setting, and the selected model is compared in different data sets and tasks to select the most suitable base model for the tasks related to the present application, and the next step of model fine-tuning is carried out to improve the effect;

[0150] S3.2, fine-tune the model by using the divided training set, optimize the model parameters to adapt to specific tasks, which includes the following steps:

[0151] S3.2.1, by introducing the four types (including classification, extraction, translation, English data) of data sets in the fine-tuning process in turn, create four variants of the ICE-INTENT model to evaluate the influence of various data types on the fine-tuned bilingual model;

[0152] According to the task requirements, the data set is divided into four different types (classification, extraction, translation, English data), and these data sets have different data formats and label types;

[0153] In the fine-tuning process, the four types of data sets are introduced to create four variants of bilingual models; first, fine-tune from the classification data set, and then introduce the extraction, translation, and English data sets one by one each time, observe the changes in the evaluation indicators on the model after the integration of each data set, and create four model variants according to different data set combinations, including: the model ICEdIc-7B fine-tuned only with classification task data set, the model ICEdIe-7B fine-tuned with classification + extraction task data set, the model ICEdtt-7B fine-tuned with classification + extraction + translation data set, and the model ICEfull-7B fine-tuned with all four types of data sets;

[0154] S3.2.2, optimize memory usage and reduce training time by introducing advanced fine-tuning technology (QLora); only the necessary low-rank matrix needs to be updated, while the use of quantization reduces the storage and calculation requirements of the model, making large-scale model fine-tuning more efficient and convenient;

[0155] In this embodiment, by loading the scholar·Pu language (InternLM) base model into the environment, all weights of the model are frozen immediately after loading to ensure that the weights will not be updated during fine-tuning; then insert low-rank decomposition matrices in the attention layer and fully connected layer in the self-attention (Transformer) architecture of each layer; then quantize the weights in the model and convert them to 4-bit representation;

[0156] S3.2.3, fine-tune the model with QLora technology on a platform containing 8 NVIDIA HGXA100 sxm 4 GPUs with a batch size of 24;

[0157] By uniformly cutting the training sequence (text data input into the model for training) into a length of 2048 tokens, using the AdamW optimizer, the initial learning rate is 5e-5 and the weight decay is 1e-5 for all parameters; In addition, the warm-up step (the process of gradually increasing the learning rate in the early stage of training) is set to 1% of the total training steps (total number of training iterations), and in the first 1% of the training steps, the learning rate will gradually increase from 0 to the set initial learning rate;

[0158] The training sequence refers to the text data input to the model for training. In natural language processing (NLP) tasks, the training sequence usually refers to the pre-processed text data being cut into fixed-length segments (2048 tokens) for batch processing in the model. The warm-up step is a strategy to gradually increase the learning rate in the first few steps of training. The learning rate starts from 0 and gradually rises to the initial learning rate set (such as 5e-5). This can prevent the model from converging unstably in the early stages of training due to a high learning rate, thereby increasing the stability of the model. The total training steps refer to the total number of steps required in the entire training process. In deep learning, training is usually completed through multiple iterations (epochs), and each epoch traverses the entire training data set. By setting the warm-up step to 1% of the total training steps, it can be ensured that the learning rate is gradually increased in the initial stage of training.

[0159] During training, the training task is divided into 24 batches for GPU parallel processing, while monitoring memory usage and training speed. By merging the pre-trained weights of the base model InternLM with the low-rank matrix, the specific steps are as follows: perform rank decomposition on the base model weights to obtain low-rank matrices A and B, then update the weight matrix in each layer of the Transformer as follows:

[0160] W' = W + (A x B)

[0161] In the formula, W is the original pre-trained weight matrix of the base model, which contains the knowledge and parameters learned by the model during training; A is part of the low-rank matrix, usually a smaller matrix that will be updated during training to adapt to the needs of a specific task; B is another low-rank matrix related to A, also usually a smaller matrix that matches the dimensions of A, used to reduce the number of model parameters; A x B represents the product of low-rank matrices, generating a new matrix with the same dimensions as the original weight matrix W, representing new parameters added to the original weights after low-rank decomposition adjustment; W' represents the final model weight matrix after merging

[0162] After training is complete, the final model's weights are composed of the original weights and the low-rank matrix, preserving the knowledge of the base model and achieving optimization for specific tasks through low-rank matrix adjustment;

[0163] S3.3, by observing the changes in the loss function (such as training loss and validation loss), accuracy (such as training accuracy and validation accuracy), F1 score, precision, recall, and area under the curve (AUC-ROC) during the process, to ensure effective learning of the model;

[0164] S3.4. Adjust hyperparameters to avoid overfitting by evaluating model performance using the test set at the end of each training epoch;

[0165] By evaluating the model performance using the test set at the end of each training epoch, the model's performance can be monitored in real-time, ensuring that the model is optimized towards the desired goal, and any overfitting issues that may arise during training can be detected and addressed effectively. Based on the evaluation results of the test set, hyperparameters such as learning rate and weight decay can be optimized to improve the model's generalization ability and avoid situations where the model performs well on the training set but poorly in real-world applications. This dynamic evaluation mechanism (by monitoring the performance of the validation set during training, we can determine whether the model is overfitting. If the performance of the validation set does not improve within a certain number of training epochs, training can be stopped early to avoid overfitting. In addition, the learning rate can be dynamically adjusted during training to optimize the convergence speed and generalization ability of the model) makes the training process more flexible, allowing the model to be optimized at different stages and ultimately resulting in a more robust and accurate model;

[0166] S4. Evaluate the model performance using the 20% instruction test set, and develop different evaluation indicators for different tasks. Each indicator helps to comprehensively analyze the model's performance;

[0167] The evaluation benchmark is based on ten key financial task types, which can be used to classify and evaluate large language models (LLM) to ensure that their performance in various financial application scenarios is comprehensively tested. These tasks are classified according to content as follows: sentiment analysis, semantic matching, news classification, negative judgment, answer selection, stock prediction, relationship extraction, title classification, credit classification, and hawk-pigeon classification;

[0168] As shown in Table 1, each task uses accuracy, F1 score, and Matthews correlation coefficient (MCC) to comprehensively measure the model's performance. In addition to general tasks, specific tasks also incorporate specific evaluation criteria, such as EMAccuracy used in question-answering tasks and evaluation indicators such as Rouge and BERTScore used in text summarization. Through these evaluations, we can gain a deeper understanding of the performance, strengths, and limitations of LLMs when handling different financial tasks.

[0169] Through the above classification and evaluation methods, the training and application strategies of LLMs can be optimized to ensure that the models have stronger practicality and accuracy in specific tasks in the financial field;

[0170] As shown in Table 1, the data set contained in the evaluation benchmark disclosed by the application is divided into two parts: data for model training and data different from model training; by comparing the evaluation results on the two different data sets, the ability of the model is comprehensively evaluated;

[0171] By Figure 5 The data set format in the unified evaluation benchmark is tested on the variants ICEdlc, ICEdle, ICEdtt and ICEfull of the fine-tuned different English-Chinese bilingual model ICE-INTERN, the performances of different variants in various tasks are compared, and the evaluation results of Chinese, English and English-Chinese on different models are shown in Tables 2, 3 and 4 (bold is the optimal indicator, and underlined is the best indicator except GPT), the performance of the series of variants of the application in Chinese tasks is better than all baseline large models, and even exceeds large LLMs such as GPT-4; in bilingual tasks, the average performance of ICE-INTERN is slightly worse than GPT-4, but exceeds all other LLMs, and the bilingual model performs well in classification CLS and prediction PRE bilingual tasks, exceeding all models;

[0172] However, the performance of the bilingual model in reasoning REA and untrained domain DOT tasks is not outstanding, and sometimes even weaker than FinMA-7B and Bloomz-7B1 and other LLMs such as ICEdle-7B and ICEdtt-7B, which are not fine-tuned using English data sets, so their performance is slightly lower than GPT-4, and the reason is that they have more parameters and focus on English, among the evaluation indicators of 63 specific tasks, compared with other models, the ICE-INTERN variants achieve the best performance in 68.25% (excluding ChatGPT and GPT-4) and 58.73% indicators, and as for the overall best result, the application after comprehensive fine-tuning is better than all baseline models, especially better than ChatGPT, and even exceeds GPT-4.

[0173] Table 2: Evaluation results of different types of Chinese data test sets on different models

[0174] Model / Average ZH-CLS ZH-PRE ZH-EXT ZH-GEN ZH-TRA ZH.Avg InternLM-7B 0.277 0.003 0.001 -1.119 0.272 -0.113 Bloomz-7B1 0.170 0.131 0.038 -1.650 0.240 -0.214 ChatGLM3-6B 0.391 0.220 0.013 -1.031 0.311 -0.019 GPT-4 0.549 0.383 0.098 -1.577 0.266 -0.056 FinMA-7B 0.229 0.000 0.016 -2.009 0.250 -0.303 DISCFin-13B 0.290 0.315 0.022 -1.858 0.290 -0.188 ICEdlc-7B 0.598 0.340 0.020 -1.762 0.337 -0.093 ICEdle-7B 0.607 0.637 0.569 -1.024 0.327 0.223 ICEdtt-7B 0.607 0.630 0.566 -1.024 0.448 0.245 ICEfull-7B 0.631 0.617 0.567 -0.972 0.409 0.250

[0175] Table 3: Evaluation results of different types of English data test sets on different models

[0176] Model / Average EN-CLS EN-PRE EN-EXT EN-REA EN-DOT EN.Avg InternLM-7B 0.301 0.236 0.000 0.000 -1.251 -0.143 Bloomz-7B1 0.406 0.238 0.000 0.005 -1.238 -0.118 ChatGLM3-6B 0.434 0.262 0.248 0.000 -0.833 0.022 GPT-4 0.618 0.283 0.830 0.695 -0.640 0.357 FinMA-7B 0.371 0.233 0.392 0.000 -1.187 -0.038 DISCFin-13B 0.446 0.236 0.039 0.002 -1.158 -0.087 ICEdlc-7B 0.331 0.282 0.000 0.000 -1.227 -0.123 ICEdle-7B 0.334 0.275 0.006 0.000 -1.203 -0.118 ICEdtt-7B 0.357 0.268 0.000 0.000 -1.182 -0.111 ICEfull-7B 0.494 0.250 0.362 0.000 -1.193 -0.017

[0177] Table 4: Evaluation results of different types of data test sets on different models

[0178]

[0179]

[0180] Embodiment Two

[0181] In detail Figure 7 , based on the same inventive concept, a bilingual financial large model construction and evaluation system, comprising:

[0182] (1) A financial original data collection module 201, said financial data collection module is used to extract relevant content from financial news websites, market analysis reports and financial forums. Each data needs to contain clear market analysis instructions, and the corresponding response is extracted from the actual market analysis article to obtain the preliminary data to be processed;

[0183] (2) A financial instruction data set construction module 202, said financial instruction data set construction module, specifically, according to the instruction data set format required by the framework and the cross-language data requirements of the invention, through ChatGPT translation query question and answer, stock classification, sentiment analysis and other task-heavy data set construction translation cross-language Chinese data set, 20-30 high-quality prompts Prompt written by financial experts are used for artificial evaluation and correction, combined with financial task description based on Figure 4 JSON format standard in the financial instruction data set.

[0184] (3) A financial large model fine-tuning module 203, said financial large model fine-tuning module, refers to using financial instruction fine-tuning data set to construct bilingual financial large model through QLora technology fine-tuning. By observing the loss function, accuracy and other indicators in the model fine-tuning process, the effective learning of the model is ensured.

[0185] (4) A financial evaluation benchmark evaluation module 204, said financial evaluation benchmark evaluation module, refers to using 20% of the independent test set in the constructed financial instruction data set to evaluate the model performance, and different evaluation indexes are formulated for different financial tasks. Each index can help analyze the model performance comprehensively, and a comprehensive evaluation benchmark is constructed.

[0186] The system introduced in Embodiment Two of the present application is a system for the method of constructing and evaluating a bilingual financial large model in Embodiment One of the present application. Therefore, based on the method introduced in Embodiment One of the present application, those skilled in the art can understand the specific structure and variations of the system, and thus this place will not be repeated. Any system used in the method of Embodiment One of the present application belongs to the scope of protection of the present application.

[0187] Embodiment Three

[0188] Please see Figure 8, based on the same inventive concept, the present application discloses a system for constructing and evaluating a bilingual financial large model, namely a computer readable storage medium 300, which stores a computer program 311, and the program is executed to realize the method as described in embodiment one.

[0189] Since the computer readable storage medium introduced in embodiment three is the computer readable storage medium used for implementing the method for constructing and evaluating a bilingual financial large model in embodiment one, the specific structure and deformation of the computer readable storage medium can be understood by those skilled in the art based on the method introduced in embodiment one, and thus will not be described here. Any computer readable storage medium used in the method in embodiment one belongs to the scope of the present application.

[0190] Embodiment four

[0191] Based on the same inventive concept, the present application discloses a system for constructing and evaluating a bilingual financial large model, namely a computer device, please see Figure 9 , including storage 401, processor 402 and computer program 403 stored on the storage and executable on the processor, and the processor 402 executes the above-mentioned program to realize the method in embodiment one.

[0192] Since the computer device introduced in embodiment four is the computer device used for implementing the system for constructing and evaluating a bilingual financial large model in embodiment one, the specific structure and deformation of the computer device can be understood by those skilled in the art based on the method introduced in embodiment one, and thus will not be described here. Any computer device used in the method in embodiment one belongs to the scope of the present application.

[0193] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk storage, CD-ROM, optical storage, etc.).

[0194] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions of the flowchart block(s) or step(s) of the flowchart block(s). Figure 1 one or more functions specified by any one or more of the flowchart block(s) or step(s) Figure 1 one or more functions specified by any one or more of the flowchart block(s) or step(s)

[0195] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments described and shown, and it is therefore intended that the application cover any and all variations of the preferred embodiments which fall within the scope of the present application. Accordingly, the appended claims are intended to cover all such modifications and variations as falling within the scope of the application.

[0196] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method for constructing and evaluating large-scale financial models in both Chinese and English, characterized in that, Includes the following steps: S1. Collect raw financial data in Chinese and English, including: financial market reports, news articles, financial forums and public channels on social media; S2. By processing the collected data, construct the Chinese-English bilingual financial instruction dataset ICE-FIND containing translated data and divide the dataset into training set, validation set and test set in a ratio of 7:1:

2. S3. By fine-tuning 80% of the training set in the constructed financial instruction dataset based on the selected bilingual Chinese-English base model with superior performance, a large-scale Chinese-English bilingual financial model ICE-INTENT is constructed. The steps include: S3.1 Select InternLM-7B as the base model; S3.2 Fine-tuning the model using a pre-defined training set to optimize model parameters for a specific task includes the following steps: S3.2.

1. Four variants of the ICE-INTENT model were created by introducing four types of datasets during the fine-tuning process to evaluate the impact of various data types on the fine-tuning of the bilingual model. The four types of datasets include: classification datasets, extraction datasets, translation datasets, and English data datasets; The four model variants include: ICEdlc-7B, which is fine-tuned using only the classification task dataset; ICEdle-7B, which is fine-tuned using the classification + extraction task dataset; ICEdtt-7B, which is fine-tuned using the classification + extraction + translation dataset; and ICEfull-7B, which is fine-tuned using all four class datasets. S3.2.2 By introducing LoRa fine-tuning technology, memory usage is optimized and training time is reduced; only the necessary low-rank matrix needs to be updated, while quantization is used to reduce the storage and computation requirements of the model, making large-scale model fine-tuning more efficient and convenient. S3.2.3 Fine-tune the model with a batch size of 24 on a platform containing 8 NVIDIA HGX A100 sxm 4 GPUs; During fine-tuning, the training sequence is uniformly divided into 2048 token lengths, and the AdamW optimizer is used with an initial learning rate of 5e for all parameters. -5 The weight decays to 1e -5 ; During the fine-tuning process, the pre-trained weights of the base model Shusheng·Puyu InternLM-7B are merged with the low-rank matrix. The final model weights are composed of the original weights and the low-rank matrix to retain the knowledge of the base model. The optimization for specific tasks is achieved by adjusting the low-rank matrix. S3.

3. By observing the changes in the loss function, accuracy, F1 score, precision, recall, and area under the curve during the fine-tuning process, we can ensure the effective learning of the model. S3.

4. Evaluate model performance using a test set at the end of each training round to adjust hyperparameters and avoid overfitting; S4. Evaluate the model performance using 20% ​​of the independent test set in the constructed financial instruction dataset, formulate different evaluation indicators for different financial tasks, and build a comprehensive evaluation benchmark ICE-FLARE.

2. The method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English as described in claim 1, characterized in that, The steps for collecting original financial data in both Chinese and English are as follows: S1.

1. Clearly define the goals and scope of data collection, and determine the required data types and sources; S1.2 Use Python to write scripts to obtain data through API interfaces and save the collected data as CSV and JSON files to the database.

3. The method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English as described in claim 1, characterized in that, The steps for processing the collected data to construct a bilingual (Chinese and English) financial instruction dataset and then dividing the dataset are as follows: S2.1 Perform a preliminary check on the collected data and process missing, duplicate, and outlier values. S2.2 Clean and transform the processed data, and construct the initial dataset through feature engineering; S2.

3. Generate a high-quality English-Chinese cross-language translation dataset from the initial dataset using ChatGPT; S2.

4. Manually annotate high-quality prompts for different financial tasks based on the differences in data within the financial tasks; S2.

5. Combine high-quality manually labeled prompts and financial task descriptions to construct a financial instruction dataset.

4. The method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English as described in claim 3, characterized in that, The manual annotation of high-quality prompts specifically includes the following steps: S2.4.

1. Define the goals and requirements of each financial task, and write 20 to 30 unique hints for each dataset in different financial tasks, using different hint structures and formats; S2.4.

2. The quality of the prompts is manually evaluated and feedback is provided multiple times to ensure their effectiveness, i.e., to output the target answer.

5. The method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English as described in claim 3, characterized in that, The steps for constructing the financial instruction dataset are as follows: S2.5.1 By clearly defining the task type and input / output targets, design an instruction data format suitable for the task type, and define the JSON format for input and output as needed; Input section: Defines all the necessary information required for the task, including raw data, problem description, and context; Output section: Defines the expected result format, which is usually the model's prediction results, classification labels, and generated text; S2.5.2 Based on the type of dataset, divide the different financial task types according to a ratio of 7:1:2 and construct a Chinese-English bilingual financial instruction dataset.

6. A system for constructing and evaluating large-scale financial models in both Chinese and English, characterized in that: The aforementioned system for constructing and evaluating large-scale financial models in both Chinese and English is used to implement the method for constructing and evaluating large-scale financial models in both Chinese and English as described in any one of claims 1 to 5. The aforementioned system for constructing and evaluating large-scale financial models in both Chinese and English includes: (1) Financial data collection module, used to extract relevant content from financial market reports, news articles, financial forums and public social media channels; (2) Financial instruction dataset construction module, which is used to process the collected data to construct the Chinese-English bilingual financial instruction dataset ICE-FIND containing translated data and to divide the dataset; (3) Financial large model fine-tuning module, used to fine-tune 80% of the training set in the constructed financial instruction dataset based on the selected bilingual Chinese-English base model with superior performance to construct the Chinese-English bilingual financial large model ICE-INTENT; (4) Financial evaluation benchmark assessment module, which is used to evaluate the model performance using 20% ​​of the independent test set in the constructed financial instruction dataset, formulate different evaluation indicators for different financial tasks, and build a comprehensive evaluation benchmark ICE-FLARE.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed, the program implements a method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English, as described in any one of claims 1 to 5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a method for constructing and evaluating a large-scale financial model oriented towards both Chinese and English as described in any one of claims 1 to 5.

Images