Financial financing planning scheme providing system based on large language model

By building a database and vertical language model in the financial field, combined with retrieval and enhanced generation components, the problem of processing complexity and diversity of financial data is solved, real-time and accurate financial financing planning schemes are realized, meeting the diverse needs of users, and reducing the Financing risk.

CN119046416AActive Publication Date: 2024-11-29CHENGDU JIAOZI SHUQI TECHNOLOGY CO LTD

Patent Information

Application Number
CN202410992103.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-29
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

Existing financial software based on large language models is difficult to process the complexity and diversity of financial data, which leads to huge challenges in processing and integrating data, and is unable to provide real-time and accurate financial financing planning solutions, which affects the timeliness of decision-making and accuracy.

Method used

Build a database in the financial field, fine-tune the large language model through data cleaning and low-order adaptive technology, build a vertical language model, and design and search enhancement generation components, simulate user dialogue to generate accurate financial financing planning solutions, and continuously optimize model performance.

Benefits of technology

It realizes personalized user settings, provides accurate and efficient financing planning solutions, meets diversified needs, and avoids the high risks of a single financing method through diversified channels, which improves the model's performance in processing financial data and user information extraction and planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046416A_ABST
    Figure CN119046416A_ABST
Patent Text Reader

Abstract

The invention provides a financial financing planning scheme providing system based on a large language model, and the system improves the accuracy and efficiency of processing financial planning problems by constructing a training data set in the financial field, finely adjusting a basic large language model and designing a retrieval enhancement generation component. According to the method, fine tuning training is carried out on a basic large language model by adopting a low-order adaptive technology, and a model weight special for the financial field is provided for a system. According to the method, a retrieval enhancement generation component is designed, an existing knowledge document of a financial platform is segmented, encoded and stored in a vector knowledge base, after a user initiates a demand, a system calls the retrieval enhancement generation component to obtain related knowledge base content, and an accurate financing planning suggestion is generated in combination with the input demand. A data set, a large language model and a retrieval component are continuously optimized by designing a benchmark index. According to the system, a real-time and accurate financing planning scheme can be provided when complex problems in the financial field are processed, and the user experience and satisfaction are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and financial technology, and in particular to a financial financing planning solution providing system based on a large language model. Background Art

[0002] Fintech covers a series of technological innovations such as big data, cloud computing, artificial intelligence and blockchain, which has greatly promoted the development of the financial industry. In recent years, with the rapid development of natural language processing technology, more and more research has chosen to apply advanced natural language processing technology in the field of financial technology. In the field of financial technology, tasks that use natural language processing include sentiment analysis, named entity recognition, news classification and question-answering systems, etc. Although the processing scope of these tasks is similar to that of general large language models, if general large language models are used to process tasks in the financial field, there are often the following problems:

[0003] First, the complexity and diversity of financial data place extremely high demands on models. Financial data includes both structured data, such as financial statements and transaction records, and unstructured data, such as news reports and market comments. There is a high degree of heterogeneity between these data sources, which makes the model face huge challenges in processing and integrating data. Secondly, the financial field has extremely high requirements for the real-time and accuracy of data. Traditional natural language processing models are currently difficult to process and update large-scale data in a short period of time, which affects the timeliness and accuracy of decision-making. In addition, as the amount of data continues to increase, traditional models will encounter performance bottlenecks when processing and analyzing data and cannot provide a fast enough response time. For example, in the actual application scenario of financial financing, it is necessary to combine the various actual complex situations and financing needs of customers to quickly and accurately provide financial financing planning solutions for customers to choose. However, among the existing financial software based on large language models, there is no software system specifically used to provide financial financing planning solutions. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention proposes a financial financing planning solution providing system based on a large language model to solve the deficiencies in the prior art.

[0005] The technical solution adopted by the present invention is as follows:

[0006] In a first aspect, a method for constructing a large language model for providing a financial financing planning solution is provided, comprising the following steps:

[0007] Build a financial database based on financial knowledge and financing planning schemes; clean the data in the financial database to obtain a training data set containing representative questions and accurate answers;

[0008] The training dataset is used to fine-tune the basic large language model using low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes;

[0009] Construct a retrieval enhancement generation component to segment the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in a vector database;

[0010] Simulate the user to initiate a conversation, understand the semantics of the user's input questions, identify the key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning solution suitable for the user;

[0011] The generated financial financing planning scheme is evaluated according to benchmark indicators, and the system is iterated based on the evaluation results to continuously optimize the training data set, vertical large language model, and retrieval enhancement generation components.

[0012] Furthermore, a financial database is constructed based on the knowledge and financing planning schemes in the financial field. The data in the financial database is cleaned to obtain a training data set containing representative questions and accurate answers, including:

[0013] When building a database in the financial field, text data in the financial field is collected from different sources, including financial news, academic papers, financing planning schemes, and open regulatory documents;

[0014] After data cleaning of the data in the financial database, natural language processing tools are used to perform word segmentation, part-of-speech tagging and named entity recognition on the text data; the text data is unified in data format, and financing planning schemes corresponding to the text data are generated one by one, which are organized into user information-financing planning scheme pairs, and the user information-financing planning scheme pairs are used as a data set of financial question and answer pairs and stored in a relational database.

[0015] Furthermore, the basic large language model is fine-tuned using the training data set and low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes, including:

[0016] Load the training data set to train the basic large language model. In the process, introduce low-rank decomposable random Gaussian distribution matrix and all-zero matrix to align the pre-trained weights.

[0017] Update the low-rank decomposition matrix parameters and modify the forward transfer matrix to be the sum of the original pre-trained matrix and the updated low-rank decomposition matrix;

[0018] The loss function is used to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively.

[0019] Furthermore, the training data set is loaded to train the basic large language model. In the process, a low-rank decomposable random Gaussian distribution matrix and an all-zero matrix are introduced to align the pre-trained weights, including: introducing two trainable low-rank decomposition matrices in the Transformer architecture of each layer of the basic large language model and appending them to the pre-trained weight matrix, while keeping the original weight matrix of the large language model unchanged, which is only used for basic operations in the entire training process;

[0020] Update the low-rank decomposition matrix parameters and modify the forward transfer matrix to be the sum of the original pre-trained matrix and the updated low-rank decomposition matrix, including: initializing one low-rank decomposition matrix as a random Gaussian distribution and the other low-rank decomposition matrix as an all-zero matrix; in the process of forward propagation, calculate the weight values ​​of the two low-rank decomposition matrices respectively, perform dot multiplication on the weights of the two, add the result to the original weight as the parameter of forward propagation, and use the stochastic gradient descent optimization method to update the parameters.

[0021] Furthermore, a retrieval enhancement generation component is constructed to segment the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in a vector database, including:

[0022] Export the existing knowledge documents of the financial platform, including financial products, financial policies and financing planning schemes;

[0023] Clean and extract knowledge document data in various formats, convert them into a unified plain text format and split them into blocks;

[0024] Use knowledge graphs to construct plain text formats into hierarchical document structures;

[0025] An embedding model is used to encode the hierarchical text structure into block vector representations and stored in a vector database.

[0026] Furthermore, the system simulates the user initiating a conversation, performs semantic understanding on the questions input by the user, identifies key information in the semantics, calls the search enhancement generation component to obtain the knowledge base content, combines it with the original question and inputs it into the vertical large language model to generate an accurate financial financing planning solution suitable for the user; including:

[0027] Get the dialogue input by the user and use the embedding model to convert the query dialogue into a dialogue vector representation;

[0028] Calculate the similarity between the dialogue vector and the block vectors in the vector database in sequence, and output the first K blocks as prompts;

[0029] The retrieved prompts are concatenated with the input dialogue to obtain a coherent prompt, which is then input into the vertical large language model to generate an accurate financial financing planning solution suitable for the user, and the output result is displayed to the user.

[0030] Furthermore, the similarity between the dialogue vector and the block vector in the vector database is calculated, including:

[0031] Get the fixed-dimensional conversation query embedding and calculate the cosine similarity between the query embedding and the document block embedding at each level;

[0032] According to the calculated cosine similarity, all document block embeddings are sorted, and the K document blocks most relevant to the query are selected from high to low as the retrieval results.

[0033] The document blocks in the retrieval results will be used as the input of the generative model to expand the semantic context and assist the large language model to generate the final answer.

[0034] Furthermore, the generated financial financing planning scheme is evaluated according to the benchmark indicators, and the system is iterated according to the evaluation results to continuously optimize the training data set, vertical large language model, and retrieval enhancement generation components, including:

[0035] Define reading comprehension as the model performance indicator, calculated based on the correctness of the financial information contained in the model output text;

[0036] Continuously collect user questions and system responses and pass them into the training dataset as user information-financing plan pairs;

[0037] Iterative training is performed based on new training data sets to continuously optimize the overall performance of the large language model used to provide financial financing planning solutions.

[0038] In a second aspect, a large language model for providing financial financing planning solutions is provided, which is constructed using the method for constructing a large language model for providing financial financing planning solutions provided in the first aspect.

[0039] In a third aspect, a system for providing a financial financing planning solution based on a large language model is provided. The system uses the large language model for providing a financial financing planning solution provided in the second aspect to provide a financial financing planning solution for a user. The system includes:

[0040] The training data set construction module is used to construct a financial database based on financial knowledge and financing planning schemes. It is also used to clean the data in the financial database to obtain a training data set containing representative questions and accurate answers.

[0041] The vertical large language model training and generation module is used to fine-tune the large language model based on the basic large language model using the training data set and the low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes; it is also used to simulate users initiating conversations, understand the semantics of the questions input by users, identify key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning scheme suitable for users;

[0042] The retrieval enhancement generation component building module is used to build the retrieval enhancement generation component, divide the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in the vector database.

[0043] It can be seen from the above technical solution that the beneficial technical effects of the present invention are as follows:

[0044] 1. Users can make personalized settings according to their specific needs, provide users with accurate and efficient financing planning solutions to meet their diverse needs, and avoid the high risks brought to users by a single financing method by providing diversified financing channels.

[0045] 2. Use low-order adaptive technology to fine-tune the large language model. During the training process, freeze the weights of the pre-trained model and introduce a trainable low-rank decomposition matrix to update its parameters; modify the forward pass matrix to the sum of the original pre-trained matrix and the updated low-rank decomposition matrix. Use the loss function to iteratively train the basic semantic segmentation network and update the gradient until the network converges. The application of low-order adaptive technology not only improves the training efficiency of the model, but also enhances the performance of the model in processing financial data and extracting and planning user information.

[0046] 3. Design and implement a retrieval enhancement generation component to enable the model to retrieve relevant information from an external database and combine the retrieved information with the user-entered question as the input of the large language model. Rich contextual information can enhance the accuracy and credibility of the large language model's answers. In addition, for specific financial scenarios, the retrieval enhancement generation block can enhance the generation capabilities of the large language model by integrating knowledge from external databases, and allows continuous updating of knowledge and integration of domain-specific information, thereby overcoming problems such as outdated knowledge and illogical output of the basic large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the prior art description. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0048] Figure 1 Provide a system flow chart for the financial financing planning scheme based on the large language model according to the embodiment of the present invention;

[0049] Figure 2 This is an example diagram of a user information-financing planning scheme pair according to an embodiment of the present invention;

[0050] Figure 3 An overview diagram of the low-level adaptive technology training structure of an embodiment of the present invention;

[0051] Figure 4 This is an example diagram of the operation of the search enhancement generation component according to an embodiment of the present invention;

[0052] Figure 5 A system experimental visualization dialogue comparison example diagram is provided for the original Tongyi Qianwen 14B model and the financial financing planning scheme of the embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0054] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0055] Example

[0056] For the application scenario of providing customers with financial financing planning solutions, the software system based on the large language model not only needs to have basic question-and-answer capabilities, but also needs to process multiple rounds of conversations and generate detailed financial financing planning reports and solutions to ensure high accuracy, simplify the user financing process, and improve financing efficiency.

[0057] In view of the above requirements, if Figure 1 As shown, this embodiment discloses a large language model for providing a financial financing planning solution, and the method for constructing the model includes the following steps:

[0058] Step S1: Construct a financial database based on financial knowledge and financing planning schemes; clean the data in the financial database to obtain a training data set containing representative questions and accurate answers.

[0059] In this embodiment, the specific implementation of step S1 is as follows:

[0060] Step S11: collect text data in the financial field from different sources to build a database in the financial field. The text data includes financial news, academic papers, financing planning schemes and open regulatory documents.

[0061] Step S12: preprocess the text data. The preprocessing mainly includes data cleaning, including removing duplicate, irrelevant and noise data, and using natural language processing tools to perform word segmentation, part-of-speech tagging and named entity recognition on the text data. The implementation form of the natural language processing tool is not limited and can be implemented in any achievable manner in the prior art.

[0062] Step S13, unify the data format of the preprocessed text data, generate financing planning schemes corresponding to the text data one by one, organize them into user information-financing planning scheme pairs, and store the user information-financing planning scheme pairs as a data set of financial question and answer pairs in a relational database.

[0063] like Figure 2 As shown, Figure 2 An example of a user information-financing planning scheme pair.

[0064] This step collects text data in the financial field from different sources, uses natural language processing tools to preprocess the text data and unify the data format, generates questions corresponding to the text, organizes them into user information-financing planning scheme pairs and stores them in a relational database. By building a database in the financial field, it is ensured that the data set contains representative financing planning schemes, so that the system can learn and extract financing planning templates and related content from a large amount of high-quality data, providing semantic context information for subsequent model training.

[0065] Step S2: Select a basic large language model suitable for the financial field, use the training data set to fine-tune the large language model using low-order adaptive technology, and obtain a vertical large language model suitable for the task of formulating financial financing planning solutions.

[0066] In this embodiment, the specific implementation of step S2 is as follows:

[0067] Step S21: Select the current open source model Tongyi Qianwen 14B as the basic large language model

[0068] The Tongyi Qianwen 14B model is open sourced by Alibaba Cloud. It is a high-performance open source model that supports multiple languages. The overall training data exceeds 3 trillion tokens and has strong reasoning, cognition, planning, and memory capabilities. Initialize a pre-trained Tongyi Qianwen 14B large language dialogue model, and use SFT (supervised fine-tuning) technology and RLHF (reinforcement learning based on human feedback) technology to achieve alignment, and use the obtained basic large language model as the initial model.

[0069] Step S22: Load the training data set obtained in step S1 to train the basic large language model. In the process, introduce a low-rank decomposable random Gaussian distribution matrix and an all-zero matrix to align the pre-trained weights.

[0070] Specifically, two trainable low-rank decomposition matrices are introduced into the Transformer architecture of each layer of the basic large language model and attached to the pre-trained weight matrix to achieve weight update; at the same time, the original weight matrix of the large language model is kept unchanged and is only used for basic operations during the entire training process. In this embodiment, the introduced trainable matrices M and N can be expressed as:

[0071] M~η(0,σ 2 )

[0072] N=0

[0073] Where M represents a random Gaussian distribution; η represents a normal distribution (also called a Gaussian distribution); parameters 0 and σ 2 represent mean and variance respectively; N represents an all-zero matrix.

[0074] Step S23: Update the low-rank decomposition matrix parameters and modify the forward transfer matrix to be the sum of the original pre-trained matrix and the updated low-rank decomposition matrix.

[0075] Specifically, one low-rank decomposition matrix is ​​initialized as a random Gaussian distribution, and the other low-rank decomposition matrix is ​​initialized as an all-zero matrix; in the process of forward propagation, the weight values ​​of the two low-rank decomposition matrices are calculated respectively, and the weights of the two are multiplied by a dot multiplication operation. The result is added to the original weight as the parameter of the forward propagation, and the parameters are updated using the stochastic gradient descent optimization method. In this embodiment, the forward propagation calculation process can be expressed as:

[0076] Y=XW+XΔW * =XW+XMN T

[0077] Where Y represents the output vector of forward propagation; X represents the pre-trained weight matrix, whose dimension is d×k;

[0078] W represents the input vector, whose dimension is k; ΔW * Represents the weight update matrix, which is calculated by the low-rank decomposition matrices M and N.

[0079] During training, W is frozen and does not receive gradient updates, while the low-rank factorization matrices M and N contain trainable parameters.

[0080] Step S24: Use the loss function to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively

[0081] Specifically, the AdamW optimization algorithm (an improved adaptive learning rate optimization algorithm) and the cosine annealing strategy are used to update weights and adjust learning rates. Through multiple rounds of training and optimization, the model parameters and hyperparameters are continuously adjusted until the network converges, thus obtaining a vertical large language model suitable for processing tasks in the financial field.

[0082] like Figure 3 As shown, Figure 3 An overview of the structure of introducing trainable low-rank decomposition matrices for training in the Transformer architecture of each layer in a large language model is shown.

[0083] In this step, the large language model is fine-tuned using low-order adaptive techniques. During the training process, the weights of the pre-trained model are frozen and a trainable low-rank decomposition matrix is ​​introduced to update its parameters. The forward pass matrix is ​​modified to the sum of the original pre-trained matrix and the updated low-rank decomposition matrix. The basic semantic segmentation network is iteratively trained using the loss function and the gradient is updated until the network converges. The application of low-order adaptive techniques not only improves the training efficiency of the model, but also enhances the performance of the model in processing financial data and extracting and planning user information.

[0084] Step S3: construct a search enhancement generation component, divide the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in a vector database;

[0085] In this embodiment, the specific implementation of step S3 is as follows:

[0086] Step S31: Export existing knowledge documents of the financial platform, where the knowledge documents include financial products, financial policies and financing planning schemes.

[0087] Step S32: clean and extract knowledge document data in various formats (such as PDF, Word, Excel), convert them into a unified plain text format and divide them into blocks. In a specific implementation, in order to adapt to the contextual restrictions of the Tongyi Qianwen 14B large language dialogue model, the text in a unified plain text format will be divided into relatively small blocks.

[0088] Step S33: Use the knowledge graph to construct the plain text format into a hierarchical document structure

[0089] When constructing, the nodes in the knowledge graph represent paragraphs, sentences or other structural units in the document, and the edges represent the semantic and lexical similarities and contextual associations between nodes.

[0090] Step S34: Use the embedding model to encode the hierarchical text structure into a block vector representation and store it in the vector database

[0091] Each text unit is converted into a vector representation using an embedding model, and the distance between two node vectors is calculated. The implementation form of the embedding model is not limited and can be implemented in any achievable manner in the prior art. The distance formula can be expressed as:

[0092]

[0093] In the formula, v i Indicates T i Vector representation of text unit; v j Indicates T j The vector representation of the text unit; ‖v i ‖ represents the vector v i The norm (i.e., length) of j ‖ represents the vector v j The norm (i.e., length) of d(v i ,v j ) represents the vector v i and v j The larger the distance calculated value, the greater the difference between the two vectors.

[0094] After calculating the distances of all nodes, merge the two nodes with the smallest distance and update the distance matrix. Finally, repeat the merging steps to gradually build a tree hierarchy until the number of levels is less than 10.

[0095] The block vector representation obtained in this step is used to calculate the similarity between two vectors when judging the semantic similarity in the subsequent step.

[0096] In this step, we design and implement a retrieval enhancement generation component to prompt the model to retrieve relevant information from an external database and combine the retrieved information with the user-entered question as the input of the large language model. Rich contextual information can enhance the accuracy and credibility of the large language model's answers. In addition, for specific financial scenarios, the retrieval enhancement generation block can enhance the generation capability of the large language model by integrating knowledge from external databases, and allows for continuous updating of knowledge and integration of domain-specific information, thereby overcoming problems such as outdated knowledge and illogical output of the basic large language model.

[0097] Step S4: simulate the user to initiate a dialogue, understand the semantics of the user's input questions, identify the key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning solution suitable for the user

[0098] In this embodiment, the specific implementation of step S4 is as follows:

[0099] Step S41, obtain the dialogue input by the user, and use the embedding model encoded in the same way as step S34 to convert the query dialogue into a dialogue vector representation.

[0100] Specifically, after the user initiates a conversation, the same embedding model as in the vector database is used to process the questions or conversations input by the user and convert them into a fixed-dimensional conversation vector representation, denoted as q.

[0101] Step S42, sequentially calculating the similarity between the dialogue vector and the block vectors in the vector database, and outputting the first K blocks as prompts;

[0102] Specifically, the conversation vector q and each block vector v in the vector database are calculated in turn. i First, we obtain the fixed-dimensional conversation query embedding, calculate the cosine similarity between the query embedding and the document block embedding at each level, and then sort all the document block embeddings according to the calculated cosine similarity, and select the K document blocks that are most relevant to the query from high to low as the retrieval results. These document blocks in the retrieval results will be used as the input of the generation model to expand the semantic context and assist the large language model in generating the final answer.

[0103] The calculation formula of cosine similarity is as follows:

[0104]

[0105] Where q represents the dialogue vector input by the user; v i represents the vector representation of the i-th block in the vector database; q·v i Indicates q and v i The dot product of ; ‖q‖ represents the norm of q; ‖v i ‖ represents v i The norm of S i Denotes the conversation vector q and the block vector v i The similarity matrix between .

[0106] According to the similarity matrix S i The value of , sorts the block vectors, and selects the first K blocks with the highest similarity. The index set obtained after sorting is {i1,i2,i3,...,i k}, the system will select the first K blocks with the highest similarity from the set as prompts, and the value of K is determined according to the actual amount of data.

[0107] Through step S42, the similarity between the dialogue vector and the block vector in the vector database can be effectively calculated, and the most relevant block can be selected as a prompt, thereby improving the accuracy and relevance of the answers generated by the large language model.

[0108] Step S43: Concatenate the retrieved prompt with the input dialogue to obtain a coherent prompt, input the coherent prompt into the vertical large language model trained in step S2, generate an accurate financial financing planning solution suitable for the user, and output the result for display to the user.

[0109] like Figure 4 As shown, Figure 4 The specific operation steps of this embodiment using the retrieval enhancement generation component to improve the accuracy of the financing planning provided by the model are demonstrated.

[0110] Step S5: Evaluate the generated financial financing plan according to the benchmark indicators, iterate the system according to the evaluation results, and continuously optimize the training data set, vertical large language model, and retrieval enhancement generation components.

[0111] In this embodiment, the specific implementation of step S5 is as follows:

[0112] Step S51, defining reading comprehension ability as a model performance indicator, and calculating it based on the correctness of the financial information contained in the model output text;

[0113] Step S52: continuously collect user questions and system responses and transfer them into the training data set as user information-financing planning scheme pairs;

[0114] Step S53: Perform iterative training based on the new training data set to continuously optimize the overall performance of the large language model used to provide financial financing planning solutions.

[0115] This embodiment uses reading comprehension ability to evaluate the accuracy of the financial financing planning solution providing system, where the original application's Tongyi Qianwen 14B large language dialogue model is used as a benchmark. The model needs to give a corresponding financing planning solution based on user information and questions, and the accuracy rate is used as the main evaluation indicator. The experimental results are shown in Table 1:

[0116] Table 1

[0117] method Task 1 Task 2 Task 3 Task 4 Original Tongyi Qianwen 14B model 82.5 62.2 58.0 74.5 Financial financing planning scheme provides system model 90.2 72.9 58.8 76.7

[0118] In Table 1, Task 1 specifically requires the model to provide a financing plan based on user information and questions, Task 2 specifically requires the model to provide multiple financing plans based on the content of the paragraph, Task 3 specifically requires the model to select the correct answer from multiple similar financial-related options, and Task 4 specifically requires the model to judge whether the answer is correct or wrong based on the content of the financial article. Through the four tasks, this embodiment comprehensively evaluates the ability of the large language model used to provide financial financing planning solutions in handling various types of reading comprehension tasks.

[0119] like Figure 5 As shown, Figure 5An experimental visualization dialogue comparison between the original Tongyi Qianwen 14B model and a large language model for providing financial financing planning solutions is shown.

[0120] From Table 1 and Figure 5 It can be seen that the large language model for providing financial financing planning solutions provided by the present invention provides efficient and personalized financing planning services by integrating advanced data analysis and artificial intelligence technologies; it can accurately analyze and process complex financial data and market information, and provide scientific and feasible financing suggestions. Through multi-channel data integration and intelligent analysis technology, relevant information can be quickly extracted and processed from massive data to improve the efficiency of financing planning. In addition, the large language model of this embodiment has strong intelligent analysis capabilities, and users can make personalized settings according to specific needs, providing users with detailed and tailored financing solutions to meet the diverse needs of users, and by providing diversified financing channels, avoiding the high risks brought to users by a single financing method; it supports dynamic adjustment and enhances user experience and satisfaction.

[0121] This embodiment also provides a large language model for providing financial financing planning solutions, and the model is constructed using the model construction method described above.

[0122] In some embodiments, a system for providing a financial financing planning solution based on a large language model is also provided, the system comprising:

[0123] The training data set construction module is used to build a financial database based on financial knowledge and financing planning schemes. It is also used to clean the data in the financial database to obtain a training data set containing representative questions and accurate answers.

[0124] The vertical large language model training and generation module is used to fine-tune the large language model based on the basic large language model using the training data set and low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes; it is also used to simulate users initiating conversations, perform semantic understanding of questions input by users, identify key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning scheme suitable for users.

[0125] The retrieval enhancement generation component building module is used to build the retrieval enhancement generation component, divide the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in the vector database.

[0126] The financial financing planning scheme providing system based on the large language model provided in this embodiment can be personalized by users according to specific needs, providing users with accurate and efficient financing planning schemes to meet the diverse needs of users, and avoiding the high risks brought to users by a single financing method by providing diversified financing channels.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for constructing a large language model for providing financial financing planning solutions, characterized in that: The following steps are involved: Build a financial database based on financial knowledge and financing planning schemes; Clean the data in the financial database to obtain a training data set containing representative questions and accurate answers; The training dataset is used to fine-tune the basic large language model using low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes; Construct a retrieval enhancement generation component to segment the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in a vector database; Simulate the user to initiate a conversation, understand the semantics of the user's input questions, identify the key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning solution suitable for the user; The generated financial financing planning scheme is evaluated according to benchmark indicators, and the system is iterated based on the evaluation results to continuously optimize the training data set, vertical large language model, and retrieval enhancement generation components.

2. The method for constructing a large language model for providing financial financing planning solutions according to claim 1, characterized in that: Build a financial database based on financial knowledge and financing planning schemes; Clean the data in the financial database to obtain a training data set containing representative questions and accurate answers, including: When building a database in the financial field, text data in the financial field is collected from different sources, including financial news, academic papers, financing planning schemes, and open regulatory documents; After data cleaning of the data in the financial database, natural language processing tools are used to perform word segmentation, part-of-speech tagging and named entity recognition on the text data; the text data is unified in data format, and financing planning schemes corresponding to the text data are generated one by one, which are organized into user information-financing planning scheme pairs, and the user information-financing planning scheme pairs are used as a data set of financial question and answer pairs and stored in a relational database.

3. The method for constructing a large language model for providing financial financing planning solutions according to claim 1, characterized in that: The training dataset is used to fine-tune the basic large language model using low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes, including: Load the training data set to train the basic large language model. In the process, introduce low-rank decomposable random Gaussian distribution matrix and all-zero matrix to align the pre-trained weights. Update the low-rank decomposition matrix parameters and modify the forward transfer matrix to be the sum of the original pre-trained matrix and the updated low-rank decomposition matrix; The loss function is used to iteratively train the basic semantic segmentation network and update the gradient until the network converges relatively.

4. The method for constructing a large language model for providing a financial financing planning solution according to claim 3, characterized in that: Load the training data set to train the basic large language model. In the process, introduce low-rank decomposable random Gaussian distribution matrices and all-zero matrices to align the pre-trained weights, including: introduce two trainable low-rank decomposition matrices in the Transformer architecture of each layer of the basic large language model and attach them to the pre-trained weight matrix, while keeping the original weight matrix of the large language model unchanged, which is only used for basic operations in the entire training process; Update the low-rank decomposition matrix parameters and modify the forward transfer matrix to be the sum of the original pre-trained matrix and the updated low-rank decomposition matrix, including: initializing one low-rank decomposition matrix as a random Gaussian distribution and the other low-rank decomposition matrix as an all-zero matrix; in the process of forward propagation, calculate the weight values ​​of the two low-rank decomposition matrices respectively, perform dot multiplication on the weights of the two, add the result to the original weight as the parameter of forward propagation, and use the stochastic gradient descent optimization method to update the parameters.

5. The method for constructing a large language model for providing financial financing planning solutions according to claim 1, characterized in that: Construct a retrieval enhancement generation component to segment the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in a vector database, including: Export the existing knowledge documents of the financial platform, including financial products, financial policies and financing planning schemes; Clean and extract knowledge document data in various formats, convert them into a unified plain text format and split them into blocks; Use knowledge graphs to construct plain text formats into hierarchical document structures; An embedding model is used to encode the hierarchical text structure into block vector representations and stored in a vector database.

6. The method for constructing a large language model for providing financial financing planning solutions according to claim 1, characterized in that: Simulate the user to initiate a conversation, understand the semantics of the questions input by the user, identify the key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning solution suitable for the user; including: Get the dialogue input by the user and use the embedding model to convert the query dialogue into a dialogue vector representation; Calculate the similarity between the dialogue vector and the block vectors in the vector database in sequence, and output the first K blocks as prompts; The retrieved prompts are concatenated with the input dialogue to obtain a coherent prompt, which is then input into the vertical large language model to generate an accurate financial financing planning solution suitable for the user, and the output result is displayed to the user.

7. The method for constructing a large language model for providing financial financing planning solutions according to claim 6, characterized in that: Calculate the similarity between the dialogue vector and the block vector in the vector database, including: Get the fixed-dimensional conversation query embedding and calculate the cosine similarity between the query embedding and the document block embedding at each level; According to the calculated cosine similarity, all document block embeddings are sorted, and the K document blocks most relevant to the query are selected from high to low as the retrieval results. The document blocks in the retrieval results will be used as the input of the generative model to expand the semantic context and assist the large language model to generate the final answer.

8. The method for constructing a large language model for providing financial financing planning solutions according to claim 1, characterized in that: Evaluate the generated financial financing plan according to the benchmark indicators, iterate the system according to the evaluation results, and continuously optimize the training data set, vertical large language model, and retrieval enhancement generation components, including: Define reading comprehension as the model performance indicator, calculated based on the correctness of the financial information contained in the model output text; Continuously collect user questions and system responses and pass them into the training dataset as user information-financing plan pairs; Iterative training is performed based on new training data sets to continuously optimize the overall performance of the large language model used to provide financial financing planning solutions.

9. A large language model for providing financial financing planning solutions, characterized in that: The large language model for providing financial financing planning solutions is constructed using the method for constructing a large language model for providing financial financing planning solutions as described in any one of claims 1 to 8.

10. A financial financing planning scheme providing system based on a large language model, characterized by: The system uses the large language model for providing financial financing planning solutions according to claim 9 to provide financial financing planning solutions for users, and the system includes: The training data set construction module is used to construct a financial database based on financial knowledge and financing planning schemes. It is also used to clean the data in the financial database to obtain a training data set containing representative questions and accurate answers. The vertical large language model training and generation module is used to fine-tune the large language model based on the basic large language model using the training data set and the low-order adaptive technology to obtain a vertical large language model suitable for the task of formulating financial financing planning schemes; it is also used to simulate users initiating conversations, understand the semantics of the questions input by users, identify key information in the semantics, call the retrieval enhancement generation component to obtain the knowledge base content, combine it with the original question and input it into the vertical large language model to generate an accurate financial financing planning scheme suitable for users; The retrieval enhancement generation component building module is used to build the retrieval enhancement generation component, divide the existing knowledge documents of the financial platform into blocks, encode them into block vector representations and store them in the vector database.

Citation Information

Patent Citations

  • Science and technology policy question and answer method and device for fine tuning language large model

    CN116842152A

  • Systems and methods for learning user representations for open vocabulary data sets

    US20200302340A1

Cited By

  • Telephone sales auxiliary system and method based on voice analysis and large model

    CN119722084A

  • A local construction method and device for a vertical large model in the field of environmental monitoring and evaluation

    CN119739874A

  • Large language model optimization system and method based on low-rank adaptive technology

    CN119886279A

  • Financial text emotion analysis method and system based on double-layer low-rank adaptation

    CN120611045A

  • Enterprise digital transformation scheme generation method and device

    CN120822496A