Financial database construction method and system based on large model

Abstract

The invention belongs to the technical field of financial informatics, and discloses a financial database construction method and system based on a large model, and the method comprises the steps: collecting and preprocessing financial data, further carrying out the feature engineering based on a processed data set, and constructing a LanZhou Model large model for self-adaptive fine adjustment. Performing tagging classification on the financial data based on the large model after self-adaptive fine adjustment; constructing a financial database according to the data labeling classification, connecting a front-end component and a rear-end component of the financial database, integrating and executing a test; based on the connection of front-end and rear-end components of a database, a monitoring strategy is deployed to track the performance of the database and formulate a feedback strategy and an optimization scheme. According to the invention, through application of detailed planning and innovative technology, efficiency and accuracy of financial data management are significantly improved, scientificity and efficiency of enterprise decision making are greatly promoted, and revolutionary revolution is brought to financial management.

Description

Technical Field

[0001] The present invention belongs to the field of financial informatics, and in particular relates to a method and system for constructing a financial database based on a large model. Background Art

[0002] In the field of financial management, with the expansion of enterprise scale and the increase of business complexity, the types and quantity of financial data have shown explosive growth. Traditional financial database construction methods rely on manual processing and simple data processing technology, which is difficult to cope with the growing amount of data and complex data structure. In recent years, although financial database systems built based on artificial intelligence technology, especially the progress of natural language processing (NLP) and machine learning, have begun to emerge, they still face many challenges. For example, for unstructured data, it is difficult for traditional methods to effectively extract valuable information from these data; on the other hand, existing database construction methods often ignore the intrinsic connection and semantic information between data, resulting in low efficiency in data organization and retrieval.

[0003] In response to the above problems, although some technical solutions have appeared in the market, such as using optical character recognition (OCR) technology and speech recognition technology to process unstructured data, and using NLP technology for text parsing and information extraction, these solutions still have shortcomings when processing large-scale complex data, especially in terms of data accuracy, processing speed and model generalization ability. In addition, the existing technology lacks an effective mechanism to continuously optimize and update the database system to adapt to the dynamic changes of data, which poses a challenge to the long-term stable operation of the system. Therefore, there is an urgent need for a more efficient and intelligent method to build a financial database to improve the accuracy and efficiency of data processing and to flexibly adapt to the changes and development of data. Summary of the invention

[0004] To solve the above problems, the present invention provides a method and system for building a financial database based on a large model, focusing on solving the shortcomings of the existing technology in data processing flexibility, model adaptability and system performance, and introducing advanced feature engineering technology and adaptive model fine-tuning strategies, as well as optimized database architecture design, in order to achieve more efficient and accurate financial data analysis and management.

[0005] The method for constructing a financial database based on a large model described in the present invention comprises the following steps: Step 1: Collect financial data, perform data cleaning and standardization; Step 2: Based on the standardized data set, further feature engineering is performed to build the LanZhou Model for adaptive fine-tuning and to evaluate the performance of the LanZhou Model. Step 3: Based on the adaptively fine-tuned LanZhou Model, label and classify the financial data; Step 4: Build a financial database based on data labeling and classification to ensure data storage and retrieval; Step 5: Connect the financial database front-end components and back-end components, integrate and perform testing; Step 6: Based on the connection between the database front-end and back-end components, deploy a monitoring strategy to track database performance and optimize based on the feedback.

[0006] Furthermore, step 1 is specifically as follows: List all financial data types that need to be integrated and confirm the data sources; Separate data sources into structured and unstructured data; Design and implement automated data crawling scripts for identified unstructured data sources to crawl required data; For images and PDF documents in unstructured data, use OCR optical character recognition technology to extract text through scanning; for audio and video data, use speech recognition technology to convert them into text; The extracted text data is preliminarily parsed using NLP self-language processing technology; Through the parsed text data, information extraction technology is used to convert unstructured data into structured fields; Then, semantic analysis in NLP technology is used to identify and classify texts to generate structured tags; The structured data converted from processed unstructured data is merged with the original structured data through data fusion technology to form a unified data set; Check the merged data and process the missing values; Anomaly detection using Gaussian mixture models to identify outliers in the data; The adaptive quantization method is used to dynamically adjust the quantization interval according to the data distribution, completing the standardization while retaining the data distribution characteristics.

[0007] Furthermore, step 2 is specifically as follows: The key features are automatically identified by the autoencoder AE, and the expression is: , in, is a quantized data set. is the input data after preprocessing by the autoencoder The reconstructed output obtained after encoding and decoding, is a hyperparameter, is the regularization term, and W represents the set of all weight matrices in the autoencoder; Use the generative adversarial network GAN for feature enhancement, the expression is: , in, represents the objective function of the generative adversarial network, G is the generator, D is the discriminator, is the distribution of the real data, and x is the distribution of the real data A sample was extracted, is the distribution of noise, z is the distribution of The random variables drawn, is the expected logarithmic probability that the discriminator judges the true data x as true, is the sample generated by the discriminator to the generator The expected log probability of being judged false; After the enhancement, the feature values ​​enter the feature encoding stage. For the categorical variables of the original data set, the one-hot encoding method is used, and for the text data in the original data set, the TF-IDF encoding method is used. After feature encoding, StandardScaler is used to perform feature scaling to ensure that all features are at the same scale; Based on feature scaling, principal component analysis PCA is used for dimensionality reduction; Then use the lagged characteristic method and moving average method to process the time series data to capture the dynamic characteristics of the data; In order to optimize the representation of text data, the PorterStemmer function is selected from the NLTK library for stem extraction; Combined with stemming, the text is converted into a TF-IDF matrix using the TfidfVectorizer function from the scikit-learn library; Finally, cross features are manually created to explore the interactions between features; After completing feature engineering, build the LanZhou Model; Using the optimized feature set, the dataset is divided into training set, validation set and test set; Use the training set to train the model, and monitor the performance of the model on the validation set during the training process to prevent fitting; In order to avoid overfitting the training data, a meta-learning strategy is implemented, which is expressed as: , in, is the objective function of meta-learning, N represents the number of tasks, is the loss function of a single task, Where f is the model function, is the model parameter for task i, where i is an index; Adjust the model structure and hyperparameters and perform adaptive fine-tuning based on the model's performance on the validation set; Finally, the final performance of the model is evaluated using the test to ensure the generalization ability of the model.

[0008] Furthermore, step 3 is specifically as follows: Generate a confusion matrix for the LanZhou Model with good performance evaluation, and analyze the classification effect of the LanZhou Model in detail; Use the trained LanZhou Model to predict the unlabeled data, add the predicted category label to each data and post-process the prediction results; Spot check the labels generated by the model and perform manual review and correction; Based on the results of manual review, feedback is given to the LanZhou Model large model training phase for model iterative optimization; Integrate the optimized LanZhou Model with the data processing flow to build automated data classification and labeling; After the LanZhou Model is deployed, its performance is continuously monitored and the LanZhou Model is regularly updated to adapt to data changes.

[0009] Furthermore, step 4 is specifically as follows: Determine the financial database model type based on the classification structure; Design financial database tables; Understand the query requirements of the finance team and other users, and design indexes to optimize the performance of common queries; Introducing graph database technology to represent financial entities and transactions using the relationships between nodes and edges; In order to consider data partitioning, caching strategies and load balancing, data encryption and access control are added; Start building the database, configuring and deploying database instances, and setting up hardware resources; Import classified data into the database, perform data integrity checks, analyze common queries, optimize SQL statements, adjust indexes and views, and adjust storage strategies based on data access patterns; After the build is complete, set up monitoring tools to continuously monitor database performance, record database operations, and perform regular maintenance.

[0010] Furthermore, step 5 is specifically as follows: Based on the constructed financial database, the front-end and back-end components of the database need to be connected; First, backend developers create API interfaces based on RESTful principles to perform CRUD operations on data. Define the data model in the backend code; Then set up routing, define the HTTP request path, and process HTTP requests in the controller code, execute the corresponding business logic and database operations, and return the results to the front end; The front end interacts with the back end by sending HTTP requests to the back end API, requesting to send data; After receiving the data returned by the backend from the frontend, it is displayed on the user interface through the data binding mechanism; Then check the details of the request and response by viewing the Network tab in the developer tools of the front-end browser; Check the definition of the backend API interface and the code of the frontend request interface; Finally, the integration testing phase is carried out and verified.

[0011] Furthermore, step 6 is specifically as follows: After successful integration testing, deploy the monitoring strategy; Select the long short-term memory network LSTM to predict future load model to detect database performance. The formula is: , Where E is a measure of the prediction error, is the mean square error, T is the total number of time steps or data points, is the sum of squares of all prediction errors from time step 1 to T, is the actual observed value at time point t, is the predicted value at time point t, is the actual observed value and predicted values The square of the difference between According to business needs and historical data, reasonable alarm thresholds are set. When performance indicators exceed the normal range, the system can automatically issue an alarm and implement detailed log records; When the monitoring system detects a performance problem, the operations team is notified and responds immediately; Establish a regular performance reporting system to summarize recent database performance data and analyze trends and bottlenecks; Establish user feedback channels to collect direct feedback from end users on system performance; Use monitoring data and user feedback to locate performance bottlenecks; Take optimization measures for the identified bottlenecks; monitor the implementation effect of the optimization plan, continuously monitor performance changes, form an optimization closed loop, and continuously iterate and improve; Record the process and results of each performance optimization and build a knowledge base.

[0012] The present invention also provides a financial database construction system based on a large model, which is used to implement the above method. The system includes a collection and processing module, a fine-tuning module, a classification module, a construction module, a connection test module, and a monitoring optimization module; The collection and processing module is used to collect financial data and perform data cleaning and standardization processing; The fine-tuning module is used to further perform feature engineering based on the standardized data set and construct a large LanZhou Model for adaptive fine-tuning; The classification module is used to classify the financial data by labeling based on the adaptively fine-tuned LanZhou Model. The construction module is used to construct a financial database based on data labeling and classification to ensure data storage and retrieval; The connection test module is used to connect the financial database front-end component and the back-end component, integrate and perform testing; The monitoring and optimization module is used to deploy a monitoring strategy to track database performance and formulate feedback strategies and optimization plans based on the connection between the database front-end and back-end components.

[0013] The beneficial effects described in the present invention are as follows: the present invention realizes comprehensive data collection and automatic structuring through cross-departmental collaboration, optical character recognition OCR and natural language processing NLP, laying a solid foundation; feature engineering and adaptive model fine-tuning enhance learning ability, optimize processing flow, and improve prediction accuracy; database design combines front-end and back-end integration to ensure safe data storage and efficient retrieval, and complex query requirements are met; continuous performance optimization and monitoring strategies maintain system stability. In summary, an intelligent, safe and reliable financial analysis platform is created to accelerate corporate decision-making and innovate financial management. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of a method for constructing a financial database based on a large model in Example 1; Figure 2 This is a flowchart of the labeled classification in Example 1. DETAILED DESCRIPTION

[0015] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0016] Embodiment 1, as Figure 1 and Figure 2As shown in the figure, it is the first embodiment of the present invention, which provides a method for constructing a financial database based on a large model, comprising the following steps:

[0017] Step 1: Collect financial data and perform data cleaning and standardization.

[0018] Going further, list all the types of financial data that need to be integrated and confirm the data sources; Separate data sources into structured and unstructured data; Meetings with finance, IT, management, and other teams that may influence or be affected by the project to discuss their needs, expectations, potential pain points, and what they hope to gain from the new system; Gather all relevant business rules, reporting requirements, data integrity requirements, regulatory compliance standards, etc.; List all required features, such as real-time transaction processing, budget tracking, automatic generation of financial statements, etc. Data types include: transaction data, market data, internal financial reports, and regulatory reporting data; The sources of identification data are: ERP system, bank API, publicly released financial reports, internal systems, etc.

[0019] Structured data refers to data with a fixed format or pattern, usually organized into a table. Each column has a specific meaning, and each row represents an independent record. The characteristics of structured data include: Explicit data types and formats, such as integers, floating point numbers, strings, or dates; Data is usually stored in a relational database, such as a SQL database, or in a table format in spreadsheet software; SQL or other structured query languages ​​can be used for efficient query and analysis; Examples include financial records, customer information, sales data, inventory lists, etc.

[0020] Unstructured data lacks a clear structure or format and is usually free-form, making it difficult to directly apply traditional relational database systems. The characteristics of unstructured data include: There is no fixed format and rules, and it is difficult to express logically using a two-dimensional table structure; This includes multiple formats such as text, images, audio, video, emails, social media posts, etc.; Requires special technologies and algorithms to analyze, such as natural language processing (NLP), image recognition, speech recognition, etc. The meaning and context of the data may require additional context to understand; Examples include email content, research reports, meeting recordings, customer reviews, photographs, video surveillance footage, etc.

[0021] Design and implement automated data crawling scripts for identified unstructured data sources to crawl required data; For images and PDF documents in unstructured data, use OCR optical character recognition technology to extract text through scanning. For audio and video data, use speech recognition technology to convert them into text. The extracted text data is preliminarily parsed using NLP self-language processing technology; Through the parsed text data, information extraction technology is used to convert unstructured data into structured fields; Then, semantic analysis in NLP self-language processing technology is used to identify and classify texts to generate structured tags; The structured data converted from processed unstructured data is merged with the original structured data through data fusion technology to form a unified data set; Check the merged data and process the missing values; For data value new data, use mean, median and mode to fill; For time series data, methods such as linear interpolation and spline interpolation are used to fill missing values; Use decision trees to learn models to predict missing values ​​based on other variables; When the proportion of missing values ​​is high, delete these rows or columns directly; Anomaly detection using Gaussian mixture models is used to identify outliers in the data. The formula is: , in, is the log-likelihood function, which measures the rationality of the parameter set θ under a given data set; the parameter set θ includes the parameters of all Gaussian distributions and their mixing weights; i is the index of a single observation in the data set; N is the total number of observations, that is, the number of samples in the data set; j represents the index of the jth Gaussian distribution in the Gaussian mixture model; log is a function used to calculate the logarithm of a number; K is the number of Gaussian distributions in the Gaussian mixture model, that is, the number of mixed components; is the mixing weight of the jth Gaussian distribution, representing the proportion of the Gaussian distribution in the mixture model; is the density function of the jth Gaussian distribution, describing the The data points observed The probability density of is the ith observation data point in the data set, is the set of parameters associated with the jth Gaussian distribution.

[0022] After removing outliers, the adaptive quantization method is used to dynamically adjust the quantization interval according to the data distribution, and standardize the data while retaining the data distribution characteristics to obtain the quantized data. The formula is: , in, is the quantized data, Q is the adaptive quantization function, X is the original data, is the number of quantization intervals.

[0023] Adaptive quantization is a nonlinear data conversion method that dynamically adjusts the quantization interval according to the distribution of the data. It maps continuous numerical features to a finite discrete interval. This method is particularly suitable for processing features with non-uniform distribution. It can dynamically adjust the boundaries of the interval so that the number of data points in each quantization interval is roughly the same, or divide the interval according to the density of the data, so that more information can be retained.

[0024] It should be noted that when carrying out these steps, ensure compliance with data protection and privacy laws, such as GDPR or other applicable regulations. Data integration ensures that all relevant financial data is collected, whether from internal or external sources. Data cleaning removes and corrects erroneous, incomplete or inaccurate data to ensure data quality. Data standardization converts data into a unified format for subsequent analysis and processing. This step not only improves the reliability and accuracy of the data and reduces bias and errors in analysis, but also ensures the consistency and comparability of the data.

[0025] Step 2: Based on the standardized data set, further feature engineering is performed to build the LanZhou Model for adaptive fine-tuning.

[0026] Furthermore, based on the quantified data obtained after standardization, feature engineering is further performed; specifically: The key features are automatically identified by the autoencoder AE, and the expression is: , in, is the quantized dataset, which will be used as the input of the model. is the preprocessed input data of the autoencoder The reconstructed output obtained after encoding and decoding; is the square of the L2 norm, also known as the square of the Euclidean distance; is a hyperparameter used to balance the effects of reconstruction error and regularization term; is the regularization term, and W represents the set of all weight matrices in the autoencoder; Use the generative adversarial network GAN for feature enhancement, the expression is: , in, represents the objective function of the generative adversarial network, G is the generator, D is the discriminator, is the distribution of the real data, and x is the distribution of the real data A sample was extracted, is the distribution of noise, z is the distribution of The random variables drawn, is the expected logarithmic probability that the discriminator judges the true data x as true, is the sample generated by the discriminator to the generator The expected log probability of being false.

[0027] After the enhancement, the feature values ​​enter the feature encoding stage. For the categorical variables of the original data set, the one-hot encoding method is used, and for the text data in the original data set, the TF-IDF encoding method is used.

[0028] One-hot encoding is a technique for converting categorical variables into numerical data. It is particularly suitable for machine learning models because these models usually require the input to be in numeric form.

[0029] TF-IDF encoding is a statistical method used to measure the importance of a word in a document, and is often used in information retrieval and text mining.

[0030] After feature encoding, StandardScaler is used to perform feature scaling to ensure that all features are at the same scale, thereby improving the convergence speed and prediction accuracy of the model.

[0031] Based on feature scaling, principal component analysis (PCA) is used to reduce the dimension of the data, retain the most information, and reduce the computational burden and overfitting risk. First, the linear relationship between the covariance matrix response features is calculated, and the formula is: , in, represents the covariance matrix, is the Bessel correction, N is the number of samples; It is a centralized data matrix, where each row represents a sample and each column represents a feature; is the transposed matrix of X, It is to calculate the dot product between features; Based on the covariance matrix, the eigenvalues ​​and eigenvectors are solved. The formula is: , in, Represents the covariance matrix Σ and eigenvector The matrix product of represents the i-th eigenvector, which is one of the eigenvectors of the covariance matrix Σ, is the eigenvector the associated eigenvalues; The eigenvalues ​​obtained based on the covariance are arranged in descending order according to the size of the eigenvalue. The larger the eigenvalue, the more data variance the corresponding eigenvector explains; Select the matrix V consisting of the first k eigenvectors k， Convert the data to a low-dimensional space to obtain the reduced-dimensional data, expressed as: , Among them, Z is the data after dimensionality reduction, is a matrix consisting of the first k eigenvectors, where k is usually chosen based on the cumulative proportion of variance explained, for example, retaining at least 95% of the variance; After the dimension reduction data is obtained, the principal component analysis (PCA) step is completed; Then use the lag characteristic method and moving average method to process the time series data to capture the dynamic characteristics of the data and enhance the predictive ability of the model; The hysteresis characteristic formula is: , in, represents the lag feature in time series analysis, y is the observation value in the time series data, t is the index of the time point, is the observed value at time point t, is a conditional statement, indicating that the above formula is valid only when the current time point t is greater than the lag order n; The moving draw formula is: , in, represents the n-period moving average at time point t, It's time point The observation at ; n is the window size, that is, the number of observations involved in the average calculation; i is an index variable used for each element in the iterative summation process.

[0032] In order to optimize the representation of text data, the PorterStemmer function is selected from the NLTK library for stem extraction. Combined with stem extraction, the TfidfVectorizer function is used from the scikit-learn library to convert the text into a TF-IDF matrix, which improves the efficiency and accuracy of text analysis and ensures that the model can capture the semantic information in the text. Finally, cross-features are manually created to explore the interactions between features by combining different features, revealing the complex relationships hidden in the data.

[0033] After completing feature engineering, build the LanZhou Model; specifically: Using the optimized feature set, the dataset is divided into training set, validation set and test set; Use the training set to train the model, and monitor the performance of the model on the validation set during the training process to prevent fitting; In order to avoid overfitting the training data, a meta-learning strategy is implemented so that the model can quickly adapt to new tasks. The expression is: , in, is the objective function of meta-learning, N represents the number of tasks, is the loss function of a single task, Where f is the model function, is the model parameter for task i, i is an index, Indicates that the sum of all N task loss functions is to be calculated; According to the performance of the model on the validation set, adjust the model structure and hyperparameters and perform adaptive fine-tuning until the model configuration with the best performance on the validation set is found; Finally, the final performance of the model is evaluated using the test to ensure the generalization ability of the model. Performance indicators include: accuracy, recall, and F1 score.

[0034] It should be noted that this process involves multiple complex mathematical operations and algorithms, including autoencoders, generative adversarial networks, PCA, lagged features, moving averages, stem extraction, and TF-IDF encoding. Each step is crucial to the final performance of the model, ensuring adequate preparation of the data and effective training of the model. In actual operation, each step needs to be carefully monitored and adjusted to ensure that the model can learn useful information from the data and perform well on unseen data. This usually involves repeated experiments and parameter tuning to achieve the optimal model configuration. This step more effectively captures patterns and associations in the data, improves the model's predictive ability, reduces overfitting, and ensures that the model performs well on new data.

[0035] Step 3: Based on the fine-tuned LanZhou Model, label and classify the financial data.

[0036] Furthermore, the model with good performance evaluation will generate a confusion matrix to analyze the classification effect of the model in detail; the confusion matrix is ​​a specific table layout used to describe the performance of supervised learning algorithms, such as classification models. It can clearly show the difference between the actual output of the algorithm and the expected output, especially in binary or multi-classification problems. The confusion matrix is ​​usually composed of four key components: true positive examples, the number of examples correctly predicted as positive by the model when they are actually positive; false positive examples, the number of examples incorrectly predicted as positive by the model when they are actually negative; true negative examples, the number of examples correctly predicted as negative by the model when they are actually negative; false negative examples, the number of examples incorrectly predicted as negative by the model when they are actually positive. In the confusion matrix, rows represent actual categories and columns represent predicted categories. Various performance indicators can be calculated from the confusion matrix, including: accuracy, the proportion of all correctly predicted samples to the total number of samples, precision, the proportion of all samples predicted to be positive that are actually positive, recall, the proportion of all samples actually positive that are correctly predicted to be positive, F1 score, the harmonic mean of precision and recall, which is used to balance precision and recall. The confusion matrix is ​​an important tool for evaluating the performance of a classifier. It not only provides a comprehensive perspective on the performance of a classifier, but also helps identify on which types of errors the classifier makes the most mistakes, thereby guiding the direction of model improvement.

[0037] Use the trained model to predict unlabeled data, add predicted category labels to each data and post-process the prediction results, automate data processing, save labor costs, enrich the data set, and possibly improve the training effect of subsequent models.

[0038] Sampling checks on labels generated by the model and manual review and corrections are performed to improve the overall quality of the data and provide feedback for model iteration. Based on the results of manual review, the model parameters or algorithms are adjusted to improve classification accuracy, and feedback is fed back to the model training phase for model iteration optimization to make the model more robust and better able to cope with different types of data, continuously improve the performance and reliability of the model, and ensure that the model remains efficient over time; the optimized model is integrated with the data processing process to build automated data classification and labeling. After the model is deployed, its performance is continuously monitored and the model is regularly updated to adapt to changes in data.

[0039] It should be noted that throughout the process, the confusion matrix is ​​used to evaluate the classification effect of the model, the model is used to classify unlabeled data, the data processing efficiency is improved, the model is optimized according to the results of manual review, the classification accuracy is improved, the data is made more operational, easy to manage and analyze, the accuracy and practicality of the model are improved, and the model performance is continuously optimized through feedback loops.

[0040] Step 4: Build a financial database based on data labeling and classification to ensure data storage and retrieval.

[0041] Furthermore, based on the classification structure, the database model type is determined to ensure that it can effectively handle structured and semi-structured data; Plan database architecture, design database tables and fields, and ensure data consistency and integrity; Understand the query needs of the finance team and other users, design indexes to optimize the performance of common queries and speed up data retrieval; Introducing graph database technology to represent financial entities and transactions using the relationships between nodes and edges; To improve data reading and writing efficiency by considering data partitioning, caching strategy and load balancing; Add data encryption and access control to protect sensitive financial information; Start building the database, configuring and deploying database instances, and setting up hardware resources; Import classified data into the database, perform data integrity checks, analyze common queries, optimize SQL statements, adjust indexes and views, and adjust storage strategies based on data access patterns.

[0042] Optimizing SQL statements is to improve the efficiency of SQL queries, reduce query response time, enhance database performance, ensure the rational use of database resources, and avoid unnecessary CPU and I / O consumption; adjusting indexes is to speed up queries, quickly locate data through indexes, reduce I / O operations, and improve data retrieval efficiency; adjusting views is to improve query efficiency by pre-calculating and storing the results of complex queries, simplifying complex queries, and providing users with a simpler query interface.

[0043] After the build is complete, set up monitoring tools to continuously monitor database performance, record database operations, and perform regular maintenance.

[0044] It should be noted that designing and building a database to store classified data, ensuring that data can be retrieved and accessed quickly and accurately, adjusting database configuration, and improving reading and writing efficiency, following these steps to build and manage a financial database can not only ensure data security and quality, but also improve data processing efficiency, reduce costs, and support efficient business operations and strategic decision-making.

[0045] Step 5: Connect the financial database front-end components and back-end components, integrate and perform testing.

[0046] First, the backend developer creates an API interface based on the RESTful principle to perform CRUD operations on the data.

[0047] RESTful is a software architecture style, mainly used to design APIs for network applications. The core concept of the RESTful principle is to view network applications as a collection of resources that can be accessed and operated through the HTTP protocol. The main features are: stateless, each request should contain all the information needed to understand and process the request, the server will not save any state of the client, which means that the server does not need to save the session state between multiple requests, thereby simplifying the implementation of the server and improving scalability, client-server, the RESTful architecture separates the user interface client and the data storage server, allowing independent development and evolution. The client is responsible for the user interface and user experience, while the server is responsible for data storage and management. Cache. RESTful API should be able to indicate whether the response is cacheable. Through the caching mechanism, network traffic can be reduced and the response speed can be improved. Unified interface. RESTful API should have a unified interface, which includes the use of standard methods such as GET, POST, PUT, DELETE to operate resources, and the use of URI to uniquely identify resources. Layered system. RESTful architecture allows the behavior of components to be logically layered. Each layer does not need to know the implementation details of the next layer. This helps to improve security because some layers can be designed as opaque proxies that only pass requests without processing. On-demand code. Although this is not required, the RESTful architecture can support the inclusion of executable code in the response, such as JavaScript, which the client can execute after receiving the response.

[0048] The API interface is a method that allows different software applications to communicate with each other. It defines a set of rules and protocols for building and using software applications. The API interface specifies how to request services, pass data, and receive response formats, so that different software systems can understand and interact with each other.

[0049] In order to correctly transfer data between databases, define the data model in the backend code.

[0050] Then set up routing, define the HTTP request path, and process these requests in the controller code, execute the corresponding business logic and database operations, and return the results to the front end.

[0051] HTTP is the most widely used network protocol on the Internet. It is mainly used to transmit hypertext from the World Wide Web server to the local browser. It is an application layer protocol based on the TCP / IP protocol. It allows data to be transmitted between the client and the server. It is the standard way for data transmission between the Web browser and the server. The HTTP protocol specifies how the client requests a web page from the server and how the server sends the web page to the client. The HTTP request path is used to specify the requested resource location. Defining the HTTP request path is one of the key steps in creating a RESTful API. It enables the front end to call the back end service through a specific URL to obtain, modify, create or delete data. Each path usually corresponds to a resource or a group of related resources, and HTTP methods such as GET, POST, PUT, and DELETE define the type of operation on these resources.

[0052] The front-end sends HTTP requests to the back-end API to request data to interact with the back-end. After receiving the data returned by the back-end, the front-end displays it on the user interface through the data binding mechanism to visualize the data. Then, check the details of the request and response by viewing the Network tab in the developer tools of the front-end browser. Check the definition of the back-end API interface and the code of the front-end request interface. Finally, conduct the integration test phase and verify.

[0053] It should be noted that the front-end and back-end interaction of data is realized, and the various components work together to ensure the functional integrity and performance stability of the system. Through these steps, the front-end and back-end components of the financial database can be tightly integrated to provide a functionally complete, stable and secure application.

[0054] Step 6: Based on the connection between the database front-end and back-end components, deploy a monitoring strategy to track database performance and develop feedback strategies and optimization plans, specifically: Select the long short-term memory network LSTM to predict future load model to detect database performance. The formula is: , Where E is a measure of the prediction error, is the mean square error, T is the total number of time steps or data points, is the sum of squares of all prediction errors from time step 1 to T, is the actual observed value at time point t, is the predicted value at time point t, is the actual observed value and predicted values The square of the difference between .

[0055] According to business needs and historical data, reasonable alarm thresholds are set. When performance indicators exceed the normal range, the system can automatically issue an alarm and implement detailed log records; When the monitoring system detects a performance problem, the operations team is notified and responds immediately; Establish a regular performance reporting system to summarize recent database performance data and analyze trends and bottlenecks; Establish user feedback channels to collect direct feedback from end users on system performance; Use monitoring data and user feedback to locate performance bottlenecks; Take optimization measures for the identified bottlenecks; Optimization measures such as: SQL query optimization, index adjustment, hardware upgrade, load balancing, etc.; Incorporate the implementation effect of the optimization plan into monitoring, continuously monitor performance changes, form an optimization closed loop, and continuously iterate and improve; Record the process and results of each performance optimization and build a knowledge base.

[0056] Building a knowledge base is an important means for organizations to manage information, improve efficiency, promote innovation and maintain competitive advantage. It helps transform individual and team knowledge into the collective wisdom of the organization, laying the foundation for the long-term development of the organization.

[0057] It should be noted that by detecting and predicting database performance issues, establishing response and optimization processes, and adjusting the system based on monitoring data and user feedback, such a closed-loop monitoring and optimization process can ensure that the database system can continue to run efficiently, while also being able to quickly respond to and resolve any performance issues that arise, thereby improving the overall service quality and user experience.

[0058] This embodiment also provides a financial database construction system based on a large model, including a collection and processing module, a fine-tuning module, a classification module, a construction module, a connection test module, and a monitoring and optimization module; the collection and processing module is used to collect financial data and perform data cleaning and standardization processing; the fine-tuning module is used to further perform feature engineering based on the standardized data set, and build a LanZhou Model large model for adaptive fine-tuning; the classification module is used to label and classify the financial data based on the LanZhou Model large model after adaptive fine-tuning; the construction module is used to construct a financial database based on the labeled data classification to ensure data storage and retrieval; the connection test module is used to connect the front-end components and back-end components of the financial database, integrate them and perform tests; the monitoring and optimization module is used to deploy a monitoring strategy to track database performance and formulate feedback strategies and optimization plans based on the connection between the front-end and back-end components of the database.

[0059] In summary, the present invention solves the problems of low automation in financial data processing, weak model prediction ability and poor system performance through deep learning technology and data science methods. It specifically adopts data cleaning, feature engineering optimization, adaptive model fine-tuning, database architecture design and monitoring strategies to achieve intelligent data preprocessing, enhanced model generalization ability, more efficient database operation and continuous optimization of system performance, thereby comprehensively improving the accuracy and efficiency of financial data analysis.

[0060] Example 2, referring to Table 1, is the second example of the present invention. To further verify the advancement of the present invention, experimental simulation data of a financial database construction method based on a large model is provided.

[0061] In order to verify the effect of the technical solution of the present invention, a detailed comparative experiment was conducted. The experimental goal is to evaluate the performance improvement of the invention solution in processing financial data, especially in data cleaning, feature engineering, model fine-tuning and database management. The experimental subjects selected two companies of similar size, one of which used traditional financial data processing methods as the control group, and the other applied the solution of the present invention as the experimental group.

[0062] Experimental Group: Data collection and cleaning: collect transaction data, market data, internal financial reports, etc. from multiple sources such as ERP systems, bank APIs, internal systems, etc., use the automated data capture script of the present invention to process unstructured data, convert it into structured data through OCR and speech recognition technology, then use NLP technology for preliminary analysis, apply Gaussian mixture model to identify outliers, and use adaptive quantization method to standardize data; Feature engineering and model fine-tuning: Identify key features and enhance them based on autoencoders and GAN technology, use one-hot encoding and TF-IDF encoding to process categorical variables and text data, then perform feature scaling, principal component analysis, and time series processing. Finally, build a large LanZhou Model and perform adaptive fine-tuning through a meta-learning strategy. Data labeling and classification and database construction: After model evaluation, financial data is labeled and classified, the optimized model is integrated with the data processing flow, and an automated data classification and labeling financial database is built to ensure data storage and retrieval efficiency; Front-end and back-end integration and monitoring: connect front-end and back-end components, perform integration testing, deploy LSTM-based monitoring strategies, track database performance in real time, and develop feedback strategies and optimization plans; Control group: Traditional data processing methods were used, including manual data cleaning, basic feature engineering, general machine learning model training, and simple database management; The details are shown in Table 1 below.

[0063] Table 1 Experimental comparison table

[0064] Data analysis: By comparing the experimental group with the control group, it can be seen that the scheme of the present invention has significant advantages in many aspects: Data cleaning time: The experimental group took only 1.5 hours, while the control group took 8 hours, which reflects the high efficiency of the present invention in data cleaning automation; Feature engineering efficiency: The feature engineering efficiency of the experimental group was 2.3 features / minute, much higher than the 0.8 features / minute of the control group, which proved the high efficiency of automated feature recognition and enhancement technology; Model training time: The model training time of the experimental group was shortened to 4 hours, which was significantly reduced compared to 12 hours in the control group. This was attributed to the optimized model structure and meta-learning strategy. Prediction accuracy: The prediction accuracy of the experimental group reached 95%, while that of the control group was only 82%, indicating the excellent performance of LanZhouModel in data processing and prediction capabilities; Database query speed: The database query speed of the experimental group was 0.05 seconds, much faster than the 0.5 seconds of the control group, reflecting the performance improvement brought by the optimized database architecture and index design;

[0065] In summary, the scheme of the present invention shows innovation and novelty in all aspects of data processing, which not only improves the efficiency and accuracy of data processing, but also optimizes the performance of the database, providing strong technical support for financial data analysis.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A method for constructing a financial database based on a large model. It is characterized in that The following steps are involved: Step 1: Collect financial data, perform data cleaning and standardization; Step 2: Based on the standardized data set, further feature engineering is performed to build the LanZhou Model for adaptive fine-tuning and to evaluate the performance of the LanZhou Model. Step 3: Based on the adaptively fine-tuned LanZhou Model, label and classify the financial data; Step 4: Build a financial database based on data labeling and classification to ensure data storage and retrieval; Step 5: Connect the financial database front-end components and back-end components, integrate and perform testing; Step 6: Based on the connection between the database front-end and back-end components, deploy a monitoring strategy to track database performance and optimize based on the feedback.

2. A method for constructing a financial database based on a large model according to claim 1, It is characterized in that Step 1 is as follows: List all financial data types that need to be integrated and confirm the data sources; Separate data sources into structured and unstructured data; Design and implement automated data crawling scripts for identified unstructured data sources to crawl required data; For images and PDF documents in unstructured data, use OCR optical character recognition technology to extract text through scanning; for audio and video data, use speech recognition technology to convert them into text; The extracted text data is preliminarily parsed using NLP self-language processing technology; Through the parsed text data, information extraction technology is used to convert unstructured data into structured fields; Then, semantic analysis in NLP technology is used to identify and classify texts to generate structured tags; The structured data converted from processed unstructured data is merged with the original structured data through data fusion technology to form a unified data set; Check the merged data and process the missing values; Anomaly detection using Gaussian mixture models to identify outliers in the data; The adaptive quantization method is used to dynamically adjust the quantization interval according to the data distribution, completing the standardization while retaining the data distribution characteristics.

3. A method for constructing a financial database based on a large model according to claim 2, It is characterized in that Step 2 is as follows: The key features are automatically identified by the autoencoder AE, and the expression is: ， in, is a quantized data set. is the preprocessed input data of the autoencoder The reconstructed output obtained after encoding and decoding, is a hyperparameter, is the regularization term, and W represents the set of all weight matrices in the autoencoder; Use the generative adversarial network GAN for feature enhancement, the expression is: ， in, represents the objective function of the generative adversarial network, G is the generator, D is the discriminator, is the distribution of the real data, and x is the distribution of the real data A sample was extracted, is the distribution of noise, z is the distribution of The random variables drawn, is the expected logarithmic probability that the discriminator judges the true data x as true, is the sample generated by the discriminator to the generator The expected log probability of being false; After the enhancement, the feature values ​​enter the feature encoding stage. For the categorical variables of the original data set, the one-hot encoding method is used, and for the text data in the original data set, the TF-IDF encoding method is used. After feature encoding, StandardScaler is used to scale the features to ensure that all features are at the same scale. Based on feature scaling, principal component analysis PCA is used for dimensionality reduction; Then use the lagged characteristic method and moving average method to process the time series data to capture the dynamic characteristics of the data; In order to optimize the representation of text data, the PorterStemmer function is selected from the NLTK library for stem extraction; Combined with stemming, the text is converted into a TF-IDF matrix using the TfidfVectorizer function from the scikit-learn library; Finally, cross features are manually created to explore the interactions between features; After completing feature engineering, build the LanZhou Model; Using the optimized feature set, the dataset is divided into training set, validation set and test set; Use the training set to train the model, and monitor the performance of the model on the validation set during the training process to prevent fitting; In order to avoid overfitting the training data, a meta-learning strategy is implemented, which is expressed as: ， in, is the objective function of meta-learning, N represents the number of tasks, is the loss function of a single task, Where f is the model function, is the model parameter for task i, where i is an index; Adjust the model structure and hyperparameters and perform adaptive fine-tuning based on the model's performance on the validation set; Finally, the final performance of the model is evaluated using the test to ensure the generalization ability of the model.

4. A method for constructing a financial database based on a large model according to claim 3, It is characterized in that Step 3 is as follows: Generate a confusion matrix for the LanZhou Model with good performance evaluation, and analyze the classification effect of the LanZhou Model in detail; Use the trained LanZhou Model to predict the unlabeled data, add the predicted category label to each data and post-process the prediction results; Spot check the labels generated by the model and perform manual review and correction; Based on the results of manual review, feedback is given to the LanZhou Model large model training phase for model iterative optimization; Integrate the optimized LanZhou Model with the data processing flow to build automated data classification and labeling; After the LanZhou Model is deployed, its performance is continuously monitored and the LanZhou Model is regularly updated to adapt to data changes.

5. A method for constructing a financial database based on a large model according to claim 4, It is characterized in that Step 4 is as follows: Determine the financial database model type based on the classification structure; Design financial database tables; Understand the query requirements of the finance team and other users, and design indexes to optimize the performance of common queries; Introducing graph database technology to represent financial entities and transactions using the relationships between nodes and edges; In order to consider data partitioning, caching strategies and load balancing, data encryption and access control are added; Start building the database, configuring and deploying database instances, and setting up hardware resources; Import classified data into the database, perform data integrity checks, analyze common queries, optimize SQL statements, adjust indexes and views, and adjust storage strategies based on data access patterns; After the build is complete, set up monitoring tools to continuously monitor database performance, record database operations, and perform regular maintenance.

6. A method for constructing a financial database based on a large model according to claim 5, It is characterized in that Step 5 is as follows: Based on the constructed financial database, the front-end and back-end components of the database need to be connected; First, backend developers create API interfaces based on RESTful principles to perform CRUD operations on data. Define the data model in the backend code; Then set up routing, define the HTTP request path, and process HTTP requests in the controller code, execute the corresponding business logic and database operations, and return the results to the front end; The front end interacts with the back end by sending HTTP requests to the back end API, requesting to send data; After receiving the data returned by the backend from the frontend, it is displayed on the user interface through the data binding mechanism; Then check the details of the request and response by viewing the Network tab in the developer tools of the front-end browser; Check the definition of the backend API interface and the code of the frontend request interface; Finally, the integration testing phase is carried out and verified.

7. A method for constructing a financial database based on a large model according to claim 6, It is characterized in that Step 6 is as follows: After successful integration testing, deploy the monitoring strategy; Select the long short-term memory network LSTM to predict future load model to detect database performance. The formula is: ， Where E is a measure of the prediction error, is the mean square error, T is the total number of time steps or data points, is the sum of squares of all prediction errors from time step 1 to T, is the actual observed value at time point t, is the predicted value at time point t, is the actual observed value and predicted values The square of the difference between According to business needs and historical data, reasonable alarm thresholds are set. When performance indicators exceed the normal range, the system can automatically issue an alarm and implement detailed log records; When the monitoring system detects a performance problem, the operations team is notified and responds immediately; Establish a regular performance reporting system to summarize recent database performance data and analyze trends and bottlenecks; Establish user feedback channels to collect direct feedback from end users on system performance; Use monitoring data and user feedback to locate performance bottlenecks; Take optimization measures for the identified bottlenecks; Incorporate the implementation effect of the optimization plan into monitoring, continuously monitor performance changes, form an optimization closed loop, and continuously iterate and improve; Record the process and results of each performance optimization and build a knowledge base.

8. A financial database construction system based on a large model, It is characterized in that A method for constructing a financial database based on a large model according to any one of claims 1 to 7, wherein the system comprises a collection and processing module, a fine-tuning module, a classification module, a construction module, a connection test module, and a monitoring and optimization module; The collection and processing module is used to collect financial data and perform data cleaning and standardization processing; The fine-tuning module is used to further perform feature engineering based on the standardized data set and construct a large LanZhouModel model for adaptive fine-tuning; The classification module is used to classify the financial data by labeling based on the adaptively fine-tuned LanZhou Model. The construction module is used to construct a financial database based on data labeling and classification to ensure data storage and retrieval; The connection test module is used to connect the financial database front-end component and the back-end component, integrate and perform testing; The monitoring and optimization module is used to deploy a monitoring strategy to track database performance and formulate feedback strategies and optimization plans based on the connection between the database front-end and back-end components.

Images