A method for optimizing storage of financial big data

Abstract

The present invention relates to the technical field of enterprise management methods, and more specifically, to a method for optimizing storage of financial big data, comprising the following steps: step 1. aggregating multi-source heterogeneous financial data; step 2. subject domain division and data classification; step 3. multimodal semantic encoding: designing corresponding semantic encoders for different types of data, and realizing cross-modal semantic fusion by introducing adversarial constraints; step 4. semantic enhancement and compression; step 5. distributed storage under a federated learning framework; step 7. storage scheduling and automatic scaling: designing a storage resource scheduling strategy based on reinforcement learning, dynamically optimizing the storage location and number of copies of data according to data heat and access mode factors, and triggering automatic scaling of storage clusters according to real-time monitoring and prediction of system load; step 8. metadata management and version control.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage methods, and more specifically, to a financial big data optimization storage method. Background Art

[0002] In recent years, with the rapid development of financial services and the in-depth application of information technology, the data generated and accumulated by financial institutions has shown an explosive growth trend. According to statistics, the global financial big data market size has reached US$235.6 billion in 2020, and is expected to further grow to US$489.1 billion by 2025. Financial big data contains huge value and provides important support for financial innovation, risk management, customer service, etc. However, massive, heterogeneous, and rapidly growing financial big data also poses severe challenges to traditional data storage and management technologies.

[0003] First, financial big data has significant heterogeneity and unstructured characteristics. In addition to traditional structured transaction data, financial institutions also need to process a large amount of unstructured data such as text, images, audio and video, such as customer contracts, bill images, audio and video recordings, etc. According to statistics, unstructured data has accounted for more than 80% of the total financial data. For these heterogeneous data, traditional relational databases are difficult to provide efficient storage and retrieval support. Although some NoSQL databases support multiple data types, they lack a unified data model and query language, making data integration and analysis difficult.

[0004] Secondly, financial big data places extremely high demands on data security and privacy protection. Financial data usually contains sensitive data such as customer identity information, transaction records, and financial status. Once leaked, it will cause serious privacy violations and trust crises. Traditional centralized storage architectures have a high risk of single point failures and data leakage. Although some distributed storage systems can provide better availability and security, they still face the risk of privacy leakage when sharing data between different storage nodes. How to achieve safe and efficient financial big data sharing and collaborative processing while protecting privacy is a difficult problem that needs to be solved urgently.

[0005] Thirdly, the rapid growth of financial big data has put tremendous pressure on storage capacity and scalability. It is predicted that by 2025, the amount of global financial data will reach 463EB, and the corresponding storage cost will be as high as billions of dollars. Traditional data storage systems are difficult to carry such a large amount of data, and lack elastic expansion capabilities, making it difficult to adapt to the dynamic changes in data volume and business needs. Although some big data storage systems such as HDFS provide better scalability, they still have shortcomings in storage efficiency and data consistency. How to maximize data storage efficiency and access performance with limited storage resources is another challenge facing financial institutions.

[0006] Finally, massive financial big data has put forward higher requirements for data semantic understanding and real-time processing. Financial business needs to quickly extract key information from raw data to achieve real-time risk control, marketing, decision-making, etc. Traditional data processing methods such as keyword matching and statistical analysis are difficult to deeply understand data semantics, and the computational efficiency is limited. Although some machine learning methods can learn data features, they are still insufficient in cross-modal data fusion and model interpretability. How to achieve semantic layer understanding and real-time computational analysis of financial big data is another pain point for financial institutions.

[0007] In summary, traditional data storage and processing technologies can no longer meet the requirements of financial big data in terms of heterogeneity, privacy, scalability, semantics, etc., and an innovative storage optimization solution is urgently needed to break through the technical bottleneck. Therefore, it is urgent to provide a financial big data optimization storage method to solve the shortcomings of existing technologies. Summary of the invention

[0008] The present invention aims at the above technical problems and proposes a financial big data storage optimization method that integrates cross-modal semantic coding, federated learning, adaptive compression and other technologies. Compared with the existing technology, the present invention has achieved all-round breakthroughs in data semantic representation, privacy protection, storage efficiency, retrieval performance and other aspects, and will surely provide key support for the intelligent upgrade and innovative application of the financial big data platform.

[0009] The present invention provides a financial big data optimization storage method, comprising the following steps:

[0010] Step 1. Aggregation of multi-source heterogeneous financial data: Obtain structured, semi-structured and unstructured data through the data acquisition module, process them using the data cleaning module, and convert them into a unified platform standard format through the format conversion module;

[0011] Step 2. Subject domain division and data classification: divide the collected data into subject domains based on the financial business subject domain knowledge base and classify them according to data characteristics;

[0012] Step 3. Multimodal semantic encoding: Design corresponding semantic encoders for different types of data and achieve cross-modal semantic fusion by introducing adversarial constraints;

[0013] Step 4. Semantic enhancement and compression: The financial knowledge graph is used to enhance the semantic coding, and the weights of semantics of different granularities are adaptively adjusted through the attention mechanism. The semantic cascade encoding network is designed to compress the semantic representation layer by layer. The calculation formula of the l-th layer cascade unit is:

[0014] h l =g l (W l h l-1 +b l)⊙σ(U l h l-1 +c l )

[0015] where h l is the output of the l-th layer cascade unit, g l (·) is the mapping function of the lth layer, σ(·) is the gating function, W l , U l , b l , c l is the learnable parameter of the l-th layer cascade unit, ⊙ is the Hadamard product;

[0016] Step 5. Distributed storage under the federated learning framework: The semantic coding is stored in different nodes according to the subject domain, and the federated learning framework is used to coordinate the training and updating between nodes. The local node k updates the semantic coding model parameters θ based on the privacy-preserving gradient aggregation method. k :

[0017]

[0018] Where η is the learning rate, DP(.) is the differential privacy gradient perturbation function, and L k is the local objective function of node k, and the central server aggregates the updated model parameters of each node to obtain a globally consistent semantic encoding model θ:

[0019]

[0020] Among them, K is the total number of nodes, n k is the number of local samples of node k, and n is the total number of global samples;

[0021] Step 6. Consistent hash index construction: Apply the consistent hashing method on the compressed semantic coding to generate the index. The definition of the consistent hashing function is:

[0022] hash(z)=(a·z+b)mod m

[0023] Among them, z is the compressed semantic code, α and b are hash functions, and the parameter m is the number of hash buckets;

[0024] Step 7. Storage Scheduling and Automatic Scaling: Design a storage resource scheduling strategy based on reinforcement learning, dynamically optimize the storage location and number of copies of data based on data popularity and access pattern factors, and trigger automatic scaling of the storage cluster based on real-time monitoring and prediction of system load;

[0025] Step 8. Metadata management and version control: Build a unified metadata management module, organize and store metadata in the form of a graph database, and use a multi-version concurrent control mechanism to perform version management on metadata and compressed semantic coding.

[0026] Specifically, for numerical structured data, a hybrid semantic encoder based on factor decomposition machine and tree embedding is used, where the formula of the factor decomposition machine part is:

[0027]

[0028] in, is the predicted output, w 0 is the global bias term, w i is the weight of the ε-th dimension feature, <v i , v j > is the latent vector interaction between the i-th dimension and the j-th dimension feature, x i is the eigenvalue of the i-th dimension;

[0029] The formula for the tree embedding part is:

[0030]

[0031] Where e(x) is the tree embedding output vector, f(x i ) is the one-hot feature of the i-th dimension, encoding μ(l(x i )) is the embedding vector of the leaf node of the i-th dimension feature in the decision tree, l(x i ) is the leaf node index of the i-th dimension feature in the decision tree.

[0032] Specifically, for text data, a cross-layer semantic encoder based on Transformer and BERT is used, where the formula of the Transformer part is:

[0033]

[0034] MultiHead(Q,K,V)=Concat(head 1 , ..., head h )W O

[0035] Among them, Q, K, and V are query, key, and value matrices respectively, and d k is the key vector dimension,

[0036] head i =Attention(QW i Q , KW i K , VW iV ) represents the i-th attention head, W i Q , W i K , W i V , W O is the learnable matrix parameter;

[0037] The formula for the BERT part is:

[0038]

[0039] Where x=(x 1 , ..., x n ) is the word embedding column of the text, p(x i |x<i, x>i) is the i-th word x given the context xi i The conditional probability, h <i，>i is the hidden state of BERT’s bidirectional language model at the i-th position.

[0040] Specifically, for image data, a hierarchical semantic encoder based on convolutional neural network and contrastive learning is used, where the formula of the convolutional neural network part is:

[0041]

[0042] in, is the (i, j)th element of the output of the lth layer convolution, f(·) is the activation function, M l and N l is the height and width of the l-th layer convolution kernel, and b l are the weight and bias of the convolution kernel of the lth layer respectively;

[0043] The formula for the contrastive learning part is:

[0044]

[0045] in, is the contrastive learning loss function, N is the image batch size, z i is the semantic code of the i-th image, z i，+ is the positive semantic encoding of the i-th image, and τ is the temperature hyperparameter.

[0046] Specifically, in the multimodal semantic encoding step, an adversarial objective function between the modality discriminator D and the encoder E is introduced:

[0047] Through adversarial training, data of different modalities are forced to achieve distribution alignment in the semantic coding space, thus achieving the goal of cross-modal fusion.

[0048] Specifically, in the semantic enhancement and compression step, the calculation formula of the attention weight is:

[0049]

[0050] Among them, α i is the semantic code z for the i-th granularity i is the attention weight, f(·,·) is the attention score function, q is the global query vector, and m is the total number of semantic granularities. The semantic cascade encoding network significantly reduces the representation dimension and improves storage efficiency while retaining key semantics through iterative adaptive compression.

[0051] Specifically, in the distributed storage step under the federated learning framework, differential privacy gradient perturbation DP(.) is used to ensure the privacy security of the model aggregation process.

[0052] Specifically, the consistent hash index maps semantically similar codes to adjacent hash buckets to achieve efficient similarity search.

[0053] Specifically, the reinforcement learning model of the storage resource scheduling strategy uses system throughput and latency as reward signals and learns the optimal scheduling strategy through continuous trial and error.

[0054] Specifically, the metadata includes data description information, lineage relationships, and schema evolution. Metadata query, subscription, and change tracking are implemented through a graph database interface.

[0055] Specifically, the multi-version concurrent control mechanism generates a new version for each update of metadata and compressed semantic coding, forming a version chain, supporting fast historical data backtracking and snapshot analysis.

[0056] The present invention has the following beneficial effects:

[0057] First, the multimodal semantic encoding module realizes the extraction and fusion of cross-modal semantics by semantic modeling of heterogeneous data such as structured, text, and images. Among them, the hybrid encoder of factorization machine and tree embedding mines the implicit feature interactions contained in structured data, the cross-layer encoder of Transformer and BERT captures the deep semantic dependencies of text data, and the hierarchical encoder of CNN and contrastive learning refines the local and global semantics of image data. More importantly, by mapping different modal data to a shared semantic space through adversarial constraints, the modal barriers are truly broken and seamless fusion at the semantic level is achieved. This cross-modal semantic fusion can not only significantly improve the semantic representation ability of data, but also lay a solid foundation for subsequent cross-modal retrieval, reasoning and other tasks.

[0058] Secondly, the distributed storage module under the federated learning framework cleverly balances the contradiction between data privacy protection and global modeling. The traditional centralized storage model faces a high risk of privacy leakage, while completely localized storage is difficult to optimize globally. The present invention adopts the idea of ​​federated learning, while retaining data locality, to achieve collaborative learning between different nodes through encrypted communication and parameter aggregation. Each node updates the semantic encoding model locally with its own data, and uses differential privacy technology to desensitize the gradient information, and then securely shares the gradient with other nodes, and finally aggregates the central server to form a globally consistent semantic representation model. This decentralized, privacy-preserving learning paradigm not only overcomes the data island effect, but also effectively avoids the risk of privacy leakage, providing a new idea for financial big data processing.

[0059] Thirdly, the semantic enhancement and compression module continuously enriches and optimizes the semantic representation through external knowledge injection and adaptive weight adjustment, and introduces a semantic cascade coding network to achieve lossless compression. Different from simple numerical compression, the present invention starts from a semantic perspective and adaptively retains key semantics and filters secondary semantics through a gating mechanism, thereby simplifying the semantic representation to the greatest extent without losing core information. The multi-layer cascade structure makes the compression process more sophisticated and stable, which not only reduces storage overhead, but also reduces the computational burden for subsequent downstream tasks such as semantic retrieval and similarity calculation. In addition, the semantic enhancement driven by the knowledge graph further expands the extension of the representation, and the dynamic weight adjustment of the attention mechanism makes the representation more accurate. The two complement each other and jointly optimize the effect of semantic representation.

[0060] Finally, the consistent hash index and storage automatic scaling module are the finishing touches, providing efficient support for real-time retrieval and elastic storage of massive semantic data. Consistent hashing maps semantically similar data encodings to adjacent storage nodes, which not only optimizes the locality of data, but also achieves balanced load distribution. The multi-index mechanism further improves the precision and recall of retrieval. The automatic scheduling of storage resources continuously optimizes data layout and load distribution based on reinforcement learning, and triggers resource scaling in real time according to system pressure, maximizing resource utilization while ensuring service levels. The introduction of metadata management and version control mechanisms provides full life cycle support for data lineage tracking and change management. These designs complement the core modules such as semantic encoding, cross-modal fusion, and distributed storage to build an adaptive, scalable, efficient and flexible storage management system.

[0061] In summary, the present invention has comprehensively innovated the traditional financial big data storage optimization process through the idea of ​​semantic drive. Multiple key technologies are closely linked and mutually supportive, achieving extraordinary synergy in improving data processing efficiency, ensuring data security and privacy, and optimizing resource allocation and scheduling. This innovative technical solution will surely bring important value to the intelligent upgrade and business innovation of the financial big data platform, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The present invention is a flowchart of a method for optimizing storage of financial big data. DETAILED DESCRIPTION

[0063] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a financial big data optimization storage method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0064] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0065] The following is a detailed description of a specific scheme of a financial big data optimization storage method provided by the present invention in conjunction with the accompanying drawings.

[0066] Example 1

[0067] See also Figure 1 The present invention discloses a method for optimizing the storage of financial big data. The method aims to solve the challenges of storage and management brought by massive heterogeneous data in the financial field, and proposes an innovative solution integrating cross-modal semantic coding, federated learning, consistent hashing and other technologies.

[0068] The method of the present invention comprises the following steps:

[0069] Step 1: Aggregation of multi-source heterogeneous financial data. The data acquisition module is used to obtain structured data (such as transaction records), semi-structured data (such as XML / JSON messages), and unstructured data (such as contract texts, customer photos, etc.) from various financial business systems. The data cleaning module is used to process missing values, outliers, duplicate data, etc. to improve data quality. The format conversion module is used to convert heterogeneous data from different sources and formats into a unified platform standard format, laying the foundation for subsequent processing.

[0070] Step 2: Subject domain division and data classification. Based on the pre-built financial business subject domain knowledge base, the collected massive data is divided into different subject domains such as customers, products, risks, and marketing. The knowledge base adopts the ontology modeling method to formally define the core concepts, attributes, and their relationships of each subject domain. Within each subject domain, it is classified according to the data type characteristics (such as numerical type, text type, image type) to facilitate the design of a targeted semantic coding scheme.

[0071] Step 3: Multimodal semantic encoding: This is one of the core steps of the present invention. Corresponding semantic encoders are designed for different types of financial data, and the effective fusion of cross-modal semantics is achieved by introducing the idea of ​​adversarial learning.

[0072] For numerical structured data, a hybrid semantic encoder is used, combining factorization machine and tree embedding. The factorization machine part learns the implicit interaction between features, while the tree embedding part segments and embeds the features. The two complement each other and enhance the semantic representation ability. Taking customer portrait data as an example, numerical features such as age and income are encoded through this hybrid semantic encoding, which not only obtains simple numerical vectors, but also integrates the association patterns between features.

[0073] For numerical structured data, the factorization machine part of the hybrid semantic encoder is calculated by formula Modeling feature interactions. 0 is the global bias term, w i is the feature weight of the i-th dimension, <v i , v j > is the dot product of the i-th dimension and the j-th dimension feature latent vector. In this way, the high-order correlation information between features is captured. The tree embedding part is obtained by formula

[0074]

[0075] Map each eigenvalue to the corresponding tree node vector. i ) is the unique hot encoding of the i-th dimension feature, μ(l(x i )) is the leaf node vector of the i-th dimension feature in the tree. Tree embedding can adaptively divide the feature space based on the distribution of eigenvalues ​​and improve the representation ability. In practice, the latent vector dimension is usually set to 10 to 100, and the depth of the tree is determined by the number and distribution of eigenvalues, generally 4 to 8 layers.

[0076] For text data, a cross-layer semantic encoder is used, which combines the two language models of Transformer and BERT. The multi-head attention mechanism in Transformer can capture the contextual relevance of the text, while the bidirectional language model of BERT further enhances the semantic modeling capability. The two are intertwined to achieve cross-layer fusion of shallow and deep semantics. Taking the financial contract text as an example, this cross-layer semantic encoding not only extracts the literal information of the contract terms, but also mines the implicit rights and responsibilities, making subsequent semantic retrieval, risk assessment and other tasks more accurate.

[0077] For text data, the Transformer part of the cross-layer semantic encoder is first calculated by formula Calculate the self-attention weight, where Q, K, and V are query, key, and value matrices respectively, and d k is the key vector dimension. Then, by the formula MultiHead(Q, K, V)=Concat(head 1 , ..., head h )W O Implementing multi-head attention;

[0078] head i =Attention(QW i Q , KW i K , VW i V ) represents the i-th attention head, W i Q , W i K , W i V , W O is the learnable matrix parameter, where head i is the i-th attention head, W O is the output projection matrix. Multi-head attention allows the model to learn text representations from different subspaces, enhancing the ability to capture semantics.

[0079] The BERT part is calculated by the formula Modeling the bidirectional dependency of text, where x is the text word embedding sequence, p(x i |x<i, x>i) is the i-th word x given a bidirectional context i The conditional probability, h <i，>i is the hidden state of the ith position. Through bidirectional modeling, rich semantic information is mined. In practical applications, the number of attention heads of Transformer is generally set to 4 to 16, and the number of hidden layers of BERT is 12 to 24. The word embedding dimension is usually selected to be 128 to 512.

[0080] For image data, a hierarchical semantic encoder is used, combined with a convolutional neural network (CNN) and contrastive learning. CNN extracts multi-scale local features of images through stacked convolutions, pooling and other operations; contrastive learning optimizes the semantic consistency of features. The combination of the two captures both fine-grained semantics and global semantics. Taking customer photos as an example, this hierarchical semantic encoding not only depicts local information such as facial features and skin color, but also represents overall semantics such as identity and emotions, laying the foundation for subsequent applications such as identity authentication and marketing recommendations.

[0081] For image data, the CN N of the hierarchical semantic encoder is calculated by the formula Extract multi-scale local features of the image, where is the (i, j)th element of the output of the lth layer convolution, f(·) is the activation function, M l and N l is the height and width of the l-th convolution kernel, and b l are the convolution kernel weight and bias term respectively. The convolution operation can effectively capture the local pattern of the image. The contrastive learning part is implemented by formula

[0082]

[0083] Minimize the cross entropy loss of positive and negative sample pairs, where N is the image batch size, z i is the code of the i-th image, z i，+ is the positive example encoding for data enhancement, and τ is the temperature hyperparameter. Contrastive learning enhances the semantic consistency of image encoding. In practice, CNN generally uses 3 to 5 convolutional layers, and the size of each convolution kernel is between 3×3 and 11×11, with a number of 32 to 256. The temperature parameter of contrastive learning is usually set to 0.1 to 0.5.

[0084] It is worth mentioning that adversarial constraints are introduced in the above-mentioned semantic encoding process. Through the game learning between the discriminator and the encoder, different modal data are prompted to reach consistency in the semantic space, truly realizing cross-modal fusion.

[0085] The adversarial constraints introduced in the above semantic encoding process have the objective functions of the discriminator D and encoder E as follows:

[0086] By minimizing the encoder Make the semantics of each modality consistent; by maximizing the discriminator

[0087] This enables it to accurately identify the modality to which the semantics belongs, forming an adversarial game. Practice has shown that adversarial training can significantly improve the effect of cross-modal semantic fusion.

[0088] Step 4: Semantic enhancement and compression: After obtaining the preliminary semantic representation, its expressive power is further enhanced through external knowledge injection and adaptive weight adjustment, and a semantic cascade encoding network is designed to achieve lossless compression.

[0089] The financial knowledge graph is used to enrich the extension of semantic coding, map the coding to the concept nodes in the knowledge graph, and expand the semantics with the help of the relationship edges in the graph. For example, if the customer code is mapped to the "high net worth customer" node in the knowledge graph, its corresponding financial management preferences, risk tolerance and other attributes can be associated.

[0090] At the same time, the attention mechanism is used to adaptively adjust the weights of semantics at different granularities, highlighting key semantics and weakening secondary semantics. The calculation of this weight comprehensively considers the importance of the semantics itself and its relevance to the global query, making the semantic representation more refined.

[0091] On this basis, a semantic cascade encoding network was designed to compress the semantic representation layer by layer. Each layer of cascade units uses a gating mechanism to adaptively select and combine semantic elements from the previous layer, filter out redundancy, extract essence, and finally form a highly concentrated semantic encoding. Multi-layer cascades make the compression process more refined, while greatly reducing the representation dimension, retaining the original semantics to the greatest extent.

[0092] When using the financial knowledge graph to enhance semantic representation, the semantic code is mapped to the concept nodes on the graph, and the relationship edges between nodes are used to enrich the semantics. Taking the customer code as an example, if it is mapped to the "high net worth customer" node, its attribute nodes such as "risk preference" and "financial management needs" can be associated through the relationship edges to expand the semantic extension. Experience shows that the attention mechanism is the key to improving the semantic representation effect. Attention weight α i The calculation formula is: Where f(·,·) is the attention score function, which can be in the form of vector dot product, concatenation, perceptron, etc. q is the global query vector, which can usually be the mean vector of the semantic encoding. m is the total number of semantic granularities, which can be determined according to task requirements and computing resources, and is generally in the order of 10 to 100. Through the attention mechanism, the importance of different semantic components can be adaptively adjusted to highlight key information.

[0093] The calculation formula of the l-th layer cascade unit in the semantic cascade encoding network is:

[0094] h l =g l (W l h l-1 +b l )⊙σ(U l h l-1 +c l )

[0095] where q l (·) is the mapping function of the first layer, which can be identity mapping, nonlinear transformation, etc.; σ(·) is the gating function, usually the Sigmoid function; ⊙ is the Hadamard product, that is, element-wise multiplication. The gating signal σ(U l h l-1 +c l ) controls the l-1th layer semantic vector h l-1 The selective retention and discarding of each element in the network plays a role in compression. Practice shows that the cascade coding network can better retain key semantics than simple linear compression. The number of layers of the network can be determined according to the compression rate requirements of the task, usually between 2 and 5 layers.

[0096] Step 5: Distributed storage under the federated learning framework. The present invention adopts the idea of ​​federated learning, distributes and stores semantic codes to different nodes according to subject domains, coordinates local training of each node, and updates the global semantic representation model.

[0097] Each local node trains the model based on local data and uses a differential privacy mechanism to process gradient information. This decentralized, privacy-preserving learning paradigm avoids the centralized aggregation of sensitive data and reduces the risk of privacy leakage. The central server is responsible for aggregating the model parameters of local nodes to obtain a globally consistent semantic representation model.

[0098] In particular, the differential privacy gradient perturbation technology is used to inject random noise into the gradient update of local nodes, thereby ensuring the privacy security of the model aggregation process and preventing individual privacy from being leaked from the aggregation parameters.

[0099] In the federated learning framework of the present invention, the local node k is calculated by the formula

[0100]

[0101] Update the semantic encoding model parameters θ k , where η is the learning rate, DP(.) is the differential privacy gradient perturbation function, is the loss function of node k. The central server uses the formula Aggregate the updated model parameters of each node to obtain the global model θ, where K is the total number of nodes, n k is the number of samples of node k, and n is the total number of global samples. Federated learning fully utilizes the data of each node while protecting privacy, improving the generalization performance of semantic representation. It is worth mentioning that differential privacy gradient perturbation By adding random noise to the gradient, the privacy security of the model aggregation process is guaranteed. Common noise mechanisms include Gaussian mechanism and Laplace mechanism. The latter is implemented by adding Gaussian noise that obeys Lap(λ). The noise intensity σ or λ can be determined based on the balance between privacy protection strength and model effect.

[0102] Step 6: Consistent hash index construction. Based on the compressed semantic coding, the consistent hashing method is applied to generate semantic indexes to accelerate query and association analysis. The semantic coding is used as the input of the hash function and mapped to the corresponding hash bucket. Semantically similar coding is mapped to adjacent buckets, making similarity search more efficient.

[0103] Consistent hashing maps the semantic code z to one of m hash buckets, and its hash function is:

[0104] hash(z)=(a·z+b)mod m

[0105] Parameters a and b can be randomly selected to ensure the uniform distribution of hash. The number of buckets m is determined by the data scale and retrieval efficiency requirements, and is usually a prime number close to the total amount of data. To further improve the precision and recall of retrieval, multiple hash functions can be introduced so that similar semantic codes can fall into adjacent buckets in multiple hash tables.

[0106] Step 7: Storage scheduling and automatic scaling. The present invention designs an intelligent storage resource scheduling strategy based on reinforcement learning, which dynamically adjusts the storage location and number of copies of data according to factors such as data popularity and access mode. At the same time, the system load is monitored and predicted in real time to trigger the elastic scaling of the storage cluster, thereby improving resource utilization while ensuring service quality. The strategy model uses indicators such as system throughput and latency as rewards, and optimizes scheduling decisions through continuous attempts.

[0107] The storage resource scheduling strategy of the present invention adopts the policy gradient method in reinforcement learning to optimize the scheduling strategy π by maximizing the following objective function: θ :

[0108]

[0109] Where τ is the scheduling trajectory, γ is the discount factor, and r t is the immediate reward for the t-th step scheduling (such as increased system throughput, reduced latency, etc.). θ is the parameter of the policy network, which is continuously optimized through gradient ascent to obtain the optimal scheduling strategy. At the same time, the input of the policy network also includes monitoring and prediction information of the system load, so that the scheduling decision can respond to load changes in real time and trigger timely resource scaling.

[0110] Step 8: Metadata management and version control. The present invention constructs a unified metadata management module and uses a graph database to construct a logical view of metadata, including data description information, lineage relationship, Schema evolution, etc. The query, subscription and change tracking of metadata are implemented through the interface of the graph database. In addition, the present invention uses a multi-version concurrency control (MVCC) mechanism to perform version management on metadata and compressed semantic coding. Each update will generate a new version to form a version chain, which supports fast historical data backtracking and snapshot analysis. A graph database is used to maintain metadata such as data description information, lineage relationship, Schema evolution, etc. The query, subscription and change tracking of metadata can be easily implemented through the declarative query language of the graph database (such as Cypher, Gremlin, etc.). For example, all upstream data sources of a data table can be queried through the following Cypher statement: The multi-version concurrency control (MVCC) mechanism supports historical tracing and snapshot analysis of data by maintaining a version chain on metadata and semantic coding. Each update operation will generate a new node on the version chain to record meta-information such as update timestamp and operation type. Through the version chain, the state of data at any historical time point can be easily queried and restored to meet the needs of data traceability and auditing. The number of nodes in the version chain can be determined according to the frequency of data updates and the length of historical traceability. When the number of nodes exceeds a certain threshold, the old version can be archived or deleted.

[0111] In summary, the present invention systematically solves the storage problem of financial big data from a semantic perspective, with cross-modal fusion, distributed privacy protection, and adaptive lossless compression as the core. It has innovations in unified representation, efficient storage, and intelligent scheduling of massive heterogeneous data, and has important value for financial data management and data applications.

[0112] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing storage of financial big data, characterized in that: The following steps are involved: Step 1. Aggregation of multi-source heterogeneous financial data: Obtain structured, semi-structured and unstructured data through the data acquisition module, process it using the data cleaning module, and convert it into a unified platform standard format through the format conversion module; Step 2. Subject domain division and data classification: divide the collected data into subject domains based on the financial business subject domain knowledge base and classify them according to data characteristics; Step 3. Multimodal semantic encoding: Design corresponding semantic encoders for different types of data and achieve cross-modal semantic fusion by introducing adversarial constraints; Step 4. Semantic enhancement and compression: The financial knowledge graph is used to enhance the semantic coding, and the weights of semantics of different granularities are adaptively adjusted through the attention mechanism. The semantic cascade encoding network is designed to compress the semantic representation layer by layer. The calculation formula of the l-th layer cascade unit is: h l =g l (W l h l-1 +b l )⊙σ(U l h l-1 +c l ) where h l is the output of the l-th layer cascade unit, g l (·) is the mapping function of the lth layer, σ(·) is the gating function, W l , U l , b l , c l is the learnable parameter of the l-th layer cascade unit, ⊙ is the Hadamard product; Step 5. Distributed storage under the federated learning framework: The semantic coding is stored in different nodes according to the subject domain, and the federated learning framework is used to coordinate the training and updating between nodes. The local node k updates the semantic coding model parameters θ based on the privacy-preserving gradient aggregation method. k : Where η is the learning rate, DP(.) is the differential privacy gradient perturbation function, and L k is the local objective function of node k, and the central server aggregates the updated model parameters of each node to obtain a globally consistent semantic encoding model θ: Among them, K is the total number of nodes, n k is the number of local samples of node k, and n is the total number of global samples; Step 6. Consistent hash index construction: Apply the consistent hashing method on the compressed semantic encoding to generate the index. The definition of the consistent hashing function is: hash(z)=(a·z+b)mod m Among them, z is the compressed semantic code, α and b are hash functions, and the parameter m is the number of hash buckets; Step 7. Storage Scheduling and Automatic Scaling: Design a storage resource scheduling strategy based on reinforcement learning, dynamically optimize the storage location and number of copies of data based on data popularity and access pattern factors, and trigger automatic scaling of the storage cluster based on real-time monitoring and prediction of system load; Step 8. Metadata management and version control: Build a unified metadata management module, organize and store metadata in the form of a graph database, and use a multi-version concurrent control mechanism to perform version management on metadata and compressed semantic coding.

2. The method according to claim 1, characterized in that For numerical structured data, a hybrid semantic encoder based on factor decomposition machine and tree embedding is used, where the formula of the factor decomposition machine part is: in is the predicted output, w0 is the global bias term, and w i is the weight of the ε-th dimension feature, <v i , v j > is the latent vector interaction between the i-th dimension and the j-th dimension feature, x i is the eigenvalue of the i-th dimension; The formula for the tree embedding part is: Where e(x) is the tree embedding output vector, f(x i ) is the one-hot feature of the i-th dimension, encoding μ(l(x i )) is the embedding vector of the leaf node of the i-th dimension feature in the decision tree, l(x i ) is the leaf node index of the i-th dimension feature in the decision tree.

3. The method according to claim 1, characterized in that For text data, a cross-layer semantic encoder based on Transformer and BERT is used, where the formula of the Transformer part is MultiHead(Q,K,V)=Concat(head1,...,head h )W O Among them, Q, K, and V are query, key, and value matrices respectively, and d k is the key vector dimension, head i =Attention(QW i Q , KW i K , VW i V ) represents the i-th attention head, W i Q , W i K , W i V , W O is the learnable matrix parameter; The formula for the BERT part is: where \(x=(x_1,\ldots,x n )\) is the word embedding sequence of the text, \(p(x i |x_{<i},x_{>i})\) is the conditional probability of the \(i\)-th word \(x i \) given the context \(x_{<i}\) and \(x_{>i}\), and \(h <i，>i \) is the hidden state of BERT's bidirectional language model at the \(i\)-th position.

4. The method according to claim 1, characterized in that: For image data, a hierarchical semantic encoder based on convolutional neural network and contrastive learning is used, where the formula of the convolutional neural network part is: in, is the (i, j)th element of the output of the lth layer convolution, f(·) is the activation function, M l and N l is the height and width of the l-th convolution kernel, and b l are the weight and bias of the convolution kernel of the lth layer respectively; The formula for the contrastive learning part is: in, is the contrastive learning loss function, N is the image batch size, z i is the semantic code of the i-th image, z i，+ is the positive semantic encoding of the i-th image, and τ is the temperature hyperparameter.

5. The method according to claim 1, characterized in that In the multimodal semantic encoding step, Introduce the adversarial objective function between the modality discriminator D and the encoder E: Through adversarial training, data of different modalities are forced to achieve distribution alignment in the semantic coding space, thus achieving the goal of cross-modal fusion.

6. The method according to claim 1, characterized in that In the semantic enhancement and compression step, the calculation formula of attention weight is: Among them, α i is the semantic code z for the i-th granularity i is the attention weight, f(·,·) is the attention score function, q is the global query vector, and m is the total number of semantic granularities.

7. The method according to claim 1, characterized in that In the distributed storage step under the federated learning framework, differential privacy gradient perturbation DP(.) is used to ensure the privacy security of the model aggregation process; the consistent hash index maps semantically similar codes to adjacent hash buckets to achieve similarity search.

8. The method according to claim 1, characterized in that ,The reinforcement learning model of the storage resource scheduling ,strategy takes system throughput and latency as reward signals and ,learns the scheduling strategy through trial and error.

9. The method according to claim 1, characterized in that ,The metadata includes the data’s description information, lineage relationships, and Schema evolution. The query, subscription and change tracking of metadata are ,realized through the graph database interface.

10. The method according to claim 1, characterized in that ,The multi-version concurrent control mechanism generates a new version for each update of metadata and ,compressed semantic coding, forming a version chain and supporting ,rapid historical data backtracing and snapshot analysis.

Images