Multi-source data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a multi-source data processing method and device, equipment, a storage medium and a program product, and relates to the field of big data and financial science and technology. The method comprises the following steps: acquiring multi-source heterogeneous data, wherein the multi-source heterogeneous data comprises a set of data from different systems or formats; based on the knowledge graph, performing entity recognition on the multi-source heterogeneous data to obtain entities in the multi-source heterogeneous data; performing relation mapping on the entities to obtain semantic alignment data; and through context reasoning and a dynamic updating mechanism, when the business rule changes, optimizing the semantic alignment data to obtain a target data view. According to the method, the problem of semantic islands of cross-system data is solved, semantic unification and format standardization of multi-source data are realized, a high-quality and traceable unified data view is provided for subsequent data processing, and data fusion efficiency and business suitability are remarkably improved.

Description

Technical Field

[0001] This application relates to the fields of big data and financial technology, and in particular to a multi-source data processing method, apparatus, device, storage medium and program product. Background Technology

[0002] With the acceleration of digital transformation in commercial banks, data has become a core production factor, widely used in business scenarios such as intelligent risk control, precision marketing, customer service, and product recommendation. In existing scenarios, the data sources of commercial banks are complex and scattered, covering core business systems, transaction systems, and external cooperation systems, with significant differences in data formats, including structured data tables, unstructured documents, and semi-structured logs.

[0003] In existing technologies, data processing workflows struggle to achieve semantic alignment and format unification across systems, resulting in low data quality and failing to meet the demands of large models for high-quality, high-density, and standardized data. Traditional data cleaning solutions rely solely on static rules and manual operations, which are insufficient to cover business characteristics and dynamically changing needs. Commercial banks' multi-source, heterogeneous data suffers from format differences and semantic inconsistencies, making cross-system data fusion difficult and preventing the formation of a unified data asset view. Summary of the Invention

[0004] This application provides a multi-source data processing method, apparatus, device, storage medium, and program product to solve the technical problem that commercial banks face difficulties in cross-system data fusion and the inability to form a unified data asset view due to differences in format and semantics in their multi-source heterogeneous data.

[0005] Firstly, this application provides a multi-source data processing method, including:

[0006] Acquire multi-source heterogeneous data, which includes a collection of data from different systems or formats;

[0007] Based on knowledge graphs, entity recognition is performed on multi-source heterogeneous data to obtain entities in the multi-source heterogeneous data.

[0008] Perform relation mapping on entities to obtain semantically aligned data;

[0009] By using contextual reasoning and dynamic update mechanisms, semantically aligned data is optimized when business rules change, resulting in the target data view.

[0010] Secondly, this application provides a multi-source data processing apparatus, comprising:

[0011] The data acquisition module is used to acquire multi-source heterogeneous data, which includes a collection of data from different systems or formats.

[0012] The entity recognition module is used to perform entity recognition on multi-source heterogeneous data based on knowledge graphs, and to obtain the entities in the multi-source heterogeneous data;

[0013] The relation mapping module is used to perform relation mapping on entities to obtain semantically aligned data;

[0014] The data update module is used to optimize semantically aligned data and obtain the target data view when business rules change through contextual reasoning and dynamic update mechanisms.

[0015] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0016] The memory stores the instructions that the computer executes;

[0017] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0018] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0019] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0020] The multi-source data processing method, apparatus, equipment, storage medium, and program products provided in this application acquire heterogeneous data from multiple sources, including structured data tables, unstructured documents, and semi-structured logs. Based on knowledge graph technology, they extract key entities from the data and deduce implicit relationships between entities by mapping the correspondence between entities in different systems and combining contextual information from business scenarios. A dynamic update mechanism adjusts the knowledge graph structure to adapt to changes in business rules, thereby generating a unified data view and ensuring the continuous effectiveness of semantic alignment. This solves the problem of semantic silos in cross-system data, achieves semantic unification and format standardization of multi-source data, provides a high-quality, traceable, unified data view for subsequent data processing, and significantly improves data fusion efficiency and business adaptability. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 A schematic diagram of the network architecture for large-scale model training and application in commercial banks;

[0023] Figure 2A flowchart illustrating a multi-source data processing method provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of a multi-source data processing device provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0029] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0030] It should be noted that the multi-source data processing methods, apparatus, devices, storage media, and program products provided in this application can be used in the fields of big data and fintech, as well as in any other field. The application fields of the multi-source data processing methods, apparatus, devices, storage media, and program products in this application are not limited.

[0031] The specific application scenarios of this application are large-scale model training and application scenarios in commercial banks. Figure 1 A schematic diagram of the network architecture for large-scale model training and application in commercial banks, such as... Figure 1 As shown, the network architecture for large-scale model training and application in commercial banks includes: a data acquisition layer (including core business systems, external cooperation platforms, etc.), a data processing layer (including knowledge engineering toolchains, annotation and quality inspection modules, etc.), and a data application layer (including pre-training / fine-tuning / evaluation datasets and professional datasets, etc.). In business scenarios such as precision marketing, commercial banks need to extract high-value information from multi-source heterogeneous data and achieve automated decision support through large-scale models.

[0032] In existing technologies, commercial bank data is scattered across different systems (such as counter systems) and has significantly different formats (structured tables, unstructured documents), resulting in data fusion merely "transferring" data and failing to achieve cross-system semantic alignment. Furthermore, data cleaning relies on static rules (such as missing value imputation) without incorporating business rules, leading to unidentified compliance risks. Regarding labeling systems, general classification labels cannot cover the needs of financial scenarios (such as customer risk level classification), and manual labeling is inefficient and lacks standardized practices. Ultimately, the dataset lacks a dynamic update mechanism, failing to adapt to business changes (such as updates to customer repayment records) and industry regulation adjustments, resulting in insufficient timeliness of model training data and impacting the accuracy of downstream business decisions. The main shortcomings include: low data quality: due to reliance on static rules and manual cleaning, complex data errors (such as semantic inconsistencies or implicit relationship errors) are easily overlooked, leading to insufficient data accuracy. Low efficiency: manual data cleaning and integration are time-consuming and labor-intensive, unable to adapt to the large volume and frequent updates of commercial bank data, resulting in long construction cycles. Lack of dynamic adaptability: existing solutions struggle to cope with changes in data source structure or new business needs, with lagging rule updates leading to unstable dataset quality. Incomplete evaluation: Data quality assessments are often based on simple metrics, lacking multi-dimensional and real-time evaluation mechanisms, which affects the reliability and usability of the dataset. Poor interpretability: The dataset construction process lacks transparency and documentation, hindering auditing and traceability, and reducing business trust.

[0033] The multi-source data processing method, apparatus, device, storage medium, and program products provided in this application acquire multi-source heterogeneous data, including structured data tables, unstructured documents, and semi-structured logs; based on knowledge graph technology, extract key entities from the data, and deduce implicit relationships between entities by mapping the correspondence between entities in different systems and combining contextual information of business scenarios; and adjust the knowledge graph structure through a dynamic update mechanism to adapt to changes in business rules, thereby generating a unified data view, aiming to solve the above-mentioned technical problems of existing technologies.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Figure 2 This is a flowchart illustrating a multi-source data processing method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes:

[0036] S201. Obtain multi-source heterogeneous data.

[0037] In the embodiments of this application, multi-source heterogeneous data includes a collection of data from different systems or formats.

[0038] In one example, multi-source heterogeneous data is centralized to a knowledge base platform and stored according to business scenarios through methods such as interface integration, batch file uploads, and manual data entry. For instance, multi-source heterogeneous data may include: documents accumulated by business experts and offline experience summaries (such as departmental knowledge, expert experience, work records, and frequently asked questions) obtained through manual entry; core data that can be shared across the entire bank (customer profile tags, market analysis reports, etc.) obtained through data integration; highly sensitive / regional data (such as regulatory reports) obtained through batch file integration; and real-time business support scenarios obtained through interface integration, including T+0 update requirements.

[0039] Specifically, acquiring multi-source heterogeneous data can include: documents accumulated by business experts and summaries of offline experience, uploaded by various business departments or R&D departments through the group's knowledge base platform. Core data shared across the entire bank is requested by business departments, and the R&D department configures an automated workflow to map the data lake table to knowledge base directory nodes for incremental synchronization. Highly sensitive / regional data is compiled by the R&D department according to templates, including file and metadata lists, and uploaded in batches through the knowledge base platform; the system automatically identifies the directory mounting location. Real-time business support scenarios are handled by the R&D department through API interfaces for real-time / scheduled data integration, automatically injecting data into the knowledge base directory.

[0040] S202. Based on knowledge graphs, perform entity recognition on multi-source heterogeneous data to obtain entities in the multi-source heterogeneous data.

[0041] In one example, based on a knowledge graph, a knowledge engineering toolchain is used to clean, de-identify, label, and perform quality checks on the data. Natural Language Processing (NLP) is used to extract key entities (such as "customer ID" and "transaction amount") from multi-source data. For example, missing value imputation and outlier removal (cleaning) are performed on customer transaction records, sensitive information in credit reports is de-identified, "risk level" tags are added to expert experience documents (labeling), and quality checks are conducted collaboratively by business experts and the technology team (such as manual sampling).

[0042] S203. Perform relation mapping on the entities to obtain semantically aligned data.

[0043] In one example, key entities in multi-source heterogeneous data are mapped to correspondences across different systems to obtain semantically aligned data. Graph neural networks are used to infer implicit relationships between entities, mapping these relationships to obtain semantically aligned data. This generates a structured, standardized, high-quality corpus, supporting the pre-training, fine-tuning, and inference needs of large models.

[0044] S204. Through contextual reasoning and dynamic update mechanisms, the semantically aligned data is optimized when business rules change to obtain the target data view.

[0045] In one example, based on the contextual information of the business scenario, a dynamic update mechanism incrementally learns and updates the knowledge graph structure as business rules change (such as adding new data fields), ensuring the continuous effectiveness of semantic alignment and obtaining the target data view. A semantic alignment framework for multi-source heterogeneous data is built based on the knowledge graph, achieving semantic unification of data across systems through entity recognition, relationship mapping, and contextual reasoning. Specifically, knowledge graph technology utilizes natural language processing to extract key entities from multi-source heterogeneous data, and based on the contextual information of the business scenario, graph neural networks infer the implicit relationships between entities. The dynamic update mechanism incrementally learns and updates the knowledge graph structure as business rules change, ensuring the continuous effectiveness of semantic alignment.

[0046] Specifically, the target data view may include: (1) Pre-training dataset: The pre-training dataset is the "foundation" of the financial big model, mainly used for continuous pre-training of the model, injecting core banking business knowledge and general financial knowledge into the model. By continuing pre-training on a large-scale, high-quality basic corpus, the model can learn language patterns, professional terms and common sense knowledge in the financial field, thus laying a good foundation for subsequent instruction fine-tuning. The pre-training dataset is mainly used in the pre-training stage of the model. The cleaned large-scale text is input into the big model for continuous pre-training, so that the big model learns language patterns and knowledge in the financial field. Continuous pre-training can be carried out on the basis of a general pre-training model. By continuing training on financial corpus, the model parameters will be further adapted to the financial context. This dataset is usually processed by technology and does not require business intervention.

[0047] For example, the original multi-source heterogeneous data: Knowledge engineering aims to build a knowledge system adapted to the era of large-scale models, reconstruct the complete chain of knowledge production and extraction, model training, and scenario application, establish a full lifecycle management system for knowledge assets ("acquisition, construction, management, and use"), and improve knowledge asset management and organizational support. The resulting target data view, i.e., a sample of pre-trained data, is: {"text":"Knowledge engineering aims to build a knowledge system adapted to the era of large-scale models, reconstruct the complete chain of knowledge production and extraction, model training, and scenario application, establish a full lifecycle management system for knowledge assets ("acquisition, construction, management, and use"), improve knowledge asset management and organizational support, promote the bank's "Leading AI+" action plan to improve quality and efficiency, and increase the cultivation of new financial productivity."}

[0048] (2) Supervised Fine-Tuning Corpus (SFT): The core function of the Supervised Fine-Tuning Corpus is to teach the model to understand and follow instructions in specific scenarios, enabling it to perform tasks and answer questions according to user requirements. After pre-training on a general corpus, the model already possesses certain language abilities, but may not be familiar with instructions or tasks in specific formats. Through supervised training on the Supervised Fine-Tuning Corpus, the model will learn to "understand" instructions, "understand" task requirements, and provide answers in the expected format.

[0049] For example, raw, multi-source, heterogeneous data: Knowledge engineering aims to build a knowledge system adapted to the era of large-scale models, reconstruct the complete chain of knowledge production and refinement, model training, and scenario application, establish a full lifecycle management system for knowledge assets ("collection, construction, management, and use"), and improve knowledge asset management and organizational support. The resulting target data view, i.e., a sample of fine-tuned data: (Question-answer pair): {"instruction":"What are the construction goals of knowledge engineering?","output":"Knowledge engineering aims to build a knowledge system adapted to the era of large-scale models, reconstruct the complete chain of knowledge production and refinement, model training, and scenario application, establish a full lifecycle management system for knowledge assets ("collection, construction, management, and use"), and improve knowledge asset management and organizational support. {"instruction":"What is the core role of the group knowledge base?","output":"The group knowledge base is an enterprise-level knowledge management platform used to centrally collect, store, and manage the enterprise's original knowledge assets. Through data cleaning, standardization, and access control, it provides compliant and controllable corpus resources for model training."}

[0050] (3) Thinking Chain Dataset: The core objective of the Thinking Chain Dataset is to teach the model to master the reasoning process of complex problems, enabling it not only to provide answers but also to demonstrate the underlying logic. In the financial field, many problems require multi-step reasoning or the integration of multiple information sources to solve, such as financial calculations and risk assessments. By providing training samples that include reasoning steps, the model will learn to explicitly perform logical deductions when answering questions, thereby improving its analytical ability and accuracy in complex financial problems.

[0051] For example, raw, multi-source, heterogeneous data: Knowledge Engineering aims to build a knowledge system adapted to the era of large-scale models, reconstruct the complete chain of knowledge production and refinement, model training, and scenario application, establish a full lifecycle management system for knowledge assets ("collection, construction, management, and use"), and improve knowledge asset management and organizational support. The resulting target data view, i.e., a sample of the thought chain data: (question-answer pairs + thought process) {"instruction":"What are the construction goals of Knowledge Engineering?","input":"The core positioning of Knowledge Engineering is an enterprise-level knowledge management platform for large-scale models; its core function is to centrally collect and store enterprise raw knowledge assets, provide data cleaning and standardization (to ensure corpus quality), and implement access control (to ensure data security)","output":"The platform manages enterprise knowledge assets throughout the entire process (collection → cleaning → control), ultimately outputting high-quality corpus to support large-scale model training."}

[0052] (4) Evaluation Dataset: The evaluation dataset is used to evaluate the performance of the financial large model in various aspects. After the model is trained, a series of tests are needed to measure whether its performance meets expectations, identify shortcomings, and provide a basis for subsequent improvements. The evaluation dataset is independent of the training corpus and can reflect the model's generalization ability and true level on unseen data.

[0053] (5) X Inference Datasets: Inference datasets focus on the practical application of models and empowerment of personnel. This refers to various structured or unstructured industry data resources for model application, in addition to the four types of datasets for model training. This part of the knowledge provides real-time financial data and information, serving as an important supplement to textual corpora. By connecting to inference datasets, models can obtain the latest numerical data, real-time market data, and detailed records, thereby answering questions that rely on the latest data and ensuring the timeliness and accuracy of the answers. The main service types include: through knowledge engineering processing, vectorizing unstructured document-type knowledge from within and outside the enterprise into a format that the model can understand and read for external use; this is currently the most frequently used dataset in various business scenarios. Professional datasets: In-house data lake tables, which generate SQL through the model for data retrieval.

[0054] In one implementation scenario, raw knowledge such as operating procedures, dialogue records, and market information from various systems within the industry is collected and automatically tagged using pre-defined directory labels, achieving a unified classification of raw knowledge assets and forming manageable knowledge asset input. Secondly, a standardized knowledge engineering toolchain is used to clean, de-identify, label, and quality-check the raw knowledge, processing it into corpus that can be understood by large models. Subsequently, the processed knowledge assets are hierarchically classified and archived to construct a systematic knowledge structure, forming a navigable and callable knowledge system for different business scenarios. Finally, knowledge is selected and assembled according to specific application scenarios (such as pre-training, fine-tuning, testing, or inference) to form a high-quality dataset that meets the needs of model training, optimization, and validation. Through this chain, enterprises achieve closed-loop management of the entire lifecycle from raw business knowledge to model-usable data, ensuring that knowledge is controllable, traceable, and has direct application value for business scenarios.

[0055] For example, the dataset construction adopts a 6-step process: (1) Data design: Focus on verifying the value of the data and the technical feasibility. Through multi-role collaborative evaluation, output test datasets and technical solution conclusions to ensure the rationality and effectiveness of data access. (2) Access evaluation: Focus on verifying the value of the data and the technical feasibility. Through multi-role collaborative evaluation, output test datasets and technical solution conclusions to ensure the rationality and effectiveness of data access. Business personnel put forward requirements; product managers evaluate business value; developers conduct feasibility evaluation; architects design and make decisions on technical paradigms. (3) Standard quality inspection: Produce high-quality, business-approved datasets. Through dual-line collaboration of technology and business, ensure the accuracy of data annotation and scenario adaptability. (4) Model verification: The core goal of this stage is to safely and efficiently put the verified model into the production environment, and ensure that the model solves practical problems through business acceptance. Through standardized technology deployment and scenario-based business acceptance, ensure the smooth transition of the dataset from model to application, and establish a continuous optimization closed loop. (5) System launch: The goal is to integrate the model with the business system, realize production calls, provide services to users, and open up the operation log path to ensure that the operation log can be used for subsequent model optimization. (6) Scenario operation: By collecting operation logs and user feedback, continuously evaluate the model performance, identify problems and potential, and optimize the dataset and model capabilities in a targeted manner.

[0056] The multi-source data processing method provided in this embodiment acquires heterogeneous data from multiple sources, including structured data tables, unstructured documents, and semi-structured logs. Based on knowledge graph technology, it extracts key entities from the data and derives implicit relationships between entities by mapping the correspondence between entities in different systems and combining contextual information from business scenarios. A dynamic update mechanism adjusts the knowledge graph structure to adapt to changes in business rules, thereby generating a unified data view and ensuring the continuous effectiveness of semantic alignment. This solves the problem of semantic silos in cross-system data, achieves semantic unification and format standardization of multi-source data, provides a high-quality, traceable, unified data view for subsequent data processing, and significantly improves data fusion efficiency and business adaptability.

[0057] Optionally, through contextual reasoning and dynamic update mechanisms, the semantically aligned data can be optimized to obtain the target data view when business rules change. This includes: using graph neural networks to obtain implicit relationships between entities in the semantically aligned data based on contextual information of the business scenario; and updating the structure of the knowledge graph based on the implicit relationships through incremental learning when business rules change to obtain the target data view.

[0058] In one example, after generating semantically aligned data, implicit relationships between entities are inferred using a graph neural network based on contextual information from the business scenario. Subsequently, through an incremental learning mechanism, the knowledge graph structure is dynamically adjusted when business rules change (such as adding new data fields) to ensure the continued effectiveness of semantic alignment. This process supplements semantic relationships that are difficult to identify using traditional methods through contextual reasoning and adapts to business evolution through incremental learning, ultimately forming a unified data view.

[0059] By combining graph neural networks with incremental learning, this method achieves deep mining and dynamic adaptation of implicit relationships in business scenarios. Graph neural networks can deduce complex relationships between entities based on contextual information, compensating for the shortcomings of traditional rule mapping; the incremental learning mechanism ensures that the knowledge graph is updated in real time as business rules change, avoiding the lag in semantic alignment. This method not only improves the semantic consistency of multi-source data fusion but also enhances the system's responsiveness to business evolution, providing a more dynamically adaptable unified data view for downstream data processing.

[0060] Optionally, the structure of the knowledge graph can be updated through incremental learning, including: when adding a new data field, expanding the knowledge graph nodes based on the semantics of the new data field; and / or, when business rules change, adjusting the edges of the knowledge graph based on the semantic relationships of the business rules.

[0061] In one example, incremental learning or rule updates continuously optimize the knowledge graph structure as business rules change. During the incremental learning update of the knowledge graph structure, if a new data field (such as "customer credit score") is added, a corresponding node is added through semantic expansion. If business rules change (such as adjustments to credit compliance logic), the relationships between entities are modified through semantic relationship adjustments. This process ensures the continued effectiveness of semantic alignment by dynamically modifying the nodes and edges of the knowledge graph.

[0062] For example, semantically extended knowledge graph nodes include adding entity nodes to the knowledge graph to reflect semantic changes in data fields. For instance, when a "customer credit score" field is added, a "customer credit score" node is added to the knowledge graph. Semantic relationship adjustments to knowledge graph edges include modifying the relationships between entities in the knowledge graph to adapt to rule changes. For example, when rule A is updated, the logical weight of the "entity A - entity B" relationship is adjusted.

[0063] By refining semantic expansion and semantic relationship adjustments, the dynamic adaptability of knowledge graphs is enhanced. Semantic expansion ensures that the semantics of newly added fields are incorporated into the data fusion process, avoiding data omissions; semantic relationship adjustments guarantee logical consistency after rule changes, improving the system's compatibility with business evolution. This method significantly optimizes the flexibility and stability of multi-source data fusion, providing a more accurate unified data view for complex business scenarios.

[0064] Optionally, after acquiring multi-source heterogeneous data, the method also includes: performing compliance verification and anomaly detection on the multi-source heterogeneous data based on a dynamic rule base.

[0065] In one example, an intelligent data cleaning engine with embedded business rules is built. This engine uses a dynamic rule base and anomaly detection algorithms to perform compliance checks and anomaly detection on multi-source heterogeneous data. After acquiring the multi-source heterogeneous data, compliance checks are performed based on the dynamic rule base, and outliers or logical contradictions are detected through semantic comparison using analysis or semantic recognition models.

[0066] For example, a dynamic rule base includes: a scalable database that stores business rules and supports real-time updates. For instance, the dynamic rule base contains a "Entity A - Entity B" rule and is automatically updated when industry rules change. Compliance verification includes: verifying whether multi-source heterogeneous data conforms to predefined business rules or regulatory requirements. For example, verifying whether "Entity A - Entity B" meets the compliance indicators in Rule A. Anomaly detection includes: identifying outliers or logical contradictions in multi-source heterogeneous data. For example, detecting abnormal transaction amounts through analysis, or identifying logical contradictions in text through semantic recognition models.

[0067] By combining a dynamic rule base with anomaly detection algorithms, the compliance and business adaptability of data cleaning are improved. The dynamic rule base ensures that the cleaning logic is closely matched with the characteristics of banking business, avoiding the lag of static rules; the anomaly detection algorithm can identify hidden errors that are difficult to find by traditional methods, thereby significantly optimizing data quality and providing highly reliable input data for subsequent model training.

[0068] Optionally, the method also includes: generating labels corresponding to the target data view based on a preset scene label library and using an automated labeling system, wherein the automated labeling system is used to extract information and match labels for the target data view.

[0069] In one example, after generating a unified data view, labels are generated for the data based on a contextualized label library and an automated labeling system. Semantic recognition technology extracts key information (such as "probability intervals"), a rule engine matches predefined labels (such as "customer risk level"), and business experts validate and optimize the labeling logic.

[0070] For example, a scenario-based tag library includes a predefined multi-level tag system for specific business scenarios. For instance, a scenario-based tag library might include financial scenario tags such as "customer risk level" and "probability range." An automated tagging system includes a system that automatically extracts key information and matches tags using semantic recognition technology and a rule engine. For example, it might use a semantic recognition model to extract "probability ranges" and match them with predefined tags.

[0071] By combining a scene label library with an automated annotation system, the scene adaptability of data annotation is enhanced. The scene label library ensures that the labels are highly matched with the needs of financial business, while the automated annotation system improves annotation efficiency through natural language processing technology and a rule engine. Ultimately, this provides high-quality, scene-specific labeled data for model training, enhancing the model's generalization ability in financial business scenarios.

[0072] Optionally, the method further includes: acquiring real-time logs, which are used to characterize logs related to the data quality of the target data view; obtaining quality monitoring results corresponding to the target data view based on the real-time logs and in combination with preset multi-dimensional indicators; and optimizing the target data view based on the quality monitoring results.

[0073] In one example, quality monitoring of the target data view is performed based on real-time log collection and multi-dimensional indicator analysis. After generating the target data view, quality-related data is acquired through real-time log collection, and multi-dimensional indicators (such as completeness and accuracy) are calculated in real time using a streaming computing framework. The target data view is then further processed based on the dynamic quality monitoring results (such as anomaly alerts). This further processing (such as re-cleaning or labeling) of the target data view is then performed, and data quality is optimized through verification by business experts.

[0074] For example, real-time log collection includes: collecting data quality-related logs in real time from business systems, model applications, and user feedback. For instance, collecting transaction logs, prediction result logs, and user operation logs. Multi-dimensional indicator analysis includes: calculating dynamic evaluation indicators for coverage completeness, accuracy, and compliance. For instance, calculating indicators such as the proportion of missing fields, prediction result deviation rate, and sensitive field anonymization rate. Reprocessing includes: re-cleaning, labeling, or formatting the data based on quality monitoring results. For instance, when quality monitoring finds that the proportion of missing fields exceeds the standard, the data is re-cleaned.

[0075] By combining real-time logs with multi-dimensional metrics for analysis, the timeliness and stability of the dataset are improved. Real-time log collection ensures that quality monitoring covers the entire process, while multi-dimensional metric analysis provides a comprehensive quality assessment. Ultimately, this enables dynamic updates of the dataset and synchronous adaptation to business changes, enhancing the reliability of model decisions in dynamic business environments. Continuous optimization of the dataset is achieved by optimizing the target data view based on quality monitoring results. Dynamic quality monitoring ensures timely detection of problematic data, and the reprocessing mechanism improves data quality through a closed-loop optimization process. Ultimately, this provides a high-quality, real-time updated unified data view for large models, significantly enhancing the model's generalization ability and reliability in complex business environments.

[0076] Optionally, before the step of obtaining multi-source heterogeneous data, the method further includes: performing distributed training and privacy protection processing on the multi-source heterogeneous data based on a federated learning framework to obtain processed multi-source heterogeneous data.

[0077] In one example, before acquiring multi-source heterogeneous data, distributed training (such as horizontal federated learning) is performed on the multi-source heterogeneous data using a federated learning framework, and differential privacy or homomorphic encryption techniques are used to protect the privacy of the multi-source heterogeneous data.

[0078] For example, federated learning frameworks are distributed machine learning frameworks that allow data to be shared during local model training. For instance, horizontal federated learning allows different business units to collaboratively train models without data leaving their local locations. Privacy protection measures include protecting data privacy through differential privacy or homomorphic encryption. For example, differential privacy anonymization is used on sensitive data such as customer transaction records to prevent the leakage of sensitive information.

[0079] By combining federated learning with privacy protection, the compliance and security of data sharing are enhanced. The federated learning framework breaks down data silos between departments, while privacy protection technologies ensure that sensitive information is not leaked. Ultimately, this provides an efficient and secure data collaboration mechanism for cross-departmental collaborative modeling, promoting the large-scale application of large models in financial scenarios. Through the combination of distributed training and privacy protection, efficient data sharing is achieved under compliant conditions.

[0080] Figure 3 This is a schematic diagram of the structure of a multi-source data processing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the multi-source data processing device 30 provided in this embodiment includes:

[0081] The data acquisition module 301 is used to acquire multi-source heterogeneous data, which includes a collection of data from different systems or formats.

[0082] The entity recognition module 302 is used to perform entity recognition on multi-source heterogeneous data based on knowledge graphs to obtain entities in the multi-source heterogeneous data;

[0083] The relation mapping module 303 is used to perform relation mapping on entities to obtain semantically aligned data;

[0084] The data update module 304 is used to optimize semantically aligned data and obtain the target data view when business rules change through contextual reasoning and dynamic update mechanisms.

[0085] In one possible implementation, the data update module 304 is specifically used to: obtain the implicit relationships between entities in semantically aligned data based on the context information of the business scenario and using a graph neural network; and update the structure of the knowledge graph based on the implicit relationships and through incremental learning when the business rules change, to obtain the target data view.

[0086] In one possible implementation, the data update module 304 is further specifically used to: expand the knowledge graph nodes based on the semantics of the newly added data fields when adding new data fields; and / or adjust the edges of the knowledge graph based on the semantic relationships of the business rules when business rules change.

[0087] In one possible implementation, the multi-source data processing device is also specifically used to: perform compliance verification and anomaly detection on multi-source heterogeneous data based on a dynamic rule base.

[0088] In one possible implementation, the multi-source data processing device is further specifically used to: generate labels corresponding to the target data view based on a preset scene label library and using an automated labeling system, wherein the automated labeling system is used to extract information and match labels on the target data view.

[0089] In one possible implementation, the multi-source data processing device is further specifically used to: acquire real-time logs, which are used to characterize logs related to the data quality of the target data view; based on the real-time logs and combined with preset multi-dimensional indicators, obtain the quality monitoring results corresponding to the target data view; and optimize the target data view based on the quality monitoring results.

[0090] In one possible implementation, the multi-source data processing device is further specifically used to: perform distributed training and privacy-preserving processing on multi-source heterogeneous data based on a federated learning framework, to obtain processed multi-source heterogeneous data.

[0091] The multi-source data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0092] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 may include a memory 401 and a processor 402. Optionally, the electronic device may also include a transceiver 403, wherein the memory 401 and the processor 402 communicate with each other; for example, the memory 401, the processor 402 and the transceiver 403 may communicate via a communication bus 404, the memory 401 is used to store a computer program, and the processor 402 executes the computer program to implement the method of the above embodiments.

[0093] Optionally, the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps in the method embodiments disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0094] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0095] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0096] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0097] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0101] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0104] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0105] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0106] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0107] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0108] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0109] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0110] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multi-source data processing method, characterized in that, The method includes: Acquire multi-source heterogeneous data, which includes a collection of data from different systems or formats; Based on the knowledge graph, entity recognition is performed on the multi-source heterogeneous data to obtain the entities in the multi-source heterogeneous data; Perform relation mapping on the entities to obtain semantically aligned data; By using contextual reasoning and dynamic update mechanisms, the semantically aligned data is optimized when business rules change, resulting in a target data view.

2. The method according to claim 1, characterized in that, The optimization of the semantically aligned data through contextual reasoning and dynamic update mechanisms to obtain the target data view when business rules change includes: Based on the contextual information of the business scenario, the implicit relationships between entities in the semantically aligned data are obtained using a graph neural network. When business rules change, the structure of the knowledge graph is updated through incremental learning based on the implicit relationships to obtain the target data view.

3. The method according to claim 2, characterized in that, The step of updating the structure of the knowledge graph through incremental learning includes: When adding a new data field, the knowledge graph node is expanded based on the semantics of the new data field; And / or, when business rules change, the edges of the knowledge graph are adjusted based on the semantic relationships of the business rules.

4. The method according to claim 1, characterized in that, After acquiring the multi-source heterogeneous data, the method further includes: Based on a dynamic rule base, compliance verification and anomaly detection are performed on the multi-source heterogeneous data.

5. The method according to claim 1, characterized in that, The method further includes: Based on a preset scene tag library, an automated annotation system is used to generate tags corresponding to the target data view. The automated annotation system is used to extract information and match tags for the target data view.

6. The method according to claim 1, characterized in that, The method further includes: Obtain real-time logs, which are used to characterize logs related to the data quality of the target data view; Based on the real-time logs and combined with preset multi-dimensional indicators, the quality monitoring results corresponding to the target data view are obtained. Based on the quality monitoring results, the target data view is optimized.

7. The method according to any one of claims 2-6, characterized in that, Prior to the step of acquiring multi-source heterogeneous data, the method further includes: Based on the federated learning framework, distributed training and privacy protection processing are performed on the multi-source heterogeneous data to obtain the processed multi-source heterogeneous data.

8. A multi-source data processing device, characterized in that, The device includes: The data acquisition module is used to acquire multi-source heterogeneous data, which includes a collection of data from different systems or formats; An entity recognition module is used to perform entity recognition on the multi-source heterogeneous data based on a knowledge graph, so as to obtain the entities in the multi-source heterogeneous data; The relationship mapping module is used to perform relationship mapping on the entities to obtain semantic alignment data; The data update module is used to optimize the semantically aligned data and obtain the target data view when business rules change through contextual reasoning and dynamic update mechanisms.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images