Financial product data processing method, system and device and readable storage medium

Abstract

The embodiment of the invention relates to the field of micro-services, and provides a financial product data processing method, system and device and a readable storage medium, and the method comprises the steps: creating a project research and development activity process on a development, operation and maintenance integrated platform, and filling in system information, a production date and a research and development activity purpose; based on a project research and development activity process, creating a development branch and a design branch on the model design management platform; carrying out physical model change on the design branch, submitting auditing after change, and combining the content of the design branch to the development branch after passing auditing; and a database change script is generated, the generated script is audited, and the script is published to the code hosting platform after passing the audit. According to the embodiment of the invention, a project research and development activity process can be well fused, so that a physical model design change activity is managed by the project research and development activity, and a forward release behavior of a model designer can be ensured, thereby achieving an expected effect.

Description

Technical Field

[0001] The present application relates to the technical field of financial data processing, and in particular to a financial product data processing method, system, device and readable storage medium. Background Art

[0002] With the rapid development of the financial market, the demand for financial products from financial institutions and investors has increased, and a large number of financial products have appeared on the market. These financial products are provided by different financial institutions, with different data formats and contents, and there are significant differences. When publishing financial product information, each financial institution uses different naming rules, coding standards and data structures, which makes cross-institutional data analysis and processing complicated and inefficient. Existing technologies mainly rely on manual methods for data cleaning and integration, which is inefficient and error-prone, and cannot meet the rapidly changing market needs.

[0003] To this end, there is an urgent need to propose a technical solution that can achieve efficient cleaning and integration of wealth management product data provided by different financial institutions, solve the inconsistency of product naming and coding through intelligent algorithms, improve the level of automation in data processing, reduce manual intervention, and improve data accuracy and processing speed. Summary of the invention

[0004] The purpose of this application is to achieve efficient cleaning and integration of financial product data provided by different financial institutions; and to solve the inconsistency of product naming and coding through intelligent algorithms. And to provide a unified and scalable data model to manage the merged financial product information; improve the automation level of data processing, reduce manual intervention, and improve data accuracy and processing speed. For the above purposes, the technical solutions adopted by this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for processing financial product data, the method comprising:

[0006] Step S1: Data source analysis and preprocessing, including data source identification, preliminary data cleaning and cross-border financial product data processing;

[0007] Step S2: Intelligent field identification and fusion algorithm, used to identify similar fields of data from different sources and standardize and fuse them;

[0008] Step S3: data cleaning logic, further cleaning and optimizing the processed data;

[0009] Step S4: Data integration and output: The cleaned and processed data are integrated into a standard data model and output through a flexible interface.

[0010] Optionally, in step S1, the preliminary data cleaning includes the following sub-steps:

[0011] Missing value handling, including detecting and handling missing values ​​in the data;

[0012] Format validation, including verifying that the format of key fields is correct;

[0013] Data integrity checks to ensure that all key fields have valid values;

[0014] Outlier detection, detects and corrects outliers through automated data anomaly detection models.

[0015] Optionally, in step 2, the intelligent field recognition and fusion algorithm includes the following sub-steps:

[0016] Field mapping and matching: Use the BERT model to semantically standardize field names in different data sources and calculate the similarity between different fields.

[0017] Field matching model, using random forest model to match fields;

[0018] Unify the generation of product codes and generate unique identification codes for financial products based on multiple key fields.

[0019] Optionally, in step 3, the data cleaning logic includes the following sub-steps:

[0020] Duplicate items are processed to filter out the most relevant financial product items;

[0021] Date screening logic, screening products based on investment start and end dates;

[0022] Automated data anomaly detection and correction, automatically identifying and correcting outliers in the data.

[0023] Optionally, in step 4, the data integration and output includes the following sub-steps:

[0024] Data integration, integrating cleaned, field-matched, anomaly-detected, and corrected data into a unified data model;

[0025] Data storage and retrieval, storing the integrated data in a relational database;

[0026] Output interface design, design data output interface, the output interface supports API and multiple data formats;

[0027] Data access and update mechanism to achieve real-time data access and batch data export from other systems or modules.

[0028] In a second aspect, the present invention provides a financial product data processing system, the system comprising: a data input module, a data preprocessing module, an intelligent field recognition and fusion module, a data cleaning and anomaly detection module, a data integration and unified data model generation module, and a data output and access interface, wherein:

[0029] Data input module, used to receive and store raw data from different data sources to provide input for subsequent processing;

[0030] A data preprocessing module, used to perform data source identification, preliminary data cleaning and data format standardization;

[0031] Intelligent field identification and fusion module, used to identify similar fields in data from different sources and standardize and fuse them;

[0032] Data cleaning logic module, used to clean and optimize processed data;

[0033] A data integration and unified data model generation module is used to integrate the cleaned and fused data into a unified data model and store it in the database; at the same time, it generates a data set that meets the data output standard;

[0034] Data output and access interface, used to provide API interface and multiple data formats.

[0035] Optionally, the data preprocessing module includes:

[0036] Detect and handle missing values ​​in data;

[0037] Verify that the key fields are in the correct format;

[0038] Ensure that all key fields have valid values;

[0039] Detect and correct outliers with automated data anomaly detection models.

[0040] Optionally, the intelligent field recognition and fusion module includes:

[0041] Use the BERT model to perform semantic standardization on field names in different data sources, and generate a unified product code through multiple hash checks; perform field mapping and matching at the same time, generate a unified field mapping table, and calculate the similarity between different fields;

[0042] A random forest model is used to match the fields;

[0043] Generate a unique identification code for a financial product based on multiple key fields.

[0044] In a third aspect, the present invention provides an electronic device, including a processor and a memory;

[0045] The memory is used to store operation instructions;

[0046] The processor is used to execute the above-mentioned financial product data processing method by calling the operation instruction.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the above-mentioned method for processing financial product data.

[0048] The financial product data processing solution disclosed in the embodiments of this specification has the following beneficial effects through the steps of data source analysis and preprocessing; intelligent field recognition and fusion algorithm; data cleaning logic and data integration and output:

[0049] (1) Improve the accuracy and availability of financial product data and ensure data consistency and integrity through innovative data processing technology.

[0050] (2) Simplify the comparison and analysis process of financial products, help financial institutions improve decision-making efficiency and enhance market responsiveness.

[0051] (3) Support a wider range of data sources and product types, enhance the applicability and flexibility of the system, and meet the needs of different financial institutions.

[0052] (4) Reduce labor costs through automated processing, enhance the continuity and stability of data processing, and improve overall work efficiency BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0054] Figure 1 A flowchart of a method for processing financial product data provided by an embodiment of the present application;

[0055] Figure 2 A schematic diagram of the structure of a financial product data processing system provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0058] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0059] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0060] The core of the present invention is to provide a method for processing financial product data. Figure 1 The flowchart of the method for processing financial product data provided by this embodiment is shown in FIG. Figure 1 The technical solution of this embodiment is described in detail:

[0061] Step S1: Data source analysis and preprocessing, including data source identification, preliminary data cleaning and cross-border financial product data processing;

[0062] Step S2: Intelligent field identification and fusion algorithm, used to identify similar fields of data from different sources and standardize and fuse them;

[0063] Step S3: data cleaning logic, further cleaning and optimizing the processed data;

[0064] Step S4: Data integration and output: The cleaned and processed data are integrated into a standard data model and output through a flexible interface.

[0065] As a preferred embodiment of this embodiment, in step S1, the preliminary data cleaning includes the following sub-steps:

[0066] Missing value handling, including detecting and handling missing values ​​in the data;

[0067] Format validation, including verifying that the format of key fields is correct;

[0068] Data integrity checks to ensure that all key fields have valid values;

[0069] Outlier detection, detects and corrects outliers through automated data anomaly detection models.

[0070] As a preferred embodiment of this embodiment, in step S2, the intelligent field recognition and fusion algorithm includes the following sub-steps:

[0071] Field mapping and matching: Use the BERT model to semantically standardize field names in different data sources and calculate the similarity between different fields.

[0072] Field matching model, using random forest model to match fields;

[0073] Unify the generation of product codes and generate unique identification codes for financial products based on multiple key fields.

[0074] As a preferred embodiment of this embodiment, in step S3, the data cleaning logic includes the following sub-steps:

[0075] Duplicate items are processed to filter out the most relevant financial product items;

[0076] Date screening logic, screening products based on investment start and end dates;

[0077] Automated data anomaly detection and correction, automatically identifying and correcting outliers in the data.

[0078] As a preferred method of this embodiment, in step S4, the data integration and output includes the following sub-steps:

[0079] Data integration, integrating cleaned, field-matched, anomaly-detected, and corrected data into a unified data model;

[0080] Data storage and retrieval, storing the integrated data in a relational database;

[0081] Output interface design, design data output interface, the output interface supports API and multiple data formats;

[0082] Data access and update mechanism to achieve real-time data access and batch data export from other systems or modules.

[0083] The following will take development activities as an example to introduce the specific implementation process of using the financial product data processing method described in the embodiment of the present application to solve the problem that data between different data sources are difficult to directly merge and compare due to the lack of unified data processing standards.

[0084] 1. Data source analysis and preprocessing module

[0085] 1.1 Data Source Identification

[0086] Before processing multi-source financial product data, each data source needs to be identified. This step ensures that the system can identify and classify data from different sources, laying the foundation for subsequent data cleaning and integration.

[0087] Step Description:

[0088] Identify the unique identifiers and structural characteristics of each data source.

[0089] Classify financial product data from different sources, such as domestic data sources and international data sources, structured data and unstructured data, etc.

[0090] Example:

[0091]

[0092] 1.2 Preliminary data cleaning

[0093] Preliminary data cleaning is the first step to ensure data quality, which mainly includes missing value processing, format verification, data integrity check and outlier detection.

[0094] 1.2.1 Missing value processing:

[0095] Detect and handle missing values ​​in data, such as removing invalid records or filling in default values.

[0096] Example:

[0097]

[0098] 1.2.2 Format verification:

[0099] Verify that key fields are in the correct format, such as the date field should be in the format of "YYYY-MM-DD".

[0100] Use case description: If malformed data is found, the system will mark it as invalid and remove it.

[0101] 1.2.3 Data integrity check:

[0102] Make sure all key fields have valid values ​​and follow the expected logic. For example, "invest_start_date" should not be later than "invest_end_date".

[0103] 1.2.4 Outlier Detection:

[0104] Detect and correct outliers using automated data anomaly detection models such as Isolation Forest.

[0105] Example:

[0106]

[0107]

[0108] 1.3 Data processing of cross-border financial products

[0109] Cross-border data processing is the key to the data preprocessing module, which aims to solve the problem of inconsistent data standards in different countries and regions.

[0110] 1.3.1 Unification of data standards:

[0111] The BERT model is used to perform semantic analysis on financial product data from different countries and standardize them into a unified format.

[0112] Use case description: For example, unify the currency units of different countries into US dollars and unify the date format into "YYYY-MM-DD".

[0113] 1.3.2 International coding rules:

[0114] A multi-hash verification mechanism is used to generate a globally unique product identification code to ensure the uniqueness and reliability of cross-border data.

[0115] Example:

[0116]

[0117] 2. Intelligent field recognition and fusion algorithm

[0118] Intelligent field recognition and fusion algorithm is the core step of multi-source financial product data fusion. Its main task is to identify similar fields in data from different sources and standardize and fuse them. This part will combine BERT natural language processing technology and machine learning models to improve the accuracy and efficiency of field matching.

[0119] 2.1 Field Mapping and Matching

[0120] 2.1.1 Field name standardization:

[0121] Step Description:

[0122] Use the BERT model to perform semantic standardization on field names in different data sources. BERT can understand the semantics of different fields through context and map them into unified standard names.

[0123] Example:

[0124]

[0125] 2.1.2 Similarity calculation:

[0126] Step Description:

[0127] The similarity between different fields is calculated through the field vectors generated by BERT. Fields with high similarity will be identified as the same standard field.

[0128] Commonly used similarity calculation methods include cosine similarity.

[0129] Example:

[0130] from sklearn.metrics.pairwise import cosine_similarity

[0131] # Example field vector

[0132] vec1=standardize_field("start_date")

[0133] vec2=standardize_field("begin_date")

[0134] # Calculate similarity

[0135] similarity=cosine_similarity(vec1.detach().numpy(),vec2.detach().numpy())

[0136] print(f"Similarity:{similarity[0][0]}")

[0137] 2.2 Field Matching Model

[0138] Step Description:

[0139] The random forest model is used to match the fields. The model takes the similarity features generated by BERT as input and outputs the probability of whether the field matches.

[0140] Example:

[0141] from sklearn.ensemble import RandomForestClassifier

[0142] #Sample data

[0143] X_train = [[0.95], [0.85], [0.10]] # Cosine similarity as feature

[0144] y_train = [1, 1, 0] # 1: match, 0: mismatch

[0145] # Random Forest Classifier

[0146] clf=RandomForestClassifier()

[0147] clf.fit(X_train,y_train)

[0148] #Predict field matching

[0149] test_similarity = [[0.90]]

[0150] prediction=clf.predict(test_similarity)

[0151] print(f"Field match:{prediction[0]}")

[0152] 2.3 Generation of unified product codes

[0153] Step Description:

[0154] After the field matching is completed, a unique identification code for the financial product is generated based on multiple key fields.

[0155] A multiple hash verification mechanism is introduced to ensure that the generated identification code has high-dimensional uniqueness and collision resistance.

[0156] Example:

[0157]

[0158] 2.4 Cross-border financial product field matching

[0159] Step Description:

[0160] For cross-border financial product data, the BERT model is used to process field differences in different countries and solve the problems of inconsistent field naming rules and data formats.

[0161] Utilize BERT's cross-language capabilities to handle field recognition and matching in multilingual environments.

[0162] Use case description:

[0163] For example, the financial product fields with Chinese and English names are standardized into unified English field names through the BERT model to ensure global data consistency.

[0164] 3. Data cleaning logic

[0165] The data cleaning logic module is responsible for further cleaning and optimizing the data processed in the data source analysis and preprocessing module. Through this step, the consistency, accuracy and completeness of the data are ensured. Especially when dealing with cross-border financial products, the automated data anomaly detection and correction mechanism is particularly important.

[0166] 3.1 Duplicate entry processing

[0167] Step Description:

[0168] For financial product entries with the same bank_product_code, use the Tiered Share Type field to filter out the most relevant entries.

[0169] For duplicate entries, financial products with a tiered share type of sub-shares are retained first, while products with a tiered share type of total shares are removed.

[0170] Example:

[0171]

[0172] 3.2 Date filtering logic

[0173] Step Description:

[0174] For products with the same bank_product_code, filter according to the following rules:

[0175] Compare the investment start dates (invest_start_date) and keep the product with the latest investment start date.

[0176] If the investment start date is the same, the investment end date (invest_end_date) is further compared, and the product with the latest investment end date is retained.

[0177] Example:

[0178]

[0179]

[0180] 3.3 Automated data anomaly detection and correction

[0181] Step Description:

[0182] Introduce automated data anomaly detection and correction mechanisms to automatically identify and correct outliers in the data through machine learning models (such as Isolation Forest or BERT-based anomaly detection models).

[0183] This mechanism is particularly suitable for handling anomalies in cross-border wealth management product data and ensuring data consistency and accuracy.

[0184] Example:

[0185]

[0186] 3.4 Cross-border data exception handling

[0187] Step Description:

[0188] When processing cross-border wealth management products, the automated data anomaly detection mechanism will be customized according to the market regulations and data characteristics of different countries.

[0189] By combining contextual understanding of international market data standards, outliers in cross-border data can be automatically corrected, such as yield anomalies caused by exchange rate changes.

[0190] Use case description:

[0191] For example, when processing data from different countries, the system will automatically detect and correct abnormal returns caused by exchange rate fluctuations, and generate reasonable correction values ​​by comparing market standards and historical data.

[0192] 4. Data integration and output

[0193] The data integration and output module is the last step of the entire system. It is responsible for integrating the cleaned and processed data into a standard data model and outputting it through a flexible interface so that other systems or modules can easily access and use the data. This module also needs to ensure consistency and accuracy in the integration of cross-border wealth management product data.

[0194] 4.1 Data Integration

[0195] Step Description:

[0196] Integrate the financial product data that has been cleaned, field matched, anomaly detected and corrected into a unified data model.

[0197] The data model needs to be highly scalable to support new data fields and product types added in the future, especially the processing of cross-border financial products.

[0198] Unified data model design:

[0199] Field extensibility: The data model adopts an object-oriented design method and supports data structure expansion of different data sources through inheritance and polymorphism.

[0200] Cross-border data processing: The model is designed with differences in international standards in mind, and includes fields for adaptability to different market regulations.

[0201] Use case description:

[0202] For example, for a new financial product data source, the data model can extend its unique fields by inheriting the base class without making major adjustments to the overall structure.

[0203] 4.2 Data Storage and Retrieval

[0204] Step Description:

[0205] Store the integrated data in a relational database (such as MySQL) for subsequent access, analysis, and processing. The database structure design needs to support efficient query and cross-border data processing.

[0206] Example database structure:

[0207]

[0208] Use case description:

[0209] When storing data, data from different countries is partitioned and stored according to the country_code field to optimize the efficiency of cross-border data queries.

[0210] 4.3 Output interface design

[0211] Step Description:

[0212] Design a flexible data output interface that supports API and multiple data formats (such as CSV, JSON, etc.) so that other systems or modules can easily access and use the integrated data.

[0213] API interface design:

[0214] Provides a RESTful API interface that supports multiple query methods, such as filtering queries by country, product code or yield.

[0215] Supports paging query, batch data export and data update operations.

[0216] API interface example:

[0217]

[0218]

[0219] 4.4 Data access and update mechanism

[0220] Step Description:

[0221] Real-time data access: Real-time data access to other systems or modules is achieved through API interfaces, supporting dynamic query and use to meet the real-time analysis needs of financial institutions.

[0222] Batch data export: Provides batch data export function, allowing users to download large amounts of data at one time for offline analysis.

[0223] Data update: Design a data update mechanism to ensure that the system can automatically clean, integrate and update data when the data source changes.

[0224] Use case description:

[0225] For example, when new cross-border wealth management product data is imported, the system will automatically trigger the data update process, including data cleaning, anomaly detection, field matching and data integration, and finally provide real-time query and batch export functions through the API.

[0226] Next, another embodiment is used to specifically describe how this embodiment solves the problem that data from different data sources are difficult to directly merge and compare due to the lack of a unified data processing standard.

[0227] Suppose there is financial product data from three different financial institutions (A, B, C). These data formats are not uniform and there are data quality issues, so they need to be cleaned and integrated through our system.

[0228] Data Entry

[0229] Data of Institution A (CSV format):

[0230] ProductID,Name,Issuer,Start,End,ExpectedReturn,Country101,"AlphaFund","Bank A","2023-01-10","2024-01-10",4.5,"US"102,"Beta Fund","Bank A","2023-02-15","2024-02-15",5.0,"US"Institution B's data (JSON format):

[0231]

[0232] Data of institution C (XML format):

[0233]

[0234]

[0235] Step 1: Data source analysis and preprocessing

[0236] 1.1 Data source identification:

[0237] The system recognizes the format of the data source (CSV, JSON, XML) and determines the data structure characteristics.

[0238] Data identification results:

[0239] Institution A:{'format':'CSV','identifier':'Bank A'}

[0240] Institution B:{'format':'JSON','identifier':'Bank B'}

[0241] Institution C:{'format':'XML','identifier':'Bank C'}

[0242] 1.2 Preliminary data cleaning:

[0243] Perform missing value processing, format verification, and anomaly detection on each data source.

[0244] Data cleaning process:

[0245] Institution A: The inspection found no missing values ​​and all dates were in the correct format.

[0246] Institution B: There are no missing values, but the Yield field needs to be converted to floating-point format.

[0247] Institution C: All fields are complete, with no abnormal data.

[0248] Cleaning results:

[0249] Datasets A, B, and C all went through preliminary data cleaning.

[0250] Step 2: Intelligent field identification and fusion

[0251] 2.1 Field mapping and matching:

[0252] Field Standardization:

[0253] The system uses NLP (such as BERT) to standardize the fields and recognizes ProductID, Prod_Code, ID, and Name, Prod_Name, and Title as similar fields.

[0254] Field Matching Model:

[0255] The system matches the fields through machine learning models such as random forests.

[0256] Result: ProductID,Prod_Code,ID→Product_Code; Name,Prod_Name,Title→Product_Name.

[0257] Generate a field mapping table:

[0258]

[0259] 2.2 Unified product code generation:

[0260] Key field selection:

[0261] Select Product_Name, Issuer, Start, Country as key fields.

[0262] Multiple hash verification mechanism:

[0263] Generate a unique identification code for each product, ensuring that the same product from different sources has the same bank_product_code.

[0264] Generates the following result:

[0265] Alpha Fund(A)→SHA-256("Alpha Fund|Bank A|2023-01-10|US")→bank_product_code:af9...

[0266] Alpha Fund(B)→SHA-256("Alpha Fund|Bank B|2023-01-10|US")→bank_product_code:bf9...

[0267] Step 3: Data cleaning logic

[0268] 3.1 Duplicate entry processing:

[0269] Product processing with the same bank_product_code:

[0270] Products with the same bank_product_code are processed repeatedly and filtered based on investment date.

[0271] Example: Keep products with the latest investment start date.

[0272] 3.2 Automated data anomaly detection and correction:

[0273] Outlier Detection:

[0274] Use the Isolation Forest model to detect and label outliers.

[0275] If an abnormal value is found in the ExpectedReturn field, correct the value.

[0276] Cleaning results:

[0277] Duplicates have been processed and there is no abnormal data.

[0278] Step 4: Data integration and output

[0279] 4.1 Data Integration:

[0280] Unified Data Model:

[0281] All cleaned data are integrated into a unified data model, and the data standards of cross-border wealth management products are also processed in it.

[0282] Example of unified model results:

[0283]

[0284] 4.2 Data Storage and Retrieval:

[0285] Database storage:

[0286] Store the integrated data in the MySQL database and design the corresponding table structure.

[0287] Once stored, the data can be retrieved and analyzed through SQL queries.

[0288] 4.3 Output interface design:

[0289] API interface:

[0290] The system provides a RESTful API interface that supports queries based on fields such as product name and country.

[0291] Example: Get all financial products from the US through API request.

[0292] 4.4 Data access and update:

[0293] Real-time access:

[0294] External systems can access the integrated data in real time through the API interface, supporting dynamic query and batch export.

[0295] The technical solutions provided in the embodiments of this specification improve the accuracy and availability of financial product data and ensure the consistency and integrity of data through innovative data processing technology; simplify the comparison and analysis process of financial products, help financial institutions improve decision-making efficiency and enhance market responsiveness; support a wider range of data sources and product types, enhance the applicability and flexibility of the system, and meet the needs of different financial institutions; reduce labor costs through automated processing, enhance the continuity and stability of data processing, and improve overall work efficiency.

[0296] based on Figure 1The financial product data processing method provided by the embodiment shown, Figure 2 A financial product data processing system provided by an embodiment of the present application is shown. Figure 2 As shown, the system can mainly include: data input module 201, data preprocessing module 202, intelligent field recognition and fusion module 203, data cleaning and anomaly detection module 204, data integration and unified data model generation module 205, data output and access interface 206, wherein,

[0297] Data input module 201, used to receive and store raw data from different data sources to provide input for subsequent processing;

[0298] A data preprocessing module 202, for performing the data source identification, preliminary data cleaning and data format standardization;

[0299] Intelligent field identification and fusion module 203, used to identify similar fields of data from different sources and standardize and fuse them;

[0300] Data cleaning logic module 204, used to clean and optimize the processed data;

[0301] The data integration and unified data model generation module 205 is used to integrate the cleaned and fused data into a unified data model and store it in the database; and at the same time generate a data set that meets the data output standard;

[0302] The data output and access interface 206 is used to provide an API interface and multiple data formats.

[0303] As a preferred embodiment of this embodiment, the data preprocessing module 202 includes:

[0304] Detect and handle missing values ​​in data;

[0305] Verify that the key fields are in the correct format;

[0306] Ensure that all key fields have valid values;

[0307] Detect and correct outliers with automated data anomaly detection models.

[0308] As a preferred embodiment of this embodiment, the intelligent field recognition and fusion module 203 includes:

[0309] Use the BERT model to perform semantic standardization on field names in different data sources, and generate a unified product code through multiple hash checks; perform field mapping and matching at the same time, generate a unified field mapping table, and calculate the similarity between different fields;

[0310] A random forest model is used to match the fields;

[0311] Generate a unique identification code for a financial product based on multiple key fields.

[0312] In the embodiments of the present application, innovative data processing technology is used to improve the accuracy and availability of financial product data, ensure the consistency and integrity of the data; simplify the comparison and analysis process of financial products, help financial institutions improve decision-making efficiency and enhance market responsiveness; support a wider range of data sources and product types, enhance the applicability and flexibility of the system, and meet the needs of different financial institutions; reduce labor costs through automated processing, enhance the continuity and stability of data processing, and improve overall work efficiency.

[0313] It can be understood that the above modules of the financial product data processing system in this embodiment have the function of realizing Figure 1 The functions of the corresponding steps of the method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of the above modules, please refer to Figure 1 The corresponding description of the method in the embodiment shown in will not be repeated here.

[0314] An embodiment of the present application provides an electronic device, including a processor and a memory;

[0315] A memory, used for storing operation instructions;

[0316] The processor is used to execute the financial product data processing method provided in any embodiment of the present application by calling an operation instruction.

[0317] As an example, Figure 3 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Figure 3 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0318] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0319] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0320] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0321] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0322] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the financial product data processing method provided in any embodiment of the present application.

[0323] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0324] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the financial product data processing method shown in the above method embodiment is implemented.

[0325] The computer-readable storage medium provided in the embodiments of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0326] The data processing scheme for financial products provided in the embodiment of the present application is to create a project R&D activity process on a development and operation integrated platform, fill in system information, production date and R&D activity purpose; based on the project R&D activity process, create a development branch and a design branch on the model design management platform; and make physical model changes on the design branch, submit for review after the change, and merge the design branch content into the development branch after the review is passed; generate a database change script, and review the generated script, and publish it to the code hosting platform after the review is passed. It realizes the integration of physical model design release into the project R&D activity process, connecting the two important stages of model design change and release, which has important guidance and practical significance for the forward design release of the model. This technical solution can be well integrated into the project R&D activity process, so that the physical model design change activity is managed by the project R&D activity, and the forward release behavior of the model designer can be guaranteed, thereby achieving the expected effect.

[0327] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific system can be divided into different functional modules to complete all or part of the functions described above.

[0328] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0329] In several embodiments covered by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0330] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0331] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0332] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for processing financial product data, characterized in that: The following steps are involved: Step S1: Data source analysis and preprocessing, including data source identification, preliminary data cleaning and cross-border financial product data processing; Step S2: Intelligent field identification and fusion algorithm, used to identify similar fields of data from different sources and standardize and fuse them; Step S3: data cleaning logic, further cleaning and optimizing the processed data; Step S4: Data integration and output: The cleaned and processed data are integrated into a standard data model and output through a flexible interface.

2. The financial product data processing method according to claim 1, characterized in that: In step S1, the preliminary data cleaning includes the following sub-steps: Missing value handling, including detecting and handling missing values ​​in the data; Format validation, including verifying that the format of key fields is correct; Data integrity checks to ensure that all key fields have valid values; Outlier detection, detects and corrects outliers through automated data anomaly detection models.

3. The financial product data processing method according to claim 1, characterized in that: In step 2, the intelligent field recognition and fusion algorithm includes the following sub-steps: Field mapping and matching: Use the BERT model to semantically standardize field names in different data sources and calculate the similarity between different fields. Field matching model, using random forest model to match fields; Unify the generation of product codes and generate unique identification codes for financial products based on multiple key fields.

4. The financial product data processing method according to claim 1, characterized in that: In step 3, the data cleaning logic includes the following sub-steps: Duplicate items are processed to filter out the most relevant financial product items; Date screening logic, screening products based on investment start and end dates; Automated data anomaly detection and correction, automatically identifying and correcting outliers in the data.

5. The financial product data processing method according to claim 1, characterized in that: In step 4, the data integration and output includes the following sub-steps: Data integration, integrating cleaned, field-matched, anomaly-detected, and corrected data into a unified data model; Data storage and retrieval, storing the integrated data in a relational database; Output interface design, design data output interface, the output interface supports API and multiple data formats; Data access and update mechanism to achieve real-time data access and batch data export from other systems or modules.

6. A financial product data processing system, characterized in that: The system includes: a data input module, a data preprocessing module, an intelligent field recognition and fusion module, a data cleaning and anomaly detection module, a data integration and unified data model generation module, and a data output and access interface, wherein: Data input module, used to receive and store raw data from different data sources to provide input for subsequent processing; A data preprocessing module, used to perform data source identification, preliminary data cleaning and data format standardization; Intelligent field identification and fusion module, used to identify similar fields in data from different sources and standardize and fuse them; Data cleaning logic module, used to clean and optimize processed data; A data integration and unified data model generation module is used to integrate the cleaned and fused data into a unified data model and store it in the database; at the same time, it generates a data set that meets the data output standard; Data output and access interface, used to provide API interface and multiple data formats.

7. The financial product data processing system according to claim 6, characterized in that: The data preprocessing module comprises: Detect and handle missing values ​​in data; Verify that the key fields are in the correct format; Make sure all key fields have valid values; Detect and correct outliers with automated data anomaly detection models.

8. The financial product data processing system according to claim 6, characterized in that: The intelligent field recognition and fusion module includes: Use the BERT model to perform semantic standardization on field names in different data sources, and generate a unified product code through multiple hash checks; perform field mapping and matching at the same time, generate a unified field mapping table, and calculate the similarity between different fields; A random forest model is used to match the fields; Generate a unique identification code for a financial product based on multiple key fields.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 5 by calling the operation instruction.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images