Data management method, device and equipment across databases and computer storage medium
By managing the data to be analyzed from multiple databases in a unified manner using a standard database, the problem of low efficiency in cross-database data management is solved, and efficient data quality assurance and management flexibility are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUOSEN SECURITIES
- Filing Date
- 2023-05-24
- Publication Date
- 2026-05-12
AI Technical Summary
Existing cross-database data management is inefficient, especially in heterogeneous databases where efficient data management is difficult to achieve. It also requires high technical skills from administrators and has low code reusability.
By receiving the analysis object information configured by the user, the system extracts data to be analyzed from multiple databases and converts it into target data in a standard database. Then, it performs data analysis in the standard database according to the analysis rules set by the user, thereby achieving unified management across databases.
It improves the efficiency of cross-database data management, ensures data quality, reduces the technical requirements for administrators, and enhances code reusability and management flexibility.
Smart Images
Figure CN116594990B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, specifically to a cross-database data management method, apparatus, device, and computer storage medium. Background Technology
[0002] As the concepts of data elements and data value realization become increasingly prevalent, high-quality data has become a fundamental element driving corporate strategy, decision-making, marketing, risk control, and research processes. Particularly in data-intensive industries such as finance, data quality is crucial to every system, report, and business decision. Therefore, data quality management for databases is paramount. Given the diverse distribution, massive scale, high sensitivity, and complex application scenarios of existing data, ensuring data quality typically requires each system to develop its own data quality verification tasks for its database.
[0003] The inventors discovered that when there is a need for data management across databases, especially heterogeneous databases, such as comparing data from multiple databases to determine consistency, the isolation between databases due to deployment locations or network incompatibility, coupled with differences in coding languages and management rules, makes it impossible to manage data from multiple databases simultaneously using a single data management code. This results in low efficiency for cross-database data management. Furthermore, managing multiple databases often requires database administrators to manually write data management code, demanding expertise in multiple database technologies, placing high demands on administrators, and resulting in low code reusability, thus contributing to the low efficiency of existing cross-database data management systems.
[0004] Therefore, a more efficient and practical cross-database data management method is needed. Summary of the Invention
[0005] In view of the above problems, embodiments of the present invention provide a cross-database data management method to solve the problem of low efficiency in cross-database data management in the prior art.
[0006] According to one aspect of the present invention, a cross-database data management method is provided, the method comprising:
[0007] Receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases;
[0008] Based on the configuration information of the analysis object, multiple target databases and information on the data to be analyzed in the multiple target databases are determined from multiple optional databases. Based on the information on the data to be analyzed, the data to be analyzed is extracted from the multiple target databases respectively.
[0009] The data to be analyzed is converted into target data in a standard database;
[0010] Based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, the target data analysis rule corresponding to the target data is determined.
[0011] The target data is analyzed in the standard database according to the target data analysis rules to obtain the data analysis results.
[0012] The data analysis results are then returned to the user.
[0013] In one optional approach, the analysis object configuration information includes database selection information, database connection parameters, and analysis object selection information; the method further includes:
[0014] The plurality of target databases are determined from the plurality of optional databases based on the database selection information;
[0015] Connect to the multiple target databases according to the database connection parameters corresponding to each target database, and display the optional data in the multiple target databases to the user;
[0016] Obtain the analysis object selection information from the user's optional data input for each of the target databases;
[0017] Based on the information selected for the analysis object, the data to be analyzed is searched in multiple target databases to obtain the information of the data to be analyzed.
[0018] In one alternative approach, the plurality of target databases are heterogeneous databases; the method further includes:
[0019] Perform correlation analysis on the data to be analyzed to obtain the data correlation relationships;
[0020] Based on the data association relationship, the data to be analyzed is fused to obtain the data to be stored corresponding to the target field in the standard database;
[0021] The data to be stored is stored according to the data storage type corresponding to the target field to obtain the target data.
[0022] In one optional approach, the data analysis rule configuration template includes multiple optional data analysis rules; the analysis rule configuration information includes rule selection information for the multiple optional data analysis rules by the user; the optional data analysis rules are predetermined based on historical data management requirements and the data processing specifications of the standard database; the method further includes:
[0023] The target data analysis rule is determined from the plurality of optional data analysis rules based on the rule selection information.
[0024] In one optional approach, the data analysis rule configuration template further includes multiple optional business analysis types; the analysis rule configuration information also includes user selection information for the multiple optional business analysis types.
[0025] The execution order of the data preprocessing rules and the target data analysis rules is determined based on the business selection information.
[0026] The target data is preprocessed according to the data preprocessing rules to obtain preprocessed data;
[0027] The target data analysis rules are executed on the preprocessed data according to the execution order of the rules to obtain the data analysis results.
[0028] In an optional embodiment, the analysis rule configuration information further includes business threshold setting information; the method further includes:
[0029] Based on the business threshold setting information, the data analysis results are used for alarm analysis.
[0030] In one optional approach, the analysis rule configuration information includes rule execution action triggering conditions; the method further includes:
[0031] Real-time detection of current scene information;
[0032] When it is determined that the current scene information meets the triggering condition of the rule execution action, the data to be analyzed is analyzed according to the target data analysis rule to obtain the data analysis result.
[0033] According to another aspect of the present invention, a cross-database data management apparatus is provided, comprising:
[0034] The first configuration module is used to receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases.
[0035] An extraction module is used to determine multiple target databases and information about the data to be analyzed in the multiple target databases from multiple optional databases according to the configuration information of the analysis object, and to extract the data to be analyzed from the multiple target databases respectively according to the information of the data to be analyzed;
[0036] The conversion module is used to convert the data to be analyzed into target data in a standard database;
[0037] The second configuration module is used to determine the target data analysis rule corresponding to the target data based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database.
[0038] The analysis module is used to perform data analysis on the target data in the standard database according to the target data analysis rules, and obtain data analysis results;
[0039] The return module is used to return the data analysis results to the user.
[0040] According to another aspect of the present invention, a cross-database data management device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;
[0041] The memory is used to store at least one executable instruction that causes the processor to perform operations as described in any of the embodiments of the cross-database data management method.
[0042] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes a cross-database data management device to perform operations as described in any of the embodiments of the cross-database data management method.
[0043] This invention embodiment receives analysis object configuration information generated by a user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases; based on the analysis object configuration information, multiple target databases and information on data to be analyzed in the multiple target databases are determined from the multiple optional databases; the data to be analyzed is extracted from the multiple target databases according to the information on the data to be analyzed; the data to be analyzed is converted into target data in a standard database; based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, a target data analysis rule corresponding to the target data is determined; the target data is analyzed in the standard database according to the target data analysis rule to obtain data analysis results; and the data analysis results are returned to the user. This invention differs from existing methods that cannot manage data across multiple databases simultaneously using a single data check statement. Instead, it requires manual coding and execution of separate code in each database, followed by manual aggregation and analysis of the results. This approach is inefficient and inaccurate. Instead, this invention centralizes the data to be analyzed from multiple databases into target data in a standard database. By performing data analysis on the target data in the standard database according to user-selected target data analysis rules, a single data check statement can simultaneously check data from multiple databases. This improves the efficiency of cross-database data management and ensures data quality in cross-database scenarios.
[0044] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0045] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0046] Figure 1 A flowchart illustrating a cross-database data management method provided in an embodiment of the present invention is shown.
[0047] Figure 2 This illustrates a data management task configuration interface in a cross-database data management method according to an embodiment of the present invention;
[0048] Figure 3 The configuration interface for the data acquisition statement to be analyzed in the cross-database data management method of this invention is shown.
[0049] Figure 4The diagram illustrates a configuration interface for data analysis rules in a cross-database data management method according to another embodiment of the present invention.
[0050] Figure 5 This illustrates a data analysis rule selection interface in a cross-database data management method according to another embodiment of the present invention;
[0051] Figure 6 The diagram illustrates a configuration interface for data analysis triggering conditions in a cross-database data management method according to another embodiment of the present invention.
[0052] Figure 7 This diagram illustrates a processing flowchart for a cross-database data consistency comparison scenario in a cross-database data management method provided in another embodiment of the present invention.
[0053] Figure 8 A schematic diagram of the structure of a cross-database data management device provided in an embodiment of the present invention is shown;
[0054] Figure 9 A schematic diagram of the structure of a cross-database data management device provided in an embodiment of the present invention is shown. Detailed Implementation
[0055] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0056] Figure 1 A flowchart illustrating a cross-database data management method provided by an embodiment of the present invention is shown. This method is executed by a computer processing device. The computer processing device may include a mobile phone, a laptop computer, etc. Figure 1 As shown, the method includes the following steps:
[0057] Step 10: Receive the analysis object configuration information generated by the user based on the database configuration template; the database configuration template includes information configuration items for multiple optional databases.
[0058] Users can be database administrators or maintenance personnel. Database configuration templates are used to configure information about the data objects to be analyzed, which require data management. These data objects are data stored in the database and can be structured data such as fields and tables, or optionally unstructured data such as text. Multiple optional databases can correspond to multiple business systems; considering the differences in the needs of these systems, the optional databases can be homogeneous or heterogeneous. Analysis object configuration information specifies the information of the data objects to be analyzed. Specifically, the analysis object configuration information can include configuration parameters for information configuration items for multiple optional databases. These information configuration items represent multiple storage-related dimensions of the object to be analyzed in the database, allowing the corresponding object to be analyzed to be found in the database based on these configuration items. Storage-related dimensions can include information about the database where it is stored, and information about the table or field where it is stored. Optionally, to improve data management efficiency, multiple optional databases can be displayed to users through a pre-defined database management interface for direct selection.
[0059] Step 20: Based on the analysis object configuration information, determine multiple target databases and information on the data to be analyzed in the multiple target databases from multiple optional databases, and extract the data to be analyzed from the multiple target databases respectively based on the information on the data to be analyzed.
[0060] Specifically, based on the database where each selected object to be analyzed is stored and the information of the fields or tables in the database, the database where the object to be analyzed is stored is determined as the target database. Then, the data to be analyzed is extracted from the target database according to the fields or tables where the object to be analyzed is stored.
[0061] Specifically, considering that data extraction requires first having data access permissions to the target database, and secondly, determining the identification information of the data to be analyzed within the target database, in another embodiment of the present invention, the analysis object configuration information includes database selection information, database connection parameters, and analysis object selection information. The database selection information includes the identifier and address information of the selected database; the database connection parameters include the necessary verification information for connecting to the database, such as the username and password; and the analysis object selection information includes selection information for data objects within the selected database, such as a specific table, field, or text within the selected database. It should be noted that the analysis object can be structured data, such as fields in a database, or unstructured data, such as news articles or reports.
[0062] Step 20 further includes: Step 201: Determine the plurality of target databases from the plurality of optional databases based on the database selection information.
[0063] Specifically, the identity information of each optional database is matched with the target identity information specified in the database selection information, and the matched optional database is determined as the target database.
[0064] It should be noted that, in optional embodiments, to enhance the freedom and personalization of database selection, the target database may not be selected from the optional databases, but rather configured freely by the user. For example, the user can input the data address corresponding to the target database, as well as login account and password, to facilitate the identification and connection to the target database.
[0065] Step 202: Connect to the multiple target databases according to the database connection parameters corresponding to the multiple target databases respectively, and display the optional data in the multiple target databases to the user.
[0066] In this process, each target database is logged in according to the database connection parameters. Optional data can be the data corresponding to all data objects stored in the target database.
[0067] Step 203: Obtain the analysis object selection information of the user for the optional data input of each of the target databases.
[0068] To further improve the efficiency of database management and user experience, the available data of each target database can be directly displayed to the user, so that the user can select the available data without having to log in to each target database and manually retrieve the data.
[0069] Step 204: Based on the information selected by the analysis object, search in the multiple target databases to obtain the information of the data to be analyzed.
[0070] Specifically, the target object specified in the analysis object selection information is matched with the optional objects corresponding to the optional data in the target database, and the information stored in the database of the matched optional objects is determined as the information of the data to be analyzed.
[0071] For example, the target object selected by the user could be the net asset value (NAV) of a fund. The NAV corresponds to specific tables and fields in different databases, such as the "Fund NAV" field stored in Table 1 of database A, and the "Fund Amount" field stored in Table C of database B. Users can configure the analysis object by selecting specific tables and fields. For instance, a user can select the task table (i.e., the task information table) of the account system and further search for the English names and fields in the displayed task table to select the object tables and fields that the user needs to manage and analyze. This allows for simultaneous monitoring of the NAV object across multiple different databases.
[0072] Step 30: Convert the data to be analyzed into target data in a standard database.
[0073] In this context, considering the diverse sources of the data to be analyzed, when multiple databases originate from heterogeneous databases, the isolation between these databases due to their different database languages and management standards prevents the simultaneous analysis of multiple datasets using the same database management task statements. Therefore, to achieve unified collection of heterogeneous data sources, the data to be analyzed from different databases can be collected into a common data quality verification platform, where a unified comparison is performed against a corresponding standard database. However, considering that the standard database may also be heterogeneous from the target database, the data to be analyzed needs to be transformed to obtain the target data within the standard database. The multiple target databases are heterogeneous databases, meaning they differ in type, such as database architecture or the languages used. For example, the multiple target databases could be Oracle, DM, MySQL, or other different types of databases.
[0074] In the process of data transformation, the data to be analyzed can first be subjected to correlation analysis to obtain the data objects that are commonly associated with the data to be analyzed. Then, based on the common data objects, the information under the common data objects is aggregated and stored under the target field.
[0075] Therefore, step 30 may optionally include:
[0076] Step 301: Perform correlation analysis on the data to be analyzed to obtain the data correlation relationship.
[0077] Association analysis is used to extract information from the data to be analyzed that corresponds to the same primary key field and analysis dimension. For example, the primary key field could be a securities company identifier. The data to be analyzed in target database A could include the securities company identifier and information under the corresponding first analysis dimension, which could include securities code, securities price, securities trading volume, etc. Similarly, the data to be analyzed in target database B could include the securities company identifier and information under the corresponding second analysis dimension, which could include the securities company's address, unified social credit code, etc. By identifying common fields in the data to be analyzed, the various fields in the data can be linked together to obtain the data association relationships.
[0078] It should be noted that for unstructured data such as text, there is no primary key field that can be directly extracted. Therefore, the analysis dimension can be something like the number of times keywords appear in the text. By counting the number of keyword occurrences, we can obtain the field value of the primary key field for each piece of data to be analyzed, such as the number of times keywords appear in news texts published by different news publishers.
[0079] Step 302: Perform data fusion on the data to be analyzed according to the data association relationship to obtain the data to be stored corresponding to the target field in the standard database.
[0080] Specifically, the relationships between fields corresponding to each piece of data to be analyzed are obtained through data association. The common primary key field and the fields associated with the primary key field in different databases are used as the target field. The field values of the data to be analyzed under the target field are used as the data to be stored. Continuing with the previous example, if the primary key field is "securities company identifier", the fields associated with the primary key field include "securities company address", "securities code", "securities price", "securities trading volume", and "unified social credit code".
[0081] Step 303: Store the data to be stored according to the data storage type corresponding to the target field to obtain the target data.
[0082] In particular, considering that even data under the same field may be stored in different databases in different ways, such as the fund net value field, which may be stored as an integer in target database A and as a floating-point number in target database B, in order to improve data availability, it is necessary to store the data to be stored uniformly according to the data storage type of the target field in the standard database.
[0083] Step 40: Determine the target data analysis rule corresponding to the target data based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database.
[0084] The data analysis rule configuration template includes multiple optional data analysis rules. These rules are pre-determined based on historical data management needs and the data processing specifications of standard databases. Specifically, optional data analysis rules can be components that are commonly used historical data management functions in standard databases. For example, they may include regular expression calculation, variable assignment, database statement execution, data item calculation, expression parameter parsing, field comparison, and table comparison. The analysis rule configuration information includes the user's rule selection information for the multiple optional data analysis rules. Users can select optional data analysis rules to obtain the target data analysis rule. For example, for fund market information, it is necessary to compare the securities issuance prices from the "Wind Information" database and the "Hengsheng Juyuan" database daily to see if they are consistent. Therefore, the target data analysis rule should at least include: variable assignment, used to assign the net asset value of the target fund from different data sources to a newly created variable, and data item calculation, used to compare the size of the aforementioned newly created variable. Optionally, considering that data management methods are constantly evolving with business development, needs, and changes in database management specifications and tools, the optional data analysis rules can be continuously expanded and updated.
[0085] Step 50: Perform data analysis on the target data in the standard database according to the target data analysis rules to obtain the data analysis results.
[0086] The target data is obtained from a standard database, processed according to the target data analysis rules, and the data analysis results are obtained. Specifically, the data analysis results may include consistency comparison results of data from different data sources, comparison results of data with preset thresholds, keyword statistics in the data, and other results.
[0087] Furthermore, considering that the purpose of data analysis is to enable managed data to better support business needs, such as making the data more accurate or extracting data features more accurately, while data in different business scenarios may not be directly applicable to the aforementioned pre-defined componentized target data analysis, and needs to be preprocessed according to the business rules corresponding to the business scenario.
[0088] Therefore, when analyzing target data, in addition to the preset target data analysis rules, the user's actual business analysis needs can also be considered. In another embodiment of the present invention, the data analysis rule configuration template further includes multiple optional business analysis types; the analysis rule configuration information also includes user selection information for the multiple optional business analysis types. The optional business analysis types are used to characterize data analysis types based on business needs, such as data comparison from multiple data sources, consistency comparison, data indicator statistics, and data fluctuation ratio checks. Presenting users with multiple optional business analysis types for direct selection improves data management efficiency and reduces the learning cost for users regarding database knowledge or business experience.
[0089] Step 50 also includes:
[0090] Step 501: Determine the rule execution order between the data preprocessing rules and the target data analysis rules based on the business selection information.
[0091] Among these features, the target business analysis type can be determined based on the business selection information, and the preset data preprocessing rules and rule execution order corresponding to the target business analysis type can be obtained.
[0092] Step 502: Preprocess the target data according to the data preprocessing rules to obtain preprocessed data.
[0093] For example, in the business type of fund size comparison involving data from multiple data sources, the corresponding data preprocessing rules can include calculating the fund size based on the fund's net asset value and the number of shares issued, so as to facilitate subsequent comparison of fund sizes.
[0094] Step 503: Execute the target data analysis rules on the preprocessed data according to the execution order of the rules to obtain the data analysis results.
[0095] The corresponding target data analysis rules include variable assignment and data item calculation. First, variable assignment is performed, assigning the fund size calculated in the preprocessing process to the target variable. Then, data item calculation is performed on the target variable to obtain the comparison results of fund sizes provided by different data sources.
[0096] Furthermore, considering that a common business need in data management is metric monitoring, which involves monitoring and issuing alerts on daily data from multiple dimensions such as business analysis, marketing evaluation, and risk warning, including statistics on the revenue, fluctuations, and marketing investment of various products, the thresholds for metric monitoring are generally set based on a business perspective. To allow users to freely set or adjust the monitoring thresholds for each database management task according to their business operations, in another embodiment of the present invention, the analysis rule configuration information also includes business threshold setting information.
[0097] Step 50 also includes:
[0098] Step 510: Perform alarm analysis on the data analysis results based on the business threshold setting information.
[0099] For example, a constant for a fund's net asset value (NAV) is generally a positive number greater than 1. Business threshold settings can include a threshold of 1 for the "fund NAV" field. An alert is triggered when a fund NAV is detected to be less than 1. For the volatility ratio, the threshold can be set to ensure that the value on day T is not more than 10% higher than the value on day T-1. For the number of accounts, the threshold can be set to ensure that the number of accounts per day is not less than 5000 or that the transaction amount is not negative. Based on these business threshold settings, alerts are triggered to check if the corresponding data in the data analysis results does not meet the user-defined thresholds. Optionally, alerts can be sent to users via email.
[0100] Step 60: Return the data analysis results to the user.
[0101] The data analysis results can be displayed through a pre-defined database management interface. This differs from existing cross-database management systems, which require users to write separate code for each target database to retrieve the data to be analyzed. Furthermore, since the data retrieved from different databases is not compatible with the same database language, manual analysis code must be written according to the specific data analysis specifications of each heterogeneous database. This invention provides users with customizable configurations for data analysis objects and rules, thereby improving data management efficiency. Moreover, to achieve efficient data management in cross-database scenarios, this invention extracts and stores the data to be analyzed from the target database into a standard database for data analysis based on the user's analysis object configuration information, further enhancing data analysis efficiency.
[0102] Optionally, considering that due to different data management needs and based on the changing patterns of data, the execution scenarios of data management tasks also differ. For example, in the financial securities field, data analysis of fund net asset value can be performed once after the end of each trading day, while the comparison of data consistency across multiple data sources can be performed after data release. Therefore, in another embodiment of the present invention, the analysis rule configuration information may further include rule execution action triggering conditions. Rule execution action triggering conditions are used to characterize the triggering conditions for the execution action of the target data analysis rule. When the triggering condition is triggered, the target data analysis rule is executed on the target data. Specifically, considering that the cross-database data management cycle is generally associated with the data production, consumption, and update cycle, and that the data production, consumption, and update cycle is generally determined by the business needs corresponding to the database, the triggering condition can be specific business scenario information. Therefore, step 60 further includes:
[0103] Step 601: Real-time detection of current scene information.
[0104] The current scenario information is used to represent the current time, database environment, and database update status. For example, the time could be 10:00 AM on Monday, the database environment could be standard database A, and the database update status could be that the last database update was within the last 30 minutes.
[0105] Step 602: When it is determined that the current scene information meets the triggering condition of the rule execution action, perform data analysis on the data to be analyzed according to the target data analysis rule to obtain the data analysis result.
[0106] The process involves matching the current scenario information with the triggering conditions for rule execution actions. When a match is found, the current scenario information is determined to meet the triggering conditions for the rule execution action. For example, the triggering conditions for the rule execution action could be 10:30 AM every Monday or within half an hour after data is written to standard database A. The detected current scenario information could be the current time being 10:30 AM on Monday, or the detection of data writing to standard database A. Optionally, for each data analysis task, the triggering method, cycle, time, and alarm personnel can be configured by the user through a preset interface. For example, checking whether the net asset value of the same fund in different databases is consistent every Monday at 8:00 AM.
[0107] In yet another embodiment of the invention, combined with Figures 2-6 This section explains the user interface for cross-database data management.
[0108] first, Figure 2This illustrates a data management task configuration interface in a cross-database data management method according to another embodiment of the present invention. For example... Figure 2 As shown, users can... Figure 2 Configure the fields in the interface shown to create a data management task. "Task Type" represents the type of data management task, such as a model verification task used to verify the reliability of a data model. "Task Name" represents the name of the data management task. "Database" represents the target database selected by the user, "Schema" represents the schema of the target database, "Table / View Name" represents the target table or view object selected by the user in the target database, and "Field Name" represents the field to be analyzed selected by the user in the target database. Figure 3 This illustrates the configuration interface for the data retrieval statement to be analyzed in a cross-database data management method according to another embodiment of the present invention. For example... Figure 3 As shown, users can configure the execution database, execution statements, and execution fields in this interface, thereby obtaining the data to be analyzed by executing the execution statements in the execution fields of the execution database. Figure 4 This illustrates a configuration interface for data analysis rules in a cross-database data management method according to another embodiment of the present invention. For example... Figure 4 As shown, users can edit rule codes, select corresponding optional data analysis rules, and customize the rule name and description, such as checking for anomalies in the account opening date in customer basic information. Specifically, for the optional data analysis rule selected by the user, the user can select the parameter information in the rule, such as filtering the account opening date under customer basic information and judging whether the account opening date is greater than a threshold. Figure 5 This illustrates a data analysis rule selection interface in a cross-database data management method according to another embodiment of the present invention. For example... Figure 5 As shown, users can drag and drop preset calculation components into the task execution interface to process the previously determined data to be analyzed according to the data analysis rules corresponding to the calculation components. Figure 6 This illustrates a configuration interface for data analysis triggering conditions in a cross-database data management method according to another embodiment of the present invention. For example... Figure 6 As shown, to further enhance the flexibility and personalization of cross-database data management, users can establish rule execution trigger conditions based on the trigger condition configuration information entered by the user in the trigger condition configuration template. The period refers to the execution cycle of the data management task, and the start field represents the start time of the data management task execution. For example, for the indicators of daily new customer information and daily operating reports, the check task needs to be monitored daily. Therefore, if the user checks "Daily" under the period field on the page and selects "Everyday" and "05:00:00," this task will be automatically executed every calendar day at 05:00.
[0109] In yet another embodiment of the invention, combined with Figure 7 This section describes the data management process in scenarios involving data consistency checks across multiple data sources. Figure 7 This diagram illustrates a processing flowchart for a cross-database data consistency comparison scenario in a cross-database data management method provided in another embodiment of the present invention. Figure 7 As shown, after the user configures the check object (i.e., the data to be analyzed), check constants (i.e., thresholds), check rules (i.e., target data processing rules), and execution plan (i.e., data management task triggering conditions) corresponding to the data management task, the system collects fund information table A from database A and fund information table B from database B. The data is then unified into a cross-database fund information table C, which is aggregated in the standard database C of the check platform. A new field is added to the cross-database fund information table C to record the data source, thereby achieving a comparison of data consistency across different data sources. Finally, if the check program detects inconsistencies in data from different sources, it triggers an email alert function to notify the relevant parties associated with that type of data. This embodiment of the invention effectively solves the problem of difficulty in comparing and monitoring data in heterogeneous database scenarios, effectively ensuring cross-system data quality.
[0110] The cross-database data management method provided in this embodiment of the invention receives analysis object configuration information generated by a user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases; based on the analysis object configuration information, multiple target databases and information on data to be analyzed in the multiple target databases are determined from the multiple optional databases; the data to be analyzed is extracted from the multiple target databases according to the information on the data to be analyzed; the data to be analyzed is converted into target data in a standard database; based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, a target data analysis rule corresponding to the target data is determined; the target data is analyzed in the standard database according to the target data analysis rule to obtain data analysis results; and the data analysis results are returned to the user. This invention differs from existing methods that cannot manage data across multiple databases simultaneously using a single data check statement. Instead, it requires manual coding and execution of separate code in each database, followed by manual aggregation and analysis of the results. This approach is inefficient and inaccurate. Instead, this invention centralizes the data to be analyzed from multiple databases into target data in a standard database. By performing data analysis on the target data in the standard database according to user-selected target data analysis rules, a single data check statement can simultaneously check data from multiple databases. This improves the efficiency of cross-database data management and ensures data quality in cross-database scenarios.
[0111] Figure 8 A schematic diagram of the structure of a cross-database data management device provided in an embodiment of the present invention is shown. Figure 8 As shown, the device 70 includes: a first configuration module 701, an extraction module 702, a conversion module 703, a second configuration module 704, an analysis module 705, and a return module 706. The first configuration module 701 is used to receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple selectable databases.
[0112] The extraction module 702 is used to determine multiple target databases and information about the data to be analyzed in the multiple target databases from multiple optional databases according to the analysis object configuration information, and to extract the data to be analyzed from the multiple target databases respectively according to the information of the data to be analyzed;
[0113] The conversion module 703 is used to convert the data to be analyzed into target data in a standard database;
[0114] The second configuration module 704 is used to determine the target data analysis rule corresponding to the target data based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database.
[0115] Analysis module 705 is used to perform data analysis on the target data in the standard database according to the target data analysis rules, and obtain data analysis results;
[0116] The return module 706 is used to return the data analysis results to the user.
[0117] The operation process of the cross-database data management device provided in this embodiment of the invention is largely the same as that of the aforementioned method embodiment, and will not be described again.
[0118] The cross-database data management device provided in this embodiment of the invention receives analysis object configuration information generated by a user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases; based on the analysis object configuration information, it determines multiple target databases and information on data to be analyzed in the multiple target databases; based on the information on data to be analyzed, it extracts the data to be analyzed from the multiple target databases respectively; it converts the data to be analyzed into target data in a standard database; based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, it determines the target data analysis rule corresponding to the target data; it performs data analysis on the target data in the standard database according to the target data analysis rule to obtain data analysis results; and it returns the data analysis results to the user. This invention differs from existing methods that cannot manage data across multiple databases simultaneously using a single data check statement. Instead, it requires manual coding and execution of separate code in each database, followed by manual aggregation and analysis of the results. This approach is inefficient and inaccurate. Instead, this invention centralizes the data to be analyzed from multiple databases into target data in a standard database. By performing data analysis on the target data in the standard database according to user-selected target data analysis rules, a single data check statement can simultaneously check data from multiple databases. This improves the efficiency of cross-database data management and ensures data quality in cross-database scenarios.
[0119] Figure 9 The diagram shows a structural schematic of a cross-database data management device provided in an embodiment of the present invention. The specific implementation of the cross-database data management device is not limited by the specific embodiments of the present invention.
[0120] like Figure 9 As shown, the cross-database data management device may include: a processor 802, a communications interface 804, a memory 806, and a communication bus 808.
[0121] The processor 802, communication interface 804, and memory 806 communicate with each other via communication bus 808. Communication interface 804 is used to communicate with other network elements such as clients or other servers. Processor 802 executes program 810, specifically performing the relevant steps described in the above embodiment of the cross-database data management method.
[0122] Specifically, program 810 may include program code, which includes computer-executable instructions.
[0123] Processor 802 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The cross-database data management device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0124] Memory 806 is used to store program 810. Memory 806 may include high-speed RAM or non-volatile memory, such as at least one disk drive.
[0125] Specifically, program 810 can be called by processor 802 to cause the cross-database data management device to perform the following operations:
[0126] Receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases;
[0127] Based on the configuration information of the analysis object, multiple target databases and information on the data to be analyzed in the multiple target databases are determined from multiple optional databases. Based on the information on the data to be analyzed, the data to be analyzed is extracted from the multiple target databases respectively.
[0128] The data to be analyzed is converted into target data in a standard database;
[0129] Based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, the target data analysis rule corresponding to the target data is determined.
[0130] The target data is analyzed in the standard database according to the target data analysis rules to obtain the data analysis results.
[0131] The data analysis results are then returned to the user.
[0132] The operation process of the cross-database data management device provided in this embodiment of the invention is largely the same as that of the aforementioned method embodiment, and will not be repeated here.
[0133] The cross-database data management device provided in this embodiment of the invention receives analysis object configuration information generated by a user based on a database configuration template. The database configuration template includes information configuration items for multiple optional databases. Based on the analysis object configuration information, the device determines multiple target databases and information about data to be analyzed within the multiple target databases. Based on the information about the data to be analyzed, it extracts the data to be analyzed from each of the multiple target databases. The data to be analyzed is then converted into target data in a standard database. Based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, the device determines the target data analysis rule corresponding to the target data. The target data is then analyzed in the standard database according to the target data analysis rule to obtain data analysis results. Finally, the data analysis results are returned to the user. This invention differs from existing methods that cannot manage data across multiple databases simultaneously using a single data check statement. Instead, it requires manual coding and execution of separate code in each database, followed by manual aggregation and analysis of the results. This approach is inefficient and inaccurate. Instead, this invention centralizes the data to be analyzed from multiple databases into target data in a standard database. By performing data analysis on the target data in the standard database according to user-selected target data analysis rules, a single data check statement can simultaneously check data from multiple databases. This improves the efficiency of cross-database data management and ensures data quality in cross-database scenarios.
[0134] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a cross-database data management device, causes the cross-database data management device to perform the cross-database data management method in any of the above method embodiments.
[0135] Specifically, the executable instructions can be used to cause data management devices across databases to perform the following operations:
[0136] Receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases;
[0137] Based on the configuration information of the analysis object, multiple target databases and information on the data to be analyzed in the multiple target databases are determined from multiple optional databases. Based on the information on the data to be analyzed, the data to be analyzed is extracted from the multiple target databases respectively.
[0138] The data to be analyzed is converted into target data in a standard database;
[0139] Based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, the target data analysis rule corresponding to the target data is determined.
[0140] The target data is analyzed in the standard database according to the target data analysis rules to obtain the data analysis results.
[0141] The data analysis results are then returned to the user.
[0142] The operation process of the computer-readable storage medium provided in this embodiment of the invention is largely the same as that of the aforementioned method embodiments, and will not be described again.
[0143] The computer-readable storage medium provided in this embodiment of the invention receives analysis object configuration information generated by a user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases; based on the analysis object configuration information, multiple target databases and information on data to be analyzed in the multiple target databases are determined from the multiple optional databases; the data to be analyzed is extracted from the multiple target databases according to the information on the data to be analyzed; the data to be analyzed is converted into target data in a standard database; based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database, a target data analysis rule corresponding to the target data is determined; the target data is analyzed in the standard database according to the target data analysis rule to obtain data analysis results; and the data analysis results are returned to the user. This invention differs from existing methods that cannot manage data across multiple databases simultaneously using a single data check statement. Instead, it requires manual coding and execution of separate code in each database, followed by manual aggregation and analysis of the results. This approach is inefficient and inaccurate. Instead, this invention centralizes the data to be analyzed from multiple databases into target data in a standard database. By performing data analysis on the target data in the standard database according to user-selected target data analysis rules, a single data check statement can simultaneously check data from multiple databases. This improves the efficiency of cross-database data management and ensures data quality in cross-database scenarios.
[0144] This invention provides a cross-database data management device for executing the aforementioned cross-database data management method.
[0145] This invention provides a computer program that can be invoked by a processor to cause a cross-database data management device to execute the cross-database data management method in any of the above method embodiments.
[0146] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the cross-database data management method in any of the above method embodiments.
[0147] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of the invention.
[0148] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0149] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.
[0150] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0151] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.
Claims
1. A cross-database data management method, characterized in that, The method includes: Receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases; Based on the configuration information of the analysis object, multiple target databases and information on the data to be analyzed in the multiple target databases are determined from multiple optional databases. Based on the information on the data to be analyzed, the data to be analyzed is extracted from the multiple target databases respectively. The data to be analyzed is converted into target data in a standard database; Based on the analysis rule configuration information generated by the user using the data analysis rule configuration template corresponding to the standard database, the target data analysis rule corresponding to the target data is determined. The data analysis rule configuration template includes multiple optional data analysis rules. The analysis rule configuration information includes the user's rule selection information for the multiple optional data analysis rules. The optional data analysis rules are pre-determined based on historical data management requirements and the data processing specifications of the standard database. The data analysis rule configuration template also includes multiple optional business analysis types. The analysis rule configuration information also includes the user's business selection information for the multiple optional business analysis types. The analysis rule configuration information also includes business threshold setting information and rule execution action triggering conditions. The target data is analyzed in the standard database according to the target data analysis rules to obtain the data analysis results. The data analysis results are then returned to the user.
2. The method according to claim 1, characterized in that, The analysis object configuration information includes database selection information, database connection parameters, and analysis object selection information; the step of determining multiple target databases from multiple selectable databases and information on the data to be analyzed in the multiple target databases based on the analysis object configuration information, and extracting the data to be analyzed from the multiple target databases respectively based on the information on the data to be analyzed, includes: The plurality of target databases are determined from the plurality of optional databases based on the database selection information; Connect to the multiple target databases according to the database connection parameters corresponding to each target database, and display the optional data in the multiple target databases to the user; Obtain the analysis object selection information from the user's optional data input for each of the target databases; Based on the information selected for the analysis object, the data to be analyzed is searched in multiple target databases to obtain the information of the data to be analyzed.
3. The method according to claim 1, characterized in that, The multiple target databases are heterogeneous databases; the process of converting the data to be analyzed into target data in a standard database includes: Perform correlation analysis on the data to be analyzed to obtain the data correlation relationships; Based on the data association relationship, the data to be analyzed is fused to obtain the data to be stored corresponding to the target field in the standard database; The data to be stored is stored according to the data storage type corresponding to the target field to obtain the target data.
4. The method according to claim 1, characterized in that, The step of determining the target data analysis rule corresponding to the target data based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database includes: The target data analysis rule is determined from the plurality of optional data analysis rules based on the rule selection information.
5. The method according to claim 4, characterized in that, The step of performing data analysis on the target data in the standard database according to the target data analysis rules to obtain data analysis results includes: The execution order of the data preprocessing rules and the target data analysis rules is determined based on the business selection information. The target data is preprocessed according to the data preprocessing rules to obtain preprocessed data; The target data analysis rules are executed on the preprocessed data according to the execution order of the rules to obtain the data analysis results.
6. The method according to claim 5, characterized in that, The step of performing data analysis on the target data in the standard database according to the target data analysis rules to obtain data analysis results includes: Based on the business threshold setting information, the data analysis results are used for alarm analysis.
7. The method according to claim 1, characterized in that, The step of performing data analysis on the target data in the standard database according to the target data analysis rules to obtain data analysis results includes: Real-time detection of current scene information; When it is determined that the current scene information meets the triggering condition of the rule execution action, the data to be analyzed is analyzed according to the target data analysis rule to obtain the data analysis result.
8. A cross-database data management device, characterized in that, The device includes: The first configuration module is used to receive analysis object configuration information generated by the user based on a database configuration template; the database configuration template includes information configuration items for multiple optional databases. The extraction module is used to determine multiple target databases and information about the data to be analyzed in the multiple target databases from multiple optional databases according to the configuration information of the analysis object, and to extract the data to be analyzed from the multiple target databases respectively according to the information of the data to be analyzed; The conversion module is used to convert the data to be analyzed into target data in a standard database; The second configuration module is used to determine the target data analysis rule corresponding to the target data based on the analysis rule configuration information generated by the user based on the data analysis rule configuration template corresponding to the standard database. The data analysis rule configuration template includes multiple optional data analysis rules. The analysis rule configuration information includes the user's rule selection information for the multiple optional data analysis rules. The optional data analysis rules are predetermined based on historical data management requirements and the data processing specifications of the standard database. The data analysis rule configuration template also includes multiple optional business analysis types. The analysis rule configuration information also includes the user's business selection information for the multiple optional business analysis types. The analysis rule configuration information also includes business threshold setting information and rule execution action triggering conditions. The analysis module is used to perform data analysis on the target data in the standard database according to the target data analysis rules, and obtain data analysis results; The return module is used to return the data analysis results to the user.
9. A cross-database data management device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the cross-database data management method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a cross-database data management device, causes the cross-database data management device to perform the operation of the cross-database data management method as described in any one of claims 1-7.