A processing method and system for data manipulation

Abstract

The application discloses a data operation processing method and system, comprising: obtaining database configuration information of a user and creating a database configuration scheme; obtaining data access configuration information of the user and creating a data access configuration scheme; obtaining a data operation request of the user, and splitting data information in the data operation request; performing security detection on the split data based on the created data access configuration scheme and the database configuration scheme; judging a data operation type that passes the detection, reassembling the data information, and generating a data access unit; when the operation type of the data that passes the security detection is a first operation mode, performing parallel calculation on the generated data access unit through a first processing mode to obtain result data; when the operation type of the data that passes the security detection is a second operation mode, performing parallel calculation on the generated data access unit through a second processing mode to obtain result data; and the problem of poor data access performance is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, and particularly relates to a data operation processing method and system. BACKGROUND

[0002] Currently, the development of database access functions and operation functions of software systems or platforms and Internet systems or platforms needs to make corresponding configurations, introduce drivers and write scripts for different manufacturers' database operations, especially the writing methods of relational databases and non-relational databases are completely different. For relational databases, if the data volume is large, the query speed will decrease linearly, or professional database designers and script writers are needed to optimize, and after the professional personnel optimize the database, there will be no follow-up work, which is undoubtedly a waste of cost for enterprises; for non-relational databases, the requirements for developers are higher, not only need to understand the principle of non-relational databases, but also need to have comprehensive knowledge of server performance balancing, disk distribution design, network structure design and other system distributed collaborative work, and after the platform is built, the non-relational database developers will have no place to use, which is also a waste of cost for enterprises, and it is impossible to use unified standards for relational databases and non-relational databases.

[0003] The development of database access functions and operation functions needs the demand personnel to convey the demand data and business model to the database designers and developers, and the developers need to communicate with the demand personnel many times to meet the design of demand data and business model and database model during the development of corresponding functions, and the design and development of complex business models also require the logical thinking of the developers to be relatively clear, and the design and development of large databases require personnel with rich work experience to be referred to; for business adjustment, it is difficult to find the influence level of database adjustment in time, and in serious cases, it may affect the normal use of important business functions.

[0004] Different databases have different application fields, and if a relational database or a relational database and a non-relational database coexist is generally used in the initial development of a system or platform, when the database fails to meet the requirements during the business development process, it is impossible to transform or convert the database, or the database conversion process is of high cost, or the period is very long, or there are many problems, which brings a lot of unnecessary stability and usability troubles to enterprises.

[0005] At present, there are many database operation components in the industry, but they are based on the unique configuration of a manufacturer and cannot realize the function of accessing the same data in different database manufacturers at the same time, or need to manually split the data in the program and perform data access to the operation engine of the corresponding database manufacturer; When the database manufacturer is replaced, the data conversion needs to be assisted by third-party tools, and the automatic conversion cannot be realized in the system, and the use of third-party tools also needs certain professional personnel and time cost, and cannot guarantee the correctness of data conversion, and repeated data verification is needed to complete the conversion smoothly; When the database operation performance problem occurs, the database developer needs to manually analyze the performance problem in a specific environment by using the database script execution plan function, and since there is no automatic performance prompt function, even if the performance problem point is analyzed, there is no performance optimization scheme. SUMMARY

[0006] In view of the problems in the prior art, the purpose of the present application is to provide a data operation processing method and system; The problems of poor database access function of software system or platform and internet system or platform, and insufficient automation and performance analysis of operation function development are solved.

[0007] The purpose of the present application is achieved by using the following technical solutions:

[0008] In a first aspect, the present application provides a data operation processing method, comprising the following steps:

[0009] Obtain the database configuration information of the user and create a database configuration scheme;

[0010] Obtain the data access configuration information of the user and create a data access configuration scheme;

[0011] Obtain the data operation request of the user, and split the data information in the data operation request;

[0012] Based on the created data access configuration scheme and database configuration scheme, the split data is detected for security;

[0013] Determine the operation type of the data passing the security detection, reassemble the data information and generate a data access unit;

[0014] When the operation type of the data passing the security detection is a first operation mode, the generated data access unit is calculated in parallel by a first processing mode to obtain result data; When the operation type of the data passing the security detection is a second operation mode, the generated data access unit is calculated in parallel by a second processing mode to obtain result data.

[0015] Preferably, the database configuration information includes a database link list, a database instance strategy list and a database configuration list;

[0016] The database configuration list of the user is acquired and a database configuration scheme is created, including the steps of:

[0017] The database link list of the user is acquired and database link information is created;

[0018] The database instance strategy list of the user is acquired and a database instance generation strategy is created;

[0019] The database configuration list of the user is acquired, and a database configuration scheme is created according to the created database link information and database instance generation strategy.

[0020] Further, the database link list includes database name, database type, database address, database instance information, login account and password, database configuration parameter, use period, whether valid and table space and the like.

[0021] Further, the database instance strategy list includes strategy name, strategy generation type, area dimension, digital dimension, other dimensions with rules and the effective time of database splitting.

[0022] Further, the database configuration list includes database configuration scheme name, database manufacturer type, database instance default prefix name, database link name and database instance generation strategy information.

[0023] Preferably, the data access configuration information includes a data partition strategy list and a data access configuration scheme list;

[0024] The data access configuration information of the user is acquired and a data access configuration scheme is created, including:

[0025] The data partition strategy list of the user is acquired and a data partition strategy is created;

[0026] The data access configuration scheme list of the user is acquired, and a data access configuration scheme is created according to the created data partition strategy and database configuration scheme.

[0027] Further, the data access configuration information further includes a data business logic relationship configuration scheme list;

[0028] The data access configuration information of the user is acquired and a data access configuration scheme is created, further including the steps of acquiring the data business logic relationship configuration scheme list of the user, and creating a data business logic relationship configuration scheme according to the created data access configuration scheme.

[0029] Preferably, the split data includes data table metadata, data table field metadata and database link information;

[0030] The split data is respectively subjected to security detection, including the steps of:

[0031] The split data table metadata and data table field metadata are matched with corresponding data in the created data access configuration scheme, and if they are matched, the security detection is passed, and if they are not matched, the security detection is not passed;

[0032] The split database link information is matched with corresponding database link information in the created database configuration scheme, and if they are matched, the security detection is passed, and if they are not matched, the security detection is not passed.

[0033] Preferably, the reassembling data information and generating a data access unit include the steps of:

[0034] The data access configuration scheme in which the data information in the data operation request is located is analyzed, and the data partition information corresponding to the data information in the data operation request is obtained;

[0035] The data partition information corresponding to the data information in the data operation request is assembled;

[0036] The database configuration scheme in which the data information in the data operation request is located is identified, and the database link information corresponding to the data information in the data operation request is obtained;

[0037] The database link information corresponding to the data information in the data operation request is assembled;

[0038] Based on the data partition information and the database link information corresponding to the data information in the data operation request, a data access unit is generated.

[0039] Preferably, when the operation type of the data passing the security detection is a first operation mode, the generated data access unit is subjected to parallel calculation by a first processing mode and result data is obtained, including the steps of:

[0040] The generated data access unit is subjected to data partition processing, and a first data group is obtained;

[0041] It is judged whether the first data group is multi-link or partitioned;

[0042] If the first data group is multi-link or partitioned, it is judged whether the first data group supports transaction, and if the first data group is not multi-link or not partitioned, parallel calculation and data distributed update are directly performed;

[0043] If the first data group supports transaction, an automated partition distributed transaction is started, and parallel calculation and distributed data update are performed, and if not, a subunit task of database and table partition is directly created, and parallel calculation and distributed data update are performed.

[0044] Further, the first operation mode is an update mode, and the update mode includes an increase, modification or deletion operation.

[0045] Preferably, when the operation type of the data passing through the security detection is the second operation mode, the generated data access unit is calculated in parallel through the second processing mode to obtain result data, including the steps of:

[0046] Obtaining data information in the generated plurality of data access units;

[0047] Parallel processing the data information in the plurality of data access units;

[0048] Summarizing the data processing result after the parallel processing of the data information in the plurality of data access units;

[0049] Cleaning the data processing result to obtain result data.

[0050] A data operation processing system, the processing system comprising:

[0051] A first configuration module for creating a database configuration scheme;

[0052] A second configuration module for creating a data access configuration scheme;

[0053] A data splitting module for obtaining a data operation request of a user and splitting data information in the data operation request;

[0054] A security detection module for performing security detection on the split data based on the created data access configuration scheme and database configuration scheme;

[0055] An operation type judgment module for judging the operation type of the data passing through the security detection;

[0056] A data access information generator for reassembling data information and generating a data access unit:

[0057] A heterogeneous data first executor for calculating the generated data access unit in parallel through a first processing mode to obtain result data when the operation type of the data passing through the security detection is a first operation mode;

[0058] The second executor of the heterogeneous data performs parallel calculation on the generated data access units by a second processing mode and obtains result data when the operation type of the data passing through the security detection is the second operation mode.

[0059] Preferably, the first executor of the heterogeneous data comprises:

[0060] A data partition processing unit is configured to perform data partition processing on the generated data access units and obtain a first data group.

[0061] A partition distributed transaction unit is configured to start an automatic partition distributed transaction.

[0062] A database and table partition subunit is configured to create a database and table partition subunit task.

[0063] A data distributed update unit is configured to perform data distributed update.

[0064] Preferably, the second executor of the heterogeneous data comprises:

[0065] A data acquisition unit is configured to acquire data information in the generated data access units.

[0066] A data pipeline processing unit is configured to perform parallel processing on the data information in the data access units.

[0067] A data pipeline collection unit is configured to collect data processing results after the parallel processing on the data information in the data access units.

[0068] A data cleaning unit is configured to clean the data processing results and obtain result data.

[0069] The present application has the following beneficial effects:

[0070] The application provides a data operation processing method and system, which is a software technology of a method or system for automatically selecting one or more databases of different manufacturers based on a cloud platform or system related to multi-task software development, simultaneous access and intelligent performance optimization prompting, so that system development only needs to focus on data storage logic and does not need to care about the selection of data storage and the increase of cost, timeliness and compatibility caused by the replacement of database manufacturers due to policy or strategic problems in the use process. When the database manufacturer needs to be replaced, the application has data automatic migration capability, and system developers do not need to worry about data loss or inconsistency caused by data replacement, which are automatically processed by the application. The application also provides an intelligent performance optimization prompting function, which can solve the problem of poor data access performance caused by unreasonable data storage file planning or unreasonable database operation. The application will automatically prompt system developers to make suggestions for optimization and improvement when performance problems are found. At the same time, when the automatic optimization option is turned on, the application also has an artificial intelligence automatic performance optimization processing function. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows.

[0072] Figure 1 is a data operation processing method flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0074] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be the general meanings understood by those skilled in the art to which the present application belongs.

[0075] As shown in Figure 1 , the present application provides a data operation processing method, which includes the following steps:

[0076] S1 obtains database configuration information of a user and creates a database configuration scheme;

[0077] S2 obtains data access configuration information of the user and creates a data access configuration scheme;

[0078] S3 obtains a data operation request of a user, and splits data information in the data operation request;

[0079] S4 performs security detection on the split data based on the created data access configuration scheme and database configuration scheme;

[0080] S5 judges an operation type of the data passing the security detection, reassembles the data information, and generates a data access unit;

[0081] S6 when the operation type of the data passing the security detection is a first operation mode, performs parallel calculation on the generated data access unit by a first processing mode and obtains result data; when the operation type of the data passing the security detection is a second operation mode, performs parallel calculation on the generated data access unit by a second processing mode and obtains result data.

[0082] Through the above steps, database configuration and data operation automation can be realized, so that the developer only needs to focus on data storage logic and does not need to care about the selection and system of data storage, thereby accelerating the overall development speed. Also, the situation that the professional personnel has nothing to do after the database configuration is completed is avoided.

[0083] In an embodiment, the database configuration information includes a database link list, a database instance strategy list and a database configuration list;

[0084] Obtaining a database configuration list of a user and creating a database configuration scheme, including the steps of:

[0085] Obtaining a database link list of a user and creating database link information;

[0086] Obtaining a database instance strategy list of a user and creating a database instance generation strategy;

[0087] Obtaining a database configuration list of a user, and creating a database configuration scheme according to the created database link information and database instance generation strategy.

[0088] In the above steps, the database link list includes database name, database type, database address, database instance information, login account and password, database configuration parameters, usage period, validity, and table space information. The database instance policy list includes policy name, policy generation type (such as date dimension (by year, month, day, etc.), region dimension (country, region, administrative code, etc.), numerical dimension (long integer segmentation N range, range region, etc.), and regular other dimensions), and the effective time of the database (such as starting from 2020 to perform data database, etc.). The database configuration list includes database configuration scheme name, database manufacturer type, database instance default prefix name, database link name, database instance generation policy, and other information.

[0089] Among them, the database type includes relational databases (such as mysql, mssql, oracle, Damon, Huawei GaussDB, etc.) and non-relational databases (such as Redis, h2, sqlite, bigtable, HBase, MongoDB, hdfs, Hadoop, Spark, etc.).

[0090] The database scheme configuration information is the basic channel information of data access. Multiple database link information can be configured at the same time, which provides data access channels for heterogeneous and heterogeneous data sources; when multiple data links are configured for each database, the necessary distributed database link information is provided for distributed data access capability, which is the basis for supporting simultaneous access of heterogeneous and distributed data.

[0091] Through the above steps, the database configuration information required for data operation is collected, so that the subsequent data operation can directly use the database configuration information.

[0092] In an embodiment, the data access configuration information includes a data partitioning strategy list and a data access configuration scheme list.

[0093] Obtaining the data access configuration information of the user and creating a data access configuration scheme includes the steps of:

[0094] Obtaining the data partitioning strategy list of the user and creating a data partitioning strategy.

[0095] Obtaining the data access configuration scheme list of the user, and creating a data access configuration scheme according to the created data partitioning strategy and database configuration scheme.

[0096] Among them, the data access configuration information also includes a data business logic relationship configuration scheme list.

[0097] The step of obtaining the data access configuration information of the user and creating a data access configuration scheme further comprises the steps of: obtaining a list of data service logic relationship configuration schemes of the user, and creating a data service logic relationship configuration scheme according to the created data access configuration scheme.

[0098] In an embodiment, the split data comprises data table metadata, data table field metadata and database link information.

[0099] The split data is subjected to security detection respectively, comprising the steps of:

[0100] The split data table metadata and data table field metadata are matched with corresponding data in the created data access configuration scheme, and if they are matched, the security detection is passed, and if they are not matched, the security detection is failed.

[0101] The split database link information is matched with corresponding database link information in the created database configuration scheme, and if they are matched, the security detection is passed, and if they are not matched, the security detection is failed.

[0102] The data table metadata comprises English name, Chinese name and alias of the data table, and the data table field metadata comprises English name, Chinese name, field type of the data table, and retrieval condition English name and Chinese name.

[0103] Through the above steps, it can be detected whether the data operation request of the user is pre-configured in the created data access configuration scheme, thereby providing security guarantee for subsequent data operation.

[0104] The data storage scheme configuration information is basic data partition information of data access, and is the basis for improving storage capacity of single data information and improving data information access performance, and provides analysis support for performance intelligentization prompt. In combination with the database scheme configuration information, data disaster recovery and data automatic migration functions can be realized. Meanwhile, table internal partition and data library are set, and then massive data can be accessed in quasi real time.

[0105] In an embodiment, the step of reassembling the data information and generating a data access unit comprises the steps of:

[0106] The data access configuration scheme in which the data information in the data operation request is located is analyzed, and data partition information corresponding to the data information in the data operation request is obtained;

[0107] The data partition information corresponding to the data information in the data operation request is assembled.

[0108] Identify the database configuration scheme where the data information in the data operation request is located, and obtain the database link information corresponding to the data information in the data operation request;

[0109] Assemble the database link information corresponding to the data information in the data operation request;

[0110] Based on the data partition information and database link information corresponding to the data information in the data operation request, generate a data access unit.

[0111] In an embodiment, when the operation type of the data passing through the security detection is a first operation mode, the generated data access unit is calculated in parallel through a first processing mode and the result data is obtained, including the steps of:

[0112] Perform data partition processing on the generated data access unit to obtain a first data group;

[0113] Determine whether the first data group is multi-link or partitioned;

[0114] If the first data group is multi-link or partitioned, determine whether the first data group supports transactions, and if the first data group is not multi-link or not partitioned, directly perform parallel calculation and data distributed update;

[0115] If the first data group supports transactions, start an automated partition distributed transaction, and perform parallel calculation and data distributed update, and if not, create a subunit task of database table partitioning, and perform parallel calculation and data distributed update.

[0116] The first operation mode is an update mode, and the update mode includes increase, modify or delete operation.

[0117] Further, when the operation type of the data passing through the security detection is a second operation mode, the generated data access unit is calculated in parallel through a second processing mode and the result data is obtained, including the steps of:

[0118] Obtain the data information in the generated multiple data access units;

[0119] Parallel processing of data information in multiple data access units;

[0120] Summarize the data processing results after parallel processing of data information in multiple data access units;

[0121] Clean the data processing results and obtain the result data.

[0122] The second operation mode is a query mode.

[0123] Through the above steps, data operation based on the operation request of the user is realized, so that the data operation is automatically performed, and the efficiency of the data operation is improved.

[0124] Further, the processing method of the data operation further includes the steps of:

[0125] returning the obtained result data to the user, writing and storing the operation log;

[0126] based on the written operation log, performing intelligent optimization analysis.

[0127] By writing the operation log, errors occurring in the data operation process can be found in a timely manner, and the errors can be modified in a timely manner. Based on the operation log, the optimization items in the database storage process can be found in a timely manner, and intelligent optimization analysis can be performed.

[0128] The embodiment also provides a processing system of data operation, and the processing system includes:

[0129] a first configuration module configured to create a database configuration scheme;

[0130] a second configuration module configured to create a data access configuration scheme;

[0131] a data splitting module configured to obtain a data operation request of a user and split data information in the data operation request;

[0132] a security detection module configured to perform security detection on the split data based on the created data access configuration scheme and the database configuration scheme;

[0133] an operation type judgment module configured to judge an operation type of the data passing the security detection;

[0134] a data access information generator configured to reassemble the data information and generate a data access unit:

[0135] a heterogeneous and heterogeneous data first executor configured to perform parallel calculation on the generated data access unit by a first processing mode and obtain result data when the operation type of the data passing the security detection is a first operation mode;

[0136] a heterogeneous and heterogeneous data second executor configured to perform parallel calculation on the generated data access unit by a second processing mode and obtain result data when the operation type of the data passing the security detection is a second operation mode.

[0137] Further, the processing system further includes an intelligent optimization analysis module configured to perform intelligent optimization analysis on the system.

[0138] In an embodiment, the heterogeneous and heterogeneous data first executor includes:

[0139] a data partition processing unit configured to perform data partition processing on the generated data access unit and obtain a first data group;

[0140] a partition distributed transaction unit configured to start an automated partition distributed transaction;

[0141] a database and table partitioning subunit configured to create a database and table partitioning subunit task;

[0142] a data distributed updating unit configured to perform data distributed updating.

[0143] Further, the heterogeneous and heterogeneous data second executor comprises:

[0144] a data acquisition unit configured to acquire data information in the generated data access unit;

[0145] a data pipeline processing unit configured to perform parallel processing on the data information in the data access unit;

[0146] a data pipeline collection unit configured to collect data processing results after parallel processing on the data information in the data access unit;

[0147] a data cleaning unit configured to clean the data processing results and obtain result data.

[0148] The preferred embodiments will be described in detail below.

[0149] In an embodiment, the data operation processing method comprises the following steps:

[0150] database scheme configuration information and data storage scheme configuration information;

[0151] heterogeneous and heterogeneous data updating or data acquisition;

[0152] heterogeneous and heterogeneous data updating or data acquisition result processing;

[0153] writing operation result information;

[0154] intelligent analysis and intelligent performance optimization prompt;

[0155] intelligent performance optimization / optimization prompt (manual optimization operation on the database is required according to the optimization prompt).

[0156] In an embodiment, receiving the user-submitted database configuration scheme specifically comprises receiving database scheme configuration information and receiving data storage scheme configuration information.

[0157] The data storage scheme configuration information is basic data partition information of data access, and is a basis for improving storage capacity of single data information and improving data access performance, and provides analysis support for performance intelligentization prompt. In combination with the database scheme configuration information, data disaster recovery and data automatic migration can be realized. Meanwhile, in-database partition and data sub-database are set, and quasi-real-time access of massive data can be realized.

[0158] The receiving data storage configuration information includes the following steps:

[0159] The first step is user management of data partition strategy information: filling in a strategy name, selecting a data table strategy type, including creating an instance according to a time type (creating an instance according to a year, a month, a day, etc.), creating an instance according to a business attribute (such as a region code or a device type, etc.), or other ways of cutting data, inputting a related strategy value, and determining whether it is valid during use. The data storage scheme configuration provides partition strategies for the second step.

[0160] The second step is user management of data storage scheme configuration information: editing basic information of a data storage configuration scheme, including a scheme name, data table metadata information, data table field metadata information, a commonly used search field of a configuration table, selecting data partition strategy information, selecting a database configuration scheme, and whether intelligent automatic partitioning, etc. The data storage scheme configuration interface saves, when saved, scheme basic information, a relationship between a scheme ID and data partition strategy information, a relationship between a scheme ID and a database configuration scheme ID, etc.

[0161] The third step is to configure data business logic information: selecting data table metadata information (a related table or a related table), selecting a storage configuration scheme, configuring metadata association relationship (association condition or associated condition), configuring metadata filtering information, saving, when saved, a relationship between table metadata information and a business processing identification number (bizCode), a relationship between a data storage configuration scheme ID corresponding to data table metadata information and a business processing identification number, a relationship between display information of business logic information and a business processing identification number, an association relationship between data table metadata information and other data table metadata information, and an association relationship before a data table metadata information filtering condition.

[0162] In an embodiment, the data access general processing is a core module of simultaneous access of heterogeneous and heterogeneous data, analyzes and disassembles a data operation request of a user in combination with configuration information of a data access configuration scheme and database configuration scheme information, is responsible for operation and legality detection of heterogeneous and heterogeneous partition data, automatically identifies a non-readonly attribute of a database when data is modified and deleted, generates an unoperable instruction for a database type that cannot be modified and deleted, and prompts the user to use a reasonable database manufacturer type. After operation is completed, an operation log is written, to provide artificial intelligence analysis and optimization work of intelligent prompt.

[0163] First step: when the user submits a data access application, first analyze the submitted data access application, split out the relevant data table metadata and data table field metadata and the corresponding database link information, and then obtain the metadata from the data access configuration scheme. Detect whether the submitted metadata has been configured and registered (wherein the data table metadata includes the English name, Chinese name and alias of the data table; the data table field metadata includes the English name, Chinese name, field type of the data table, and retrieval condition English name and Chinese name, etc.), if the detection is passed, then detect from the database configuration scheme whether the database link corresponding database information supports the relevant operation access operation; after the database information is also detected to pass, identify the data operation type (data storage or data reading); provide security for the second step;

[0164] Second step: data general processing module, when OPT_TYPE = DATA_READ, that is, when the data operation type is read mode, analyze the data storage scheme, assemble the data partition information (List <datapartitioninfo>), identifying database configuration schemes, assembling database link information (List <databaselinkinfo>), generate data access unit, assemble and parallel processing data structure (Parallel Data Model), create sub-library sub-table partition task manager (ParallelDataTaskManager), assemble data acquisition task pipeline set (DataLinkedListTaskQueue), start data parallel task.

[0165] According to the database link to the corresponding database to obtain the relevant business data set, parallel task end after the level of data collection, according to the retrieval condition data collection filter, finally assemble the result data (ResultModel), write operation log (OptLogger) and return the result data to the user; when OPT_TYPE = DATA_INSERT / DATA_UPDATE / DATA_DELETE, that is, when the data operation type is update mode, analyze the data storage scheme, assemble the data partition information (List <datapartitioninfo>), identifying database configuration schemes, assembling database link information (List <databaselinkinfo>), generating data access unit, data partition processing, starting automatic partition distributed transaction, creating subunit task of database and table partition (List <parallelatomicdatatask>), data distributed update, distributed transaction completion guarantee, and finally the data update result (ResultModel) is fed back to the user and written into the operation log (OptLogger). The written log provides data basis for intelligent optimization prompt and intelligent analysis;

[0166] In the implementation scheme of analyzing the data storage scheme, when the user submits a data access application to the server, the server obtains the configuration data business logic information according to the business processing identification number (bizCode). Since the configuration data business logic information in the third step of the second point configures the information of the data storage scheme, all the data storage schemes in the configuration data business logic information form a data storage scheme set Map<String, DataStoreConfigInfo>, which is saved in the parallel processing data structure in the parallel table partition task manager (ParallelDataTaskManager) for subsequent use of the data storage scheme information. Since the assembly data partition information is the partition strategy name corresponding relationship in the data storage scheme configuration information, the database configuration scheme is the database scheme corresponding relationship in the data storage scheme configuration information, and the database link information is the data source name corresponding relationship in the data storage scheme configuration information, the assembly data partition information will generate one or more data partition information according to the strategy, and jointly generate the smallest data access unit with the database link information. Each data business logic information configuration information will generate at least one data access unit, and each data access unit is a minimum granularity data access operation (except for simple data table metadata information in small software systems, most software systems involve multiple data access units);

[0167] The parallel data table partition task manager is an execution unit which can simultaneously access one or more databases of different manufacturers and has intelligent performance optimization prompting technology. Each simultaneous access function generates a parallel data table partition task manager. The parallel data table partition task managers uniformly aggregate data information and processing functions. Different access execution units are completely isolated from each other, so that data interference between them is avoided. The parallel data table partition task manager includes a parallel data model, a data acquisition task pipeline set of each processing link execution module, a final result data aggregation module (data pipeline data cascade collection) of each data acquisition task pipeline set, and a conditional data cleaning module of the final result data.

[0168] ParallelDataModel is the data information needed in ParallelDataTaskManager, including all the data information mentioned in the first and second points, containing bizCode, operation type (I data increase; U data modification; D data deletion; G data query) and BizConfigInfoModel, and BizConfigInfoModel contains the relevant information needed by each data storage unit: metaTableInfoMap, databaseConfigInMap, metaTableID2DatabaseIdMap|metaTableName2DatabaseNameMap, metaTableColumnInfoMap, metaTableColumnId2TableIdMap|metaTableColumnName2TableNameMap, metaTableQueryConditionInfoMap, metaTableQueryConditionId2TableIdMap|metaTableQueryConditionName2TableNameMap and metaTableColumnInfoMap. <string>metaTableld2TableColumnldMap | metaTableName2TableColumnNameMap;

[0169] The data acquisition task pipeline set is a collection of data acquisition task pipelines, each of which is responsible for highly encapsulating a data access unit, including obtaining configuration information of the data access unit, loading data segments of the data access unit, segment cleaning and conversion, and processing data results of the segments. Each segment of data can be automatically adjusted according to the type characteristics of the database and the memory capacity of the running machine (generally, the amount of data in each segment is 200-500 data rows), ensuring the most efficient processing of data. Parallel processing of data acquisition task pipelines formed by different data access units forms the data acquisition task pipeline set.

[0170] The data pipeline data cascade collection is a data cascade collection of the data processing results of the data acquisition task pipeline set. The data cascade collection rules and cascade collection levels are determined by the third-step data business logic information configuration information in the data storage scheme configuration information.

[0171] Conditional data cleaning is a process of filtering data during the data pipeline data cascade collection process and results. The conditional data cleaning rules determine the filtered data by the associated conditions and metadata filtering information in the third-step data business logic information configuration information in the data storage scheme configuration information.

[0172] Data partition processing is to split the data to be updated into a data access unit set according to the data table to be simultaneously added, deleted, or modified in the third-step data business logic information configuration information in the data storage scheme configuration information (the data access unit is a subunit task for creating a database table partition, List <parallelatomicdatatask>) and performs the save data operation. When the automatic partition distributed transaction is opened, all data saves are marked as a preprocessing state before all data saves are successful, and when all data saves are successful, the save marks of all data are modified to a save success state, at which time all data access unit sets simultaneously initiate parallel data updates (i.e., data distributed updates) to the database information corresponding to the data access unit; when the automatic partition distributed transaction is opened, the save mark of the data is directly a save success state when saving.

[0173] A preferred embodiment will be described below.

[0174] Step S102, the user submits a data access application to the server;

[0175] Step S1021, the user obtains a request authentication code, and submits the user authCode to the background through the userAuth interface to request a communication token token;

[0176] Step S1022, the user transmits user data, a communication token token, a business processing identification number (bizCode), and an operation type (I data addition; U data modification; D data deletion; G data query) to the background through a data exchange interface dte (dataExchange);

[0177] Step S1023, the data information of ParseUserData splitting user submitted data is executed to generate data table metadata information Map<String, MetaTableInfo> metaTableInfoMap, database configuration information Map<String, DatabaseConfigInfo> databaseConfigInMap, mapping relationship of data table metadata and database configuration information Map<String, String> metaTableID2DatabaseIdMap | metaTableName2DatabaseNameMap, data table field metadata information Map<String, MetaTableColmnInfo> metaTableColumnInfoMap, corresponding relationship of data table field metadata and data table metadata Map<String, String> metaTableColumnId2TableIdMap | metaTableColumnName2TableNameMap, search condition information Map<String, MetaTableQueryConditionInfo> metaTableQueryConditionInfoMap, relationship of search condition and data table metadata Map<String, String> metaTableQueryConditionId2TableIdMap | metaTableQueryConditionName2TableNameMap, and display field information of data table field metadata corresponding to data table metadata Map<String, List <string>metaTableld2TableColumnldMap | metaTableName2TableColumnNameMap;

[0178] Step S104, detecting the legality of user data;

[0179] Step S1041, the user data splitting result of step S102 is obtained through the interface getMetaData, the request content of the user data is checked whether it is in the data storage configuration scheme, if not in the data storage configuration scheme, return ResultModel, and assign value ResultModel.code = -0001, assign value ResultModel.message = "data unauthorized"; if the user data request content verification is passed, then enter step S106 step;

[0180] Step S106, judging the operation type;

[0181] Step S1061, if the operation type is G, if it is a simple query, then jump to the method dataExchangeForGet to execute step S108 step query operation; if it is a complex query, then jump to the method dataExchangeForPost to execute step S118 step query operation;

[0182] Step S1062, if the operation type is U, then jump to the method dataExchangeForPost to execute step S118 step update operation;

[0183] Step S1063, if the operation type is D, then jump to the method dataExchangeForPost to execute step S118 step delete operation;

[0184] Step S1064, if the operation type is I, then jump to the method dataExchangeForPost to execute step S118 step add operation;

[0185] Step S108, executing data query operation;

[0186] Step S1081, from step S102, communication token token, business processing identification number bizCode;

[0187] Step S1082, parsing the user data split in step S102;

[0188] Step S110, assembling data acquisition partition table database information;

[0189] Step S1101, obtain the partition information Map<String, List of the data table metadata corresponding to the user data through parseDataPartitionInfo <datapartitioninfo>> metaTablePartionInfoMap and the mapping relationship of data table name and database link name Map<String, String> metaTableName2DbNameMap;

[0190] In step S1102, the database configuration information Map<string, List of the data table metadata corresponding to the user data is obtained through parseDatabaseLinkInfo. <databaselinkinfo>> dbName2DatabaseLinkInfoMap.

[0191] Step S112, create data partition parallel task manager;

[0192] Step S1121, judge whether it is data addition or data deletion, if not, jump to next step, create data partition table database information Map <String, DataPartitionInfo> partitionId2DataPartitionInfoMap through createDataPartitionInfo interface to the result data of step S1063 and step S1064.

[0193] Step S1122, instantiate ParallelDataTaskManager parallel task manager, pass data partition table database information partitionId2DataPartitionInfoMap into parallel task manager, and instantiate parallel task data structure.

[0194] Step S114, create data table database manager;

[0195] Create data table database task pipeline set Queur through metaTablePartionInfoMap, metaTableName2DbNameMap and dbName2DatabaseLinkInfoMap of step S110 <datalinkedlisttaskqueue>and data hierarchical data structure Map<String, List<List<Map<String, Object>>> tableName2Part2DataListMap.

[0196] Step S116, execute data acquisition operation;

[0197] Step S1161, execute ParallelDataTaskManager.start() to start parallel execution of the database and table partition task ParallelDataTaskItem, ParallelDataTaskItem performs self-encapsulation of database manufacturer driver (database manufacturer driver includes mysql, mssql, oracle, Damon, Huawei GaussDB, Redis, h2, sqlite, bigtable, HBase, MongoDB, hdfs, Hadoop, Spark, etc.) for database manufacturer information corresponding to the partition data table, each ParallelDataTaskItem is executed in Thread, and the data executed by ParallelDataTaskItem is stored in the data hierarchical data structure tableName2Part2DataListMap.

[0198] Step S1162, clean the data hierarchical data structure tableName2Part2DataListMap according to the data cleaning condition, and store the cleaned data in List<Map<String, String>> retDataList.

[0199] Step S1163, integrate the execution result into the pre-established result model ResultModel, if the execution is successful, ResultModel.data = retDataList; if the execution is not successful, ResultModel.code = 0005, and ResultModel.message = "data acquisition failed".

[0200] Step S1164, write data operation log through OptLogger, and the log information includes LogID, start time, end time, execution database link information, execution database partition information, and execution unit ID.

[0201] Step S1165, return the query result;

[0202] Step S118, verify the compliance of the data update operation;

[0203] Step S1181, from the step S102 communication token token, business processing identification number bizCode;

[0204] Step S1182, parsing step S102 split user data;

[0205] Step S1183, execute step S110 assembly data acquisition partition table database information;

[0206] Step S1184, according to the database manufacturer type, through isDatabaseAllow method to identify whether the data update operation is allowed.

[0207] Step S120, data update data structure assembly;

[0208] Step S1201, execute step S112, create data partition parallel task manager ParallelDataTaskManager.

[0209] Step S1202, create database table partition subunit task manager ParallelAtomicDataTask, build subunit task data structure ParallelAtomicDataInfo, each subunit task contains a subunit task data structure, subunit task data structure ParallelAtomicDataInfo contains data table metadata information DataPartitionInfo, database link DatabaseLinkInfo and retrieval condition information MetaTableQueryConditionInfo.

[0210] Step S122, execute data update operation;

[0211] Step S1221, through checkTransaction to identify whether to start transaction processing.

[0212] Step S1222, start parallel subunit task ParallelAtomicDataTask.start to carry out distributed update operation.

[0213] Step S124, data update is completed;

[0214] Step S1241, identify all update operation has completed ParallelAtomicDataTask.isFinish.

[0215] Step S1242, the execution result is incorporated into the execution result model ResultModel, and if the execution is successful, ResultModel.data = ParallelAtomicDataTask.IUD.Size; if the execution is not successful, ResultModel.code = 0015, and ResultModel.message = "data update failed" is assigned.

[0216] Step S1243, data operation logs are written through OptLogger, and the log information includes LogID, start time, end time, execution database link information, execution database partition information, execution unit ID, etc.

[0217] Step S1244, the query result is returned.

[0218] The code example of the above embodiment is as follows:

[0219] The submitted data format example is

[0220]

[0221] The data format example of the returned execution result is:

[0222] {

[0223] isSucces:true

[0224] code:200

[0225] message:”

[0226] data:{

[0227] "uid”:”uojamns,mdfpqiuoikjsdfhk”,

[0228] "dataField1”:"hello-linwm”,

[0229] "dataField2”:1,

[0230] "dataField3”:”2020-01-02”

[0231] }

[0232] }

[0233] Those skilled in the art can understand that the modules and steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0234] In several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0235] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.< / datalinkedlisttaskqueue> < / databaselinkinfo> < / datapartitioninfo> < / string> < / parallelatomicdatatask> < / string> < / parallelatomicdatatask> < / databaselinkinfo> < / datapartitioninfo> < / databaselinkinfo> < / datapartitioninfo>

Claims

1. A data manipulation processing method, characterized in that, Includes the following steps: Obtain the user's database configuration information and create a database configuration scheme; Obtain the user's data access configuration information and create a data access configuration scheme; Obtain the user's data operation request and split the data information in the data operation request; Based on the existing data access configuration scheme and database configuration scheme, security checks are performed on the split data separately. Determine the operation type of the data that passes the security check, reassemble the data information, and generate a data access unit; When the operation type of the data that passes the security test is the first operation mode, the generated data access unit is processed in parallel through the first processing method to obtain the result data. When the operation type of the data that passes the security test is the second operation mode, the generated data access unit is processed in parallel using the second processing method to obtain the result data. The data access configuration information includes a list of data partitioning strategies and a list of data access configuration schemes; Obtaining user data access configuration information and creating a data access configuration scheme includes, Obtain the user's list of data partitioning policies and create a data partitioning policy. Obtain the user's list of data access configuration schemes, and create a data access configuration scheme based on the existing data partitioning strategy and database configuration scheme. The data access configuration information also includes a list of data service logic relationship configuration schemes. Obtaining user data access configuration information and creating data access configuration schemes also includes the following steps: obtaining a list of user data business logic relationship configuration schemes and creating a data business logic relationship configuration scheme based on the created data access configuration schemes; Obtaining data storage configuration information includes the following steps: User management data partitioning strategy information: Enter the strategy name, select the data table strategy type, including creating instances by time type, by business attribute, or other methods that can split data, enter the relevant strategy values, usage period, and whether it is effective, to provide partitioning strategies for configuring data storage solutions; User management data storage scheme configuration information: Edit the basic information of the data storage configuration scheme, including scheme name, data table metadata information, data table field metadata information, configuration table commonly used search fields, selected data partitioning strategy information, selected database configuration scheme, and whether intelligent automatic partitioning is enabled. The data storage scheme configuration interface saves the basic scheme information, the relationship between scheme ID and data partitioning strategy information, and the relationship between scheme ID and database configuration scheme ID when saving. Configure data business logic information: Select data table metadata information, select storage configuration scheme, configure metadata association, configure metadata filtering information, save the relationship between table metadata information and business processing identifier, save the relationship between the data storage configuration scheme ID corresponding to the data table metadata information and the business processing identifier, save the relationship between the display information of business logic information and the business processing identifier, save the association between data table metadata information and other data table metadata information, and save the association between the filtering conditions of data table metadata information.

2. The data manipulation processing method according to claim 1, characterized in that, The database configuration information includes a database connection list, a database instance policy list, and a database configuration list; Obtain the user's database configuration list and create a database configuration scheme, including the following steps: Obtain the user's database connection list and create database connection information; Obtain the user's database instance policy list and create a database instance generation policy; Obtain the user's database configuration list and create a database configuration scheme based on the created database connection information and database instance generation strategy.

3. The data manipulation processing method according to claim 2, characterized in that, The database connection list includes information such as database name, database type, database address, database instance information, login account and password, database configuration parameters, usage period, validity status, and tablespace.

4. The data manipulation processing method according to claim 2, characterized in that, The database instance policy list includes: policy name, policy generation type, regional dimension, numerical dimension, and other regular dimensions and the effective time of the database shards.

5. The data manipulation processing method according to claim 2, characterized in that, The database configuration list includes: database configuration scheme name, database vendor type, default prefix name of database instance, database connection name, and database instance generation strategy information.

6. The data manipulation processing method according to claim 1, characterized in that, The split data includes table metadata, table field metadata, and database link information; Perform security checks on the split data separately, including the following steps: The split data table metadata and data table field metadata are matched with the corresponding data in the created data access configuration scheme. If they match, the security check passes; otherwise, the security check fails. The split database connection information is matched with the corresponding database connection information in the created database configuration scheme. If they match, the security check passes; otherwise, the security check fails.

7. The data manipulation processing method according to claim 1, characterized in that, The process of reassembling data information and generating a data access unit includes the following steps: Analyze the data access configuration scheme where the data information in the data operation request is located, and obtain the data partition information corresponding to the data information in the data operation request; Assemble the data partition information corresponding to the data information in the data operation request; Identify the database configuration scheme where the data information in the data operation request is located, and obtain the database connection information corresponding to the data information in the data operation request; Assemble the database connection information corresponding to the data information in the data operation request; A data access unit is generated based on the data partition information and database link information corresponding to the data information in the data operation request.

8. The data manipulation processing method according to claim 1, characterized in that, When the operation type of the data that passes the security check is the first operation mode, the generated data access unit is subjected to parallel computation using the first processing method to obtain the result data, including the following steps: The generated data access unit is partitioned to obtain the first data group; Determine whether the first data group is a multi-link or partition; If the first data group is a multi-link or partitioned data group, then determine whether the first data group supports transactions. If the first data group is not a multi-link or partitioned data group, then directly perform parallel computing and distributed data updates. If the first data group supports transactions, then automatic partitioned distributed transactions are enabled, and parallel computing and distributed data updates are performed. Otherwise, sharding and partitioning sub-unit tasks are created directly, and parallel computing and distributed data updates are performed.

9. The data manipulation processing method according to claim 8, characterized in that, The first operation mode is the update mode, which includes add, modify, or delete operations.

10. The data manipulation processing method according to claim 1, characterized in that, When the operation type of the data that passes the security check is the second operation mode, the generated data access unit is processed in parallel using the second processing method to obtain the result data, including the following steps: Obtain data information from the generated multiple data access units; The data information in the multiple data access units is processed in parallel. The data processing results are summarized after parallel processing of the data information in multiple data access units; The data processing results are cleaned to obtain the final data.

11. A data manipulation processing system, characterized in that, The processing system includes: The first configuration module is used to create database configuration schemes; The second configuration module is used to create data access configuration schemes; The data splitting module is used to obtain the user's data operation request and split the data information in the data operation request; The security detection module is used to perform security checks on the split data based on the created data access configuration scheme and database configuration scheme. The operation type determination module determines the operation type of data that has passed the security check. A data access information generator is used to reassemble data information and generate data access units. The first executor for heterogeneous data is used to perform parallel computation on the generated data access unit and obtain the result data through the first processing method when the operation type of the data that has passed the security detection is the first operation mode. The heterogeneous data second actuator performs parallel computation on the generated data access unit and obtains the result data through the second processing method when the operation type of the data that passes the security detection is the second operation mode. The data access configuration information includes a list of data partitioning strategies and a list of data access configuration schemes. Obtaining user data access configuration information and creating a data access configuration scheme includes, Obtain the user's list of data partitioning policies and create a data partitioning policy. Obtain the user's list of data access configuration schemes, and create a data access configuration scheme based on the existing data partitioning strategy and database configuration scheme. The data access configuration information also includes a list of data service logic relationship configuration schemes. Obtaining user data access configuration information and creating data access configuration schemes also includes the following steps: obtaining a list of user data business logic relationship configuration schemes and creating a data business logic relationship configuration scheme based on the created data access configuration schemes; Obtaining data storage configuration information includes the following steps: User management data partitioning strategy information: Enter the strategy name, select the data table strategy type, including creating instances by time type, by business attribute, or other methods that can split data, enter the relevant strategy values, usage period, and whether it is effective, to provide partitioning strategies for configuring data storage solutions; User management data storage scheme configuration information: Edit the basic information of the data storage configuration scheme, including scheme name, data table metadata information, data table field metadata information, configuration table commonly used search fields, selected data partitioning strategy information, selected database configuration scheme, and whether intelligent automatic partitioning is enabled. The data storage scheme configuration interface saves the basic scheme information, the relationship between scheme ID and data partitioning strategy information, and the relationship between scheme ID and database configuration scheme ID when saving. Configure data business logic information: Select data table metadata information, select storage configuration scheme, configure metadata association, configure metadata filtering information, save the relationship between table metadata information and business processing identifier, save the relationship between the data storage configuration scheme ID corresponding to the data table metadata information and the business processing identifier, save the relationship between the display information of business logic information and the business processing identifier, save the association between data table metadata information and other data table metadata information, and save the association between the filtering conditions of data table metadata information.

12. The data manipulation processing system according to claim 11, characterized in that, The heterogeneous data first actuator includes: The data partitioning processing unit is used to perform data partitioning processing on the generated data access unit and obtain the first data group; Partitioned distributed transaction unit, used to enable automated partitioned distributed transactions; The database sharding and partitioning sub-unit is used to create database sharding and partitioning sub-unit tasks. The distributed data update unit is used to perform distributed data updates.

13. The data processing system according to claim 11, wherein the heterogeneous data second actuator comprises: The data acquisition unit is used to acquire data information from the multiple data access units generated; A data pipeline processing unit is used to process data information in multiple data access units in parallel; The data pipeline aggregation unit is used to aggregate the data processing results after parallel processing of data information from multiple data access units; The data cleaning unit is used to clean the data processing results and obtain the result data.

Images