A dynamic data processing method, system and device based on sub-library and sub-table

Abstract

The present invention relates to a dynamic data processing method, system and device based on sub-library and sub-table. The method part mainly includes: receiving input data and binding the input data with a corresponding model identifier, obtaining relevant data according to the bound model identifier and the current user identifier; verifying the input data according to the relevant data, retaining the input data that meets the requirements after verification, and calculating the sub-library and sub-table data; assigning the verified input data and the sub-library and sub-table data to an entity object, and performing ORM by the entity object to generate SQL; executing the generated SQL, binding the returned data to the entity object corresponding to the ORM, and returning the returned data to the caller after verifying it according to the relevant data corresponding to the entity object. The present invention can dynamically adapt to data of different formats without generating code or editing, and can adapt without development, thereby improving the efficiency of development and operation and maintenance.

Description

Technical Field

[0001] The present invention relates to the field of model-driven technology, and in particular to a dynamic data processing method, system and device based on sub-library and sub-table. Background Art

[0002] As the trend of enterprise digitalization and cloud migration intensifies, more and more personalized SaaS (Software-as-a-Service) applications require faster and more efficient development. Model Driven uses visual modeling technology to define data and its relationships, enabling developers and business users to quickly process various structured data and deliver applications without the need for code.

[0003] At present, the model-driven technology of the prior art generally adopts the method of dynamically generating codes and then performing CI (Continuous Integration) / CD (Continuous Deployment). Such technology requires the generation of codes and the completion of activities such as packaging and deployment before the relevant programs can take effect. The second method uses dynamic compilation to generate dynamic classes to bind objects. Such technology can take effect in real time, but requires dynamic compilation, which is easy to cause program exceptions. The third method uses a large number of reserved fields of different types, such as reserving 200 string fields to solve the ORM (Object Relational Mapping) and data processing of entities with no more than 200 string attributes. Such technology does not fundamentally solve the problem of dynamic changes in fields, and the code still needs to be changed after exceeding the range. The last method uses pure SQL (Structured Query Language) for data processing. Such technology can adapt to various formats of data, but it needs to adapt to various databases by splicing SQL by itself, and object binding after obtaining data requires additional code to implement.

[0004] On the other hand, personalized SaaS applications require that the same business be implemented differently according to business changes, and the data structure and storage library will change due to the business. The maturity of database and table sharding technology makes it possible to operate data from different libraries and tables at the same time, such as: vertical database sharding, classifying tables according to business and distributing them to different databases; vertical table sharding, dividing a table into multiple tables according to fields, each table storing a part of the fields; horizontal database sharding, splitting the data of the same table into different databases according to certain rules, and each database can be placed on a different server; horizontal table sharding, in the same database, splitting the data of the same table into multiple tables according to certain rules. Generally, database and table sharding mainly splits a table corresponding to an entity class, and the split tables have a certain correlation, which is to split the data to solve the same business. It cannot handle data from multiple types of businesses well.

[0005] In view of the above situation, how to overcome the defects of the existing technology and solve the problems of dynamic data processing requiring development and code generation, low adaptability, and inability to dynamically adapt to different data formats of various businesses are difficult problems to be solved in this technical field. Summary of the invention

[0006] In response to the above defects or improvement needs of the prior art, the present invention provides a dynamic data processing method, system and device based on sub-library and sub-table. It does not require development and can bind different business data to an entity object without generating code or compiling. It can dynamically adapt to different data formats and process data from different types of databases.

[0007] The embodiment of the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a dynamic data processing method based on sub-library and sub-table, comprising:

[0009] Receive input data and bind the input data to the corresponding model identifier, and obtain relevant data according to the bound model identifier and the current user identifier;

[0010] Verify the input data according to the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-database and sub-table data;

[0011] Assign the verified input data and the sub-library and sub-table data to the entity object, and perform ORM on the entity object to generate SQL;

[0012] Execute the generated SQL, bind the returned data to the entity object corresponding to the ORM, verify the returned data according to the relevant data corresponding to the entity object, and then return it to the caller.

[0013] Furthermore, the relevant data includes model and model field data as well as tenant and role permission data, and the obtaining of relevant data according to the bound model identifier and the current user identifier specifically includes:

[0014] Get the model and model field data according to the bound model identifier;

[0015] Get the tenant and role permission data of the current user based on the current user ID.

[0016] Furthermore, the step of verifying the input data according to the relevant data, retaining the input data that meets the requirements, and calculating the sub-library and sub-table data specifically includes:

[0017] Filter model field data according to tenant and role permission data and validate input data with the filtered model field data, retaining input data that meets the requirements after validation;

[0018] Calculate the sub-library and sub-table data based on the model and model field data as well as the tenant and role permission data.

[0019] Furthermore, the entity object is bound to a corresponding model identifier and contains sub-library and sub-table attributes and extensible key-value attributes.

[0020] Furthermore, in the process of performing ORM by the entity object to generate SQL, the sub-library and sub-table attributes and the key-value attributes are expanded and flattened, the key and value hierarchy of the key-value attributes are kept consistent with the hierarchy of the sub-library and sub-table attributes, all the keys of the sub-library and sub-table attributes and the key-value attributes are bound to the fields of SQL, and the values ​​of all the sub-library and sub-table attributes and the value of the key-value attributes are used as SQL parameters.

[0021] Furthermore, the verifying the returned data according to the relevant data corresponding to the entity object and returning the data to the caller specifically includes:

[0022] The corresponding related data is obtained according to the model identifier bound to the entity object, and the returned data is verified according to the model and model field data in the related data as well as the tenant and role permission data before being returned to the caller.

[0023] Furthermore, the SQL generated by executing specifically includes:

[0024] The generated SQL is routed to different types of databases for execution according to the pre-configured sharding rules, and the data after execution in the database is returned.

[0025] In a third aspect, the present invention provides a dynamic data processing system based on sharding of libraries and tables, which is used to implement the dynamic data processing method based on sharding of libraries and tables as described in the first aspect, wherein the system includes data modeling, access control, interpretation and execution services, and sharding of libraries and tables middleware;

[0026] The data modeling provides visual modeling capabilities, generates models and model field data, including relevant configuration data for database and table sharding rules;

[0027] The access control provides visual tenant and role permission configuration capabilities, generates tenant and role permission data for filtering models and model field data, and tenant and part of role permission data for database and table sharding rules;

[0028] The interpretation and execution service is used to obtain input data in different formats, bind the input data to the corresponding model identifier, and obtain relevant data according to the bound model identifier and the current user identifier; verify the input data according to the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-library and sub-table data; assign the verified input data and sub-library and sub-table data to the entity object, and perform ORM on the entity object to generate SQL; when the sub-library and sub-table middleware returns the data after executing SQL, bind the returned data to the entity object corresponding to the ORM, verify the returned data according to the relevant data corresponding to the entity object, and return it to the caller;

[0029] The library and table sharding middleware is used to obtain the generated SQL, route the SQL to different types of databases for execution according to the library and table sharding rules, and return the data in the database.

[0030] Furthermore, the data modeling, access control, interpretation and execution services, and sub-library and sub-table middleware provide corresponding functions by being separated into multiple microservices or merged into one microservice. When the sub-library and sub-table middleware is deployed independently, the sub-library and sub-table middleware connects to different types of databases and provides the function of dynamically increasing or decreasing database connections, thereby realizing access to other unconnected databases during operation.

[0031] In the third aspect, the present invention provides a dynamic data processing device based on sub-library and sub-table, specifically: including at least one processor and a memory, at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the dynamic data processing method based on sub-library and sub-table in the first aspect.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) Using the same entity object for ORM for data of different structures can dynamically adapt to data of different formats without generating code or editing. During system operation, data format changes take effect in real time, reducing the time for compilation, packaging, and publishing. For methods with different data formats but the same logic, they can be adapted without development, improving the efficiency of development and operation and maintenance.

[0034] (2) The use of ORM and sub-library and sub-table middleware shields developers from the read and write operations of the data layer, allowing developers to simultaneously perform read and write operations on different types of databases without having to worry about the adaptation of the data layer, thereby reducing the difficulty of development.

[0035] (3) The same business data provides different data storage methods for different tenants and users based on business differences, so that the actual business database, table, and table fields are different due to different tenants and role permissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A flowchart of a dynamic data processing method based on sub-library and sub-table provided in Example 1 of the present invention;

[0038] Figure 2 An expanded flow chart of step 200 provided in Embodiment 1 of the present invention;

[0039] Figure 3 A schematic diagram of the module structure of a dynamic data processing system based on sub-library and sub-table provided in Example 2 of the present invention;

[0040] Figure 4 A schematic diagram of a system structure in an implementation manner provided in Example 2 of the present invention;

[0041] Figure 5 A schematic diagram of a system structure in another implementation manner provided in Example 2 of the present invention;

[0042] Figure 6 A schematic diagram of a system function implementation flow provided in Example 2 of the present invention;

[0043] Figure 7 A schematic diagram of the structure of a dynamic data processing device based on sub-library and sub-table provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0046] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0047] Embodiment 1:

[0048] like Figure 1 As shown, an embodiment of the present invention provides a dynamic data processing method based on sub-library and sub-table, and the method includes the following steps:

[0049] Step 100: Receive input data and bind the input data to a corresponding model identifier, and obtain relevant data according to the bound model identifier and the current user identifier. In this step, the corresponding model identifier is the model identifier contained in the received input data or the model identifier passed in the interface parameter. The relevant data includes model and model field data as well as tenant and role permission data. Specifically, obtaining relevant data according to the bound model identifier and the current user identifier includes two steps: obtaining the model and model field data according to the bound model identifier; obtaining the tenant and role permission data of the current user according to the current user identifier.

[0050] Step 200: Verify the input data according to the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-library and sub-table data. This step verifies the input data according to the relevant data, that is, the model and model field data, as well as the tenant and role permission data, to obtain the input data that meets the requirements. In addition, the sub-library and sub-table data can be calculated through the above-mentioned relevant data. Specifically, Figure 2 As shown, step 200 can be specifically divided into the following two steps: Step 201: filter the model field data according to the tenant and role permission data and use the filtered model field data to verify the input data, and retain the input data that meets the requirements after verification. Step 202: calculate the sub-library and sub-table data according to the model and model field data as well as the tenant and role permission data.

[0051] Step 300: Assign the validated input data and sub-library and sub-table data to the entity object, and perform ORM by the entity object to generate SQL. The entity object of this step is the same entity object when facing data in different formats. When the input data and sub-library and sub-table data are assigned to the entity object, the entity object is bound to the corresponding model identifier and contains sub-library and sub-table attributes and extensible key-value attributes, which are used to store business data in different formats. All ORMs are bound by the entity object. In this step of the preferred embodiment, in the process of performing ORM by the entity object to generate SQL, it is necessary to expand and flatten the sub-library and sub-table attributes and the key-value attributes, keep the key and value hierarchy of the key-value attributes consistent with the hierarchy of the sub-library and sub-table attributes, bind all the sub-library and sub-table attributes and the key of the key-value attributes to the fields of SQL, and use the values ​​of all the sub-library and sub-table attributes and the value of the key-value attributes as SQL parameters to generate SQL.

[0052] Step 400: Execute the generated SQL, bind the returned data to the entity object corresponding to the ORM, and return the returned data to the caller after verifying it according to the relevant data corresponding to the entity object. In this step, executing the generated SQL specifically includes: routing the generated SQL to different types of databases for execution according to the pre-configured sub-library and sub-table rules, and returning the data after execution in the database. After the SQL is executed, bind the returned data to the entity object corresponding to the ORM, and then obtain the corresponding relevant data according to the model identifier bound to the entity object. After verifying the returned data according to the model and model field data in the relevant data as well as the tenant and role permission data, the data that meets the verification conditions is returned to the caller.

[0053] In this preferred embodiment, the above entity object structure is as follows:

[0054] public class FitObject <id>{

[0055] private LocalModelCache localModel; / / model data

[0056] private LocalModelField localModelField; / / Model field data

[0057] private ID id; / / Primary key. If the model's primary key field is not id, copy the value

[0058] private String model; / / used for sub-library and sub-table

[0059] private String tenant; / / used for sub-library and sub-table

[0060] private String service; / / used for sub-library and sub-table

[0061] Private Map<String,Object> dynamicAttributes; / / key-value attributes

[0062] }

[0063] Among them, the attributes localModel (core model information, such as the identifier of the microservice where the model is located, the primary key field, etc.) and localModelField (core model field information, such as the field name contained in the model, the data type of the field, etc.) are used to temporarily store the core data of the model and model fields obtained through the model identifier. The attributes id (primary key), model (model identifier), tenant (tenant identifier), and service (microservice identifier) ​​do not participate in the write operation when the field does not exist in the database table. When reading, they are assigned according to the value of the model, tenant, service or the value stored in the table. The value of the attribute id is assigned with the value of the primary key field in the model information. The attribute dynamiceAttributes is a key-value attribute, which contains the model fields and their values ​​defined and verified by the model. Different models use this entity structure for ORM. For different data structures, the core change is only the content in the attribute dynamiceAttributes. The entity structure remains unchanged, the ORM binding logic remains unchanged, and the code remains unchanged.

[0064] To sum up, through the above steps, this embodiment can bind different business data to an entity object through visual modeling, tenants, and permission control without development, without generating code or compiling. Based on the rules of sub-library and sub-table, the entity object can be simultaneously bound to tables of different formats in different types of databases, dynamically adapt to different data formats, process data in different types of databases, and make configuration modifications take effect in real time. The same business data can be stored in different ways for different tenants and users.

[0065] Embodiment 2:

[0066] Based on the dynamic data processing method based on sub-library and sub-table provided in the above-mentioned embodiment 1, this embodiment 2 takes an example of a specific implementation scenario to explain the dynamic data processing method based on sub-library and sub-table in more detail.

[0067] For step 100 (receiving input data and binding the input data with the corresponding model identifier, and obtaining relevant data according to the bound model identifier and the current user identifier), this embodiment receives the purchase order data submitted by user User1 through the interface http: / / domain / purchaseorder, wherein the received order data contains information such as orderno (order number), type (purchase type), supplier (supplier), paymentTerms (payment terms), arrivalDate (arrival date), amount (total amount), etc., binds the order data with the model identifier purchaseorder obtained in the interface URL, and obtains relevant data according to the bound model identifier purchaseorder, such as: the identifier of the microservice where the model is located is purchase, and the model contains field names orderno (order number, which is the primary key), type (purchase type), supplier (supplier), paymentTerms (payment terms), amount (total amount), and obtains relevant data according to the current user identifier User1, such as: the tenant identifier of User1 is 0001, and the available fields in the field permissions corresponding to tenant 0001 are orderno, type, supplier, and paymentTerms. For the same interface, by switching different model identifiers, users can dynamically adapt and receive business data in different formats. In this step, the corresponding model identifier obtains the value of the model identifier model through URL, Cookie, Header, Body, etc. (such as: http: / / domain name / {model}, http: / / domain name / interface address?model={model}, the received data contains the attribute model, Cookie, Header contains the model identifier).

[0068] For step 200 (validating the input data according to relevant data, retaining the input data that meets the requirements after validation, and calculating the sub-library and sub-table data), this embodiment first filters the model field data according to the tenant and role permission data and validates the input data with the filtered model field data. For example, tenant 0001 defines seven fields, namely model, tenant, service, orderno, type, supplier, and paymentTerms, in the t_purchaseorder table of the database, and tenant 0002 defines eight fields, namely model, tenant, service, orderno, type, supplier, paymentTerms, and amount, in the t_purchaseorder table of the database. The purchase order model purchaseorder is defined. The five fields orderno (order number), type (purchase type), supplier (supplier), paymentTerms (payment terms), and amount (total amount) are passed in. In fact, six fields orderno (order number), type (purchase type), supplier (supplier), paymentTerms (payment terms), arrivalDate, and amount (total amount) are passed in. However, tenant 0001 only has the permissions for the four fields orderno, type, supplier, and paymentTerms. Therefore, only the input data of the four attributes orderno, type, supplier, and paymentTerms after verification are retained. That is, according to the differences in business, the same model stores different table structures for different tenants and permissions.Then, this embodiment calculates the sub-library and sub-table data according to the model and model field data as well as the tenant and role permission data. The calculation here is mainly to define the calculation rules or calculation expressions, substitute the model and model field data as well as the tenant and role permission data for calculation to obtain the values ​​of the actual sub-library and sub-table fields, such as: the data model identifier purchaseorder, the model type default (such as: default represents a normal table, view represents a view), the microservice identifier purchase, and the user's tenant identifier 0001 have been obtained. When the sub-library rule of the model purchaseorder is the microservice identifier + "_" + tenant identifier, and the sub-table rule is the model type + "_" + data model identifier , indicating that the data of different microservices of different tenants need to be separated into different libraries, and the table name is determined by the model type and model identifier, that is, the final actual library name is purchase_0001, and the actual table name is t_purchaseorder (such as: the value corresponding to the model type default is t, and the value corresponding to the model type view is v). When the library partition rule of the model purchaseorder is the microservice identifier, and the table partition rule is model type + "_" + data model identifier + "_" + tenant identifier, it indicates that the data of different microservices need to be separated into different libraries, and the data of different tenants need to be separated into different tables in the same library, that is, the final actual library name is purchase, and the actual table name is t_purchaseorder_0001. According to the differences in business, the same model (business data) implements different database and table storage methods for different tenants and users.

[0069] For step 300, this embodiment assigns the verified input data and the sub-library and sub-table data to the entity object. The entity object contains model (model identifier), tenant (tenant identifier), service (microservice identifier), and dynamicAttributes (key-value attributes), where dynamicAttributes includes orderno (order number), type (purchase type), supplier (supplier), and paymentTerms (payment terms). The entity object performs ORM to generate an SQL statement such as: insert into t_purchaseorder(model, tenant, service, orderno, type, supplier, payment_terms) values(?,?,?,?,?,?,?,?), where "?" is replaced by the parameter value passed in by ORM during execution. The entity objects of this step are all of the same entity object structure when facing data in different formats. When the data format changes, there is no need for development, code generation, or compilation. Different data formats are controlled and adapted through models, tenants, permissions, and other data. When the input data and sub-library and sub-table data are assigned to the entity object, the entity object is bound to the corresponding model identifier and contains sub-library and sub-table attributes and extensible key-value attributes. The attributes are used to store business data in different formats, and all ORMs are bound by the entity object. In this step of the preferred embodiment, in the process of performing ORM by the entity object to generate SQL, it is necessary to expand and flatten the sub-library and sub-table attributes and the key-value attributes, keep the key and value levels of the key-value attributes consistent with the level of the sub-library and sub-table attributes, bind all the sub-library and sub-table attributes and the key-value attributes to the fields of SQL, and use the values ​​of all the sub-library and sub-table attributes and the value of the key-value attributes as SQL parameters to generate SQL.

[0070] For step 400, this embodiment first executes the generated SQL, such as the query statement select orderno, type, supplier, paymentTerms, amount, orderno as id, model, tenant, service from t_purchaseorder where model = 'purchaseorder' and tenant = '0001' and service = 'purchase'. According to the data of different microservices that need to be divided into different libraries and tables, the data of different tenants in the same library needs to be divided into different libraries and tables. The SQL statement will be routed to t_purchaseorder_0001 (model = 'purchaseorder' and tenant='0001') table, so that multiple libraries and tables can be operated at the same time through different sub-library and sub-table values ​​without changing the application. After SQL is executed, the returned data will be bound to the entity object corresponding to ORM, and the returned data will be verified according to the relevant data corresponding to the entity object. Since the current model purchaseorder does not define the model (model identifier), tenant (tenant identifier), and service (microservice identifier) ​​fields, only the information of the four fields orderno (order number), type (purchase type), supplier (supplier), and paymentTerms (payment terms) is returned to the caller. After SQL is executed, the returned data will be bound to the entity object corresponding to ORM, and then the corresponding relevant data will be obtained according to the model identifier bound to the entity object. The returned data will be verified according to the model and model field data in the relevant data as well as the tenant and role permission data, such as: whether the query field of the SQL statement conforms to the field defined by the model, whether the data type is consistent with the model definition, and whether the tenant or user has the permission for the field. The data that meets the verification conditions will be returned to the caller.

[0071] Embodiment 3:

[0072] Based on the dynamic data processing method based on sub-libraries and sub-tables provided in Example 1, this Example 3 provides a dynamic data processing system based on sub-libraries and sub-tables, which is used to implement the dynamic data processing method based on sub-libraries and sub-tables as described in Example 1.

[0073] like Figure 3 As shown, the system provided in this embodiment 2 includes data modeling, access control, interpretation and execution services, and sub-library and sub-table middleware.

[0074] Specifically, the data modeling provides visual modeling capabilities, generates models and model field data, including relevant configuration data for database and table sharding rules.

[0075] The access control provides visual tenant and role permission configuration capabilities, generates tenant and role permission data for filtering models and model field data, and tenant and partial role permission data are used for database and table sharding rules.

[0076] The interpretation and execution service is used to obtain input data in different formats, bind the input data with the corresponding model identifier, and obtain relevant data based on the bound model identifier and the current user identifier; verify the input data based on the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-library and sub-table data; assign the verified input data and sub-library and sub-table data to the entity object, and perform ORM on the entity object to generate SQL; when the sub-library and sub-table middleware returns the data after executing SQL, the returned data is bound to the entity object corresponding to the ORM, and the returned data is verified based on the relevant data corresponding to the entity object and then returned to the caller.

[0077] The library and table sharding middleware is used to obtain the generated SQL, route the SQL to different types of databases for execution according to the library and table sharding rules, and return the data in the database.

[0078] It should be noted that the above-mentioned interpretation and execution service is the main service module for processing data in this embodiment. After obtaining input data in different formats, it first binds the input data to the model identifier, and obtains the model and model field data through the model identifier, then obtains the tenant and role permission data through the current user identifier, and then filters the model field with the tenant and role permission data, verifies the input data according to the filtered field data, retains the input data that meets the requirements, and then assigns the sub-library and sub-table data calculated according to the model and model field data, tenant and role permission data and the verified input data to the entity object. The entity class is bound to the model identifier and contains sub-library and sub-table rule attributes and extensible key-value type attributes. The entity object performs ORM to generate SQL. In the process of generating SQL, all sub-library and sub-table attributes and key-value attributes are bound to the SQL fields, and the values ​​of the sub-library and sub-table attributes and the value of the key-value attributes are used as SQL parameters. Finally, when the sub-library and sub-table middleware returns data, the data is bound to the entity object corresponding to the ORM. Because the entity object contains the model identifier, the corresponding related data can be found and the returned data can be returned to the caller after verification according to the model and field data, tenant and role permission data.

[0079] Based on the above system design, data modeling, access control, interpretation and execution services, and sub-library and sub-table middleware can be deployed in different ways, by separating into multiple microservices or merging into one microservice to provide corresponding functions. When the sub-library and sub-table middleware is deployed independently, it connects to different types of databases and provides the function of dynamically increasing or decreasing database connections, so as to realize access to other unconnected databases during operation.

[0080] like Figure 4 As shown, for example, in one implementation of this embodiment, the system includes a data configuration service and an interpretation and execution service. The data configuration service includes data modeling and access control, provides visual modeling and tenant and role permission configuration capabilities, and generates models and model field data, tenant and role permission data. The interpretation and execution service is directly integrated with the sub-library and sub-table middleware, and the entire interpretation and execution service is used to connect to different databases, dynamically adapt to different structured data, and after the input data is verified by the model and model field, tenant and role permission, the sub-library and sub-table information is attached, and then the generated SQL is routed to the connected different types of databases through the sub-library and sub-table rules of the sub-library and sub-table middleware for execution.

[0081] like Figure 5 As shown, for example, in another implementation of this embodiment, the data modeling service is an independent service that provides visual modeling capabilities and generates models and model field data. The access control service is an independent service that provides visual tenant and role permission configuration capabilities and generates tenant and role permission data. The interpretation and execution service is an independent service that provides the ability to dynamically adapt to data of different structures / formats. After the input data is verified by the model and model fields, tenants and role permissions, the SQL generated after the sub-library and sub-table information is attached is sent to the sub-library and sub-table middleware. When the sub-library and sub-table middleware completes execution and returns the data, the returned data is bound to the entity object corresponding to the ORM. The entity object contains a model identifier, and the returned data is verified according to the model and field data, tenant and role permission data and returned to the caller. The sub-library and sub-table middleware is an independent application, which is used to connect to different types of databases, provides dynamic increase and decrease of database connections and SQL execution routing capabilities, routes the received SQL to different types of databases for execution through the sub-library and sub-table rules, and realizes the function of adding other unconnected databases during operation.

[0082] Based on the above system provided by this embodiment, Figure 6 As shown, the implementation process of the system function is as follows:

[0083] S01: Receive data and bind the model. This step is used to receive input data and bind the input data to the model identifier through the model identifier contained in the data or the model identifier passed in the interface parameter.

[0084] S02: Obtaining the model and model field data: This step obtains the model and model field data according to the bound model identifier.

[0085] S03: Obtain the tenant and role permission data of the current user. This step obtains the tenant and the role permission data of the binding model and its fields of the user according to the current user ID.

[0086] S04: Filter model fields according to tenant and role permissions. This step filters the model fields according to tenant and role permission data, verifies the input data with the retained field data after filtering, and retains the input data that meets the requirements.

[0087] S05: Calculate the sub-library and sub-table data based on the model and model fields, tenant and role permission data.

[0088] S06: Merge the sub-library and sub-table data and business data to generate an entity object. This step assigns the value of the sub-library and sub-table data and the verified data to the object of the entity class. The entity object is bound to the model identifier and contains the sub-library and sub-table rule attributes and the extensible key-value type attributes. The attributes are used to store business data in different formats. All ORMs are bound by the entity object.

[0089] S07: Generate SQL through ORM of the same entity object. In this step, in the process of generating SQL through ORM, the sub-library and sub-table attributes and key-value attributes are expanded and flattened, the key and value hierarchy of the key-value attributes are kept consistent with the hierarchy of the sub-library and sub-table attributes, all keys and sub-library and sub-table attributes that conform to the model field definition and permissions are bound to the SQL fields, and all values ​​and sub-library and sub-table attribute values ​​are used as SQL parameters, thereby generating SQL.

[0090] S08: Send SQL to the sharding middleware for execution. This step sends the SQL generated by ORM to the sharding middleware for execution. The sharding middleware routes the SQL value to different databases for execution according to the sharding rules configured in advance. When the sharding middleware is deployed independently, it provides the ability to dynamically increase or decrease database connections, and realizes the function of accessing other unconnected databases during operation.

[0091] S09: Bind the returned data to the model and return it to the caller. After the SQL is executed, this step binds the returned data to the ORM entity object, and returns the fields that meet the requirements to the caller according to the model bound to the entity object.

[0092] In summary, this embodiment uses the same entity object for ORM, which can dynamically adapt data of different formats without generating code or editing. During system operation, data format modifications take effect in real time, reducing the time for compilation, packaging, and publishing. For methods with different data formats but the same logic, they can be adapted without development, which improves the efficiency of development and operation and maintenance. This embodiment uses ORM and sub-library and sub-table middleware to shield developers from the read and write operations of the data layer, so that developers do not need to worry about the adaptation of the data layer, and can operate the read and write of different types of databases at the same time, reducing the difficulty of development. The same business data in this embodiment provides different data storage methods for different tenants and users according to the differences in the business, so that the actual business library, table, and table fields are different due to different tenants and role permissions.

[0093] Embodiment 4:

[0094] Based on the dynamic data processing method and system based on sub-library and sub-table provided in the above-mentioned embodiments 1 to 3, the present invention also provides a dynamic data processing device based on sub-library and sub-table that can be used to implement the above-mentioned method and system. Figure 7 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The dynamic data processing device based on sub-library and sub-table of this embodiment includes one or more processors 21 and a memory 22. Figure 7 A processor 21 is taken as an example.

[0095] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0096] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the dynamic data processing method based on sub-library and sub-table and the system module in Examples 1 and 2. The processor 21 executes various functional applications and data processing of the dynamic data processing device based on sub-library and sub-table by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, realizes the dynamic data processing method based on sub-library and sub-table and the system function of Examples 1-3.

[0097] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the dynamic data processing methods and module functions based on sub-library and sub-table in the above-mentioned embodiments 1-3 are executed. For example, the above-described Figure 1 , Figure 6 The steps shown.

[0099] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.< / id>

Claims

1. A dynamic data processing method based on sub-library and sub-table, characterized in that: include: Receive input data and bind the input data to the corresponding model identifier, and obtain relevant data according to the bound model identifier and the current user identifier; The relevant data includes model and model field data and tenant and role permission data, and the obtaining of relevant data according to the bound model identifier and the current user identifier specifically includes: obtaining the model and model field data according to the bound model identifier; obtaining the tenant and role permission data of the current user according to the current user identifier; Verify the input data according to the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-library and sub-table data; specifically, filter the model field data according to the tenant and role permission data and verify the input data with the filtered model field data, retain the input data that meets the requirements after verification; calculate the sub-library and sub-table data according to the model and model field data and the tenant and role permission data; Assign the validated input data and the sub-library and sub-table data to the entity object, and perform ORM by the entity object to generate SQL; in the process of performing ORM by the entity object to generate SQL, expand and flatten the sub-library and sub-table attributes and the key-value attributes, keep the key and value levels of the key-value attributes consistent with the levels of the sub-library and sub-table attributes, bind all the sub-library and sub-table attributes and the keys of the key-value attributes to the fields of SQL, and use the values ​​of all the sub-library and sub-table attributes and the values ​​of the key-value attributes as the parameters of SQL; Execute the generated SQL, bind the returned data to the entity object corresponding to the ORM, verify the returned data according to the relevant data corresponding to the entity object, and then return it to the caller.

2. The dynamic data processing method based on sub-library and sub-table according to claim 1 is characterized in that: The entity object is bound to a corresponding model identifier and contains sub-library and sub-table attributes and extensible key-value attributes.

3. The dynamic data processing method based on sub-library and sub-table according to claim 2 is characterized in that: The verifying the returned data according to the relevant data corresponding to the entity object and returning it to the caller specifically includes: The corresponding related data is obtained according to the model identifier bound to the entity object, and the returned data is verified according to the model and model field data in the related data as well as the tenant and role permission data before being returned to the caller.

4. The dynamic data processing method based on sub-library and sub-table according to any one of claims 1 to 3, characterized in that: The SQL generated by executing specifically includes: The generated SQL is routed to different types of databases for execution according to the pre-configured sharding rules, and the data after execution in the database is returned.

5. A dynamic data processing system based on sub-library and sub-table, characterized in that: Used to implement the dynamic data processing method based on sub-library and sub-table as described in any one of claims 1 to 4, the system includes data modeling, access control, interpretation and execution services, and sub-library and sub-table middleware; The data modeling provides visual modeling capabilities, generates models and model field data, including relevant configuration data for database and table sharding rules; The access control provides visual tenant and role permission configuration capabilities, generates tenant and role permission data for filtering models and model field data, and tenant and part of role permission data for database and table sharding rules; The interpretation and execution service is used to obtain input data in different formats, bind the input data with the corresponding model identifier, and obtain relevant data according to the bound model identifier and the current user identifier; Verify the input data according to the relevant data, retain the input data that meets the requirements after verification, and calculate the sub-database and sub-table data; assign the verified input data and sub-database and sub-table data to the entity object, and use the entity object to perform ORM to generate SQL; when the sub-database and sub-table middleware returns the data after executing SQL, bind the returned data to the entity object corresponding to the ORM, verify the returned data according to the relevant data corresponding to the entity object, and return it to the caller; The library and table sharding middleware is used to obtain the generated SQL, route the SQL to different types of databases for execution according to the library and table sharding rules, and return the data in the database.

6. The dynamic data processing system based on sub-library and sub-table according to claim 5 is characterized in that: The data modeling, access control, interpretation and execution services, and sub-library and sub-table middleware provide corresponding functions by being separated into multiple microservices or merged into one microservice. When the sub-library and sub-table middleware is deployed independently, it connects to different types of databases and provides the function of dynamically increasing or decreasing database connections, thereby realizing access to other unconnected databases during operation.

7. A dynamic data processing device based on sub-library and sub-table, characterized in that: It includes at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the dynamic data processing method based on sharding of libraries and tables as described in any one of claims 1-6.

Images