Business data real-time pushing method and device under multi-tenant architecture

Abstract

The invention discloses a service data real-time pushing method and device under a multi-tenant architecture, and the method comprises the steps: carrying out the filtering through matching a Bloom filter according to the data type of service data and a unique tenant identifier when monitoring that a service platform generates the service data, and obtaining to-be-pushed data; pushing configuration information is determined according to the data type of the to-be-pushed data and the unique tenant identifier; performing field conversion on the to-be-pushed data to generate field data; and encrypting the field data according to the push configuration information and pushing the field data to the subscription user side. Through table storage, cache data structure design and configuration, a monitoring updating strategy is changed, so that the throughput and pushing efficiency of data pushing are improved, meanwhile, the data pushing configuration updating can take effect in time, and the pushing flexibility of service data is improved.

Description

Technical Field

[0001] The present invention relates to the field of data push technology, and in particular to a method and device for real-time push of business data under a multi-tenant architecture. Background Art

[0002] Data subscription push is a data distribution mechanism that is usually used in scenarios such as data synchronization, data migration, and maintaining data consistency between Internet IT systems. Data is pushed by responding to user subscriptions of data content that the user is interested in. If data synchronization fails or errors occur, it may lead to a series of serious problems such as data confusion and production system failure.

[0003] In a multi-tenant scenario, all business data belongs to a certain tenant. The business data of a certain tenant may need to be synchronized to multiple subscribers. A certain subscriber may also need to subscribe to multiple types of business data changes of multiple tenants. It is necessary to ensure that data is pushed to subscribers in real time and efficiently, and it also requires refined data permission control and push logic processing to ensure data isolation and security and avoid unauthorized access.

[0004] When dealing with large amounts of data synchronization push, traditional data synchronization push solutions cannot guarantee accurate business data push control and reduce push reliability because data subscription configuration changes cannot take effect in real time. Summary of the invention

[0005] The present invention provides a method and device for real-time push of business data under a multi-tenant architecture, which solves the technical problem that when traditional data synchronization push solutions deal with the situation of synchronous push of large amounts of data, the data subscription configuration changes cannot take effect in real time, resulting in the inability to ensure accurate business data push control and reduced push reliability.

[0006] The present invention provides a method for real-time push of business data under a multi-tenant architecture, which is applied to a data push terminal, and the method includes:

[0007] When monitoring the business data generated by the business platform, the data type of the business data and the unique tenant identifier are matched with the Bloom filter to filter and obtain the data to be pushed;

[0008] Determine push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0009] Performing field conversion on the data to be pushed to generate field data;

[0010] The field data is encrypted according to the push configuration information and pushed to the subscribing user end.

[0011] Optionally, when the business data generated by the business platform is monitored, filtering is performed according to the data type of the business data and the unique tenant identifier matching Bloom filter to obtain the data to be pushed, including:

[0012] When monitoring the business data generated by the business platform, matching the corresponding Bloom filter according to the data type of the business data;

[0013] If the match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set;

[0014] If it does not exist, stop pushing the business data;

[0015] If it exists or the Bloom filter fails to match, the business data is determined as data to be pushed.

[0016] Optionally, the determining the push configuration information according to the data type of the data to be pushed and the unique tenant identifier includes:

[0017] According to the data type and the unique tenant identifier of the data to be pushed, matching at least one corresponding cache key value from the preloaded first push configuration cache;

[0018] Using each of the cache key values ​​to match the corresponding push key-value pairs in the preloaded second push configuration cache;

[0019] If the match is successful, extracting the push configuration information associated with the cache key value from the push key value pair;

[0020] If the match fails, the pushing of the data to be pushed is stopped.

[0021] Optionally, performing field conversion on the data to be pushed to generate field data includes:

[0022] According to the data type of the data to be pushed, a corresponding field hash table is matched from the preloaded field conversion cache;

[0023] Determine one by one whether the storage field name of each of the data to be pushed exists in the field hash table;

[0024] If it exists, convert the storage field name of the data to be pushed into the external push field name according to the field hash table to obtain the intermediate data;

[0025] If it does not exist, stop pushing the field corresponding to the storage field name;

[0026] The intermediate data are formatted according to a preset data format to generate field data.

[0027] Optionally, the push configuration information includes a user agreement, a user key and at least one target address information; and encrypting the field data according to the push configuration information and pushing it to the subscribing user terminal includes:

[0028] Encrypting the field data using the user key to obtain encrypted data;

[0029] Push the encrypted data to the subscribing user terminals corresponding to the target address information according to the user agreement.

[0030] Optionally, the subscription client is used to:

[0031] When the encrypted data is received, decrypting the encrypted data to obtain decrypted data;

[0032] Determining whether the data identifier of the decrypted data is stored in a local database;

[0033] If so, performing data operations on the local data corresponding to the data identifier according to the decrypted data type and the data change type corresponding to the decrypted data;

[0034] If not, local data is added to the local database and associated with the data identifier.

[0035] Optionally, the data push end is also in communication connection with a subscription configuration center, and the method further includes:

[0036] The subscription configuration center receives the subscription information sent by any subscription client, and creates corresponding subscription configuration information in the built-in subscription configuration table;

[0037] When the data push end monitors that the business platform generates business data or receives a scheduled task trigger instruction, the subscription configuration table is traversed through the subscription configuration center, and the existing subscription configuration information is loaded into the first push configuration cache and the second push configuration cache;

[0038] When the subscription configuration center monitors configuration update information, the second push configuration cache is updated according to the configuration update information.

[0039] Optionally, the method further comprises:

[0040] When the subscription configuration center monitors the field mapping change information, the field conversion cache is updated according to the field mapping change information.

[0041] Optionally, the data push end is also in communication connection with a rights management and control center, and the method further includes:

[0042] The authority control center responds to the authority control information sent by the management terminal, and adjusts the service data type management table, the subscribed data type management table and the authorized tenant range management table corresponding to each of the subscribing user terminals;

[0043] When the subscribed data type management table or the authorized tenant range management table is adjusted, the Bloom filter and / or the first push configuration cache is updated according to the adjustment content by the authority management and control center.

[0044] The present invention provides a real-time push device for business data under a multi-tenant architecture, which is applied to a data push terminal, and the device includes:

[0045] A preliminary filtering module is used to filter the business data generated by the business platform according to the data type and unique tenant identifier of the business data to obtain the data to be pushed;

[0046] A secondary filtering module, used to determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0047] A field conversion module, used to perform field conversion on the data to be pushed to generate field data;

[0048] The centralized push module is used to encrypt the field data according to the push configuration information and push it to the subscribing user end.

[0049] It can be seen from the above technical solutions that the present invention has the following advantages:

[0050] When monitoring the business data generated by the business platform, filter it according to the data type and unique tenant identifier of the business data, and obtain the data to be pushed; determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed; perform field conversion on the data to be pushed to generate field data; encrypt the field data according to the push configuration information and push it to the subscribed user end. Through its table storage, cache data structure design and configuration change monitoring update strategy, the throughput and push efficiency of data push can be improved, and at the same time, when the subscription push configuration is updated, it can take effect in time, improving the push flexibility of business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1A flowchart of a method for real-time push of business data in a multi-tenant architecture provided by an embodiment of the present invention;

[0053] Figure 2 A module implementation flow chart of a method for real-time push of business data under a multi-tenant architecture provided by an embodiment of the present invention;

[0054] Figure 3 A structural block diagram of a device for real-time push of business data under a multi-tenant architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention provide a method and device for real-time push of business data under a multi-tenant architecture, which is used to solve the technical problem that when traditional data synchronization push solutions deal with the situation of large-scale data synchronization push, the data subscription configuration changes cannot take effect in real time, resulting in the inability to ensure accurate business data push control and reduced push reliability. Under the premise of monitoring real-time business data changes, through methods such as data subscription configuration, data filtering preprocessing and configuration data monitoring, it is possible to cope with the scenario of large-scale data synchronization push and ensure that subscription configuration changes take effect in real time. The data push range can be controlled to the field level, ensuring the accuracy and continuity of business data push, avoiding the problem of over-pushing or under-pushing data, improving push reliability, and solving data consistency problems between multiple systems.

[0056] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] See also Figure 1 , Figure 1 A flowchart of the steps of a method for real-time push of business data under a multi-tenant architecture provided by an embodiment of the present invention.

[0058] The present invention provides a method for real-time push of business data under a multi-tenant architecture, which is applied to a data push terminal, and the method includes:

[0059] Step 101, when monitoring the business data generated by the business platform, filtering is performed according to the data type of the business data and the unique tenant identifier matching Bloom filter to obtain the data to be pushed;

[0060] Tenants refer to different customer groups that use cloud computing platforms or SaaS software services. Although these customers share the infrastructure of the cloud computing platform or the core functions of the SaaS software, they are usually independent of each other in terms of data, configuration, permissions, etc. Among them, they are divided into multi-tenant systems and single-tenant systems. Multi-tenant systems refer to a system that can support multiple tenants. A tenant is usually a group of users with similar access modes and permissions. Typical tenants are several users from the same organization or company. Multi-tenant technology or multi-tenancy technology, referred to as SaaS, is a software architecture technology that aims to allow the same set of programs to be used in a multi-user environment and to ensure data isolation between users.

[0061] A Bloom filter is a probabilistic data structure used to quickly determine whether an element is in a set. Its basic principle is to use a bit array and multiple hash functions. In this embodiment, the set is a tenant identifier set, which includes multiple unique tenant identifiers to be pushed.

[0062] As a carrier of business activities, the business platform will continuously generate various business data such as static business data and dynamic business data during operation, and its data storage can be realized through the business data table in the relational database. In this embodiment, under the multi-tenant architecture based on the shared table, the business data is stored in the same table of the database, but a field with a unique tenant identifier is added to the table to distinguish the data ownership. The addition, deletion, and modification of the business data table can be obtained by the middleware listening to the binlog log, and the change record data is sent to the topic specified by the message queue. The topic can be distinguished according to the table name of the data table or the partition can be determined according to the primary key ID. The business data table structure is shown in Table 1 below:

[0063]

[0064] In the embodiment of the present application, the data push end is connected to the business platform in communication, and the business data generated by the business platform is monitored and parsed in real time through the middleware. For static business data, it can be obtained in real time by monitoring the binlog log in the relational database, and for dynamic business data, it can be obtained in real time by monitoring the changes in the business data message queue, so as to monitor the change operations such as adding, updating or deleting business data in real time, and each business data is associated with a unique tenant identifier.

[0065] When the data push end monitors the business platform to generate business data or receives a scheduled task trigger instruction, it obtains the data type and unique tenant identifier corresponding to the business data to match the corresponding Bloom filter to filter it and obtain the data to be pushed.

[0066] In an example of the present application, step 101 may include the following sub-steps:

[0067] When monitoring the business data generated by the business platform, the corresponding Bloom filter is matched according to the data type of the business data;

[0068] If the match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set;

[0069] If it does not exist, stop pushing business data;

[0070] If it exists or the Bloom filter fails to match, the business data is determined as the data to be pushed.

[0071] The tenant identifier set refers to a set consisting of multiple unique tenant identifiers that subscribe to business data and is stored in the Bloom filter.

[0072] In this embodiment, when monitoring the business platform to generate new business data, the data push end obtains the data type and unique tenant identifier corresponding to the business data, matches the Bloom filter according to the data type, and if the match is successful, further determines whether the unique tenant identifier exists in its built-in tenant identifier set through the Bloom filter. If the match fails, it means that there is no need to filter the business data at present, or the data type of the business data has not been recorded. At this time, the business data can be determined as data to be pushed and directly enter the next processing flow.

[0073] After successfully matching the Bloom filter, if the unique tenant identifier does not exist in the built-in tenant identifier set, it means that the business data does not need to be pushed externally. At this time, the business data can be deleted or the push process can be ended directly to stop pushing business data. If the unique tenant identifier exists in the built-in tenant identifier set, it means that the business data needs to be pushed, and it is determined as the data to be pushed and enters the next processing flow. The Bloom filter mechanism can remove data that does not need to be pushed externally, reducing the amount of useless data.

[0074] The tenant identification set may be designed with a cache data structure to store all subscriber sets that have subscribed to a certain business data type, and obtain all authorized tenant identification sets according to the subscriber set. The same applies to other business data types. The specific storage cache data structure design may be shown in Table 2 below:

[0075]

[0076] Optionally, before executing step 101, in order to facilitate subsequent field conversion and matching of subscription configuration information, the following steps S11-S13 can be executed by the subscription configuration center to create subscription configuration information and load it into the first push configuration cache and the second push configuration cache, and the permission management center can execute steps S21-S22 to configure the information to create permission-related tables and load them into the Bloom filter, the first push configuration cache or the second push configuration cache of the data push end.

[0077] In an example of the present application, the data push end is also connected to the subscription configuration center for communication, and the method further includes the following steps S11-S13:

[0078] S11. Receive subscription information sent by any subscription client through the subscription configuration center, and create corresponding subscription configuration information in the built-in subscription configuration table;

[0079] In this embodiment, in order to facilitate users to subscribe to business data, a graphical user interface (GUI) for user operation can be provided through the subscription configuration center for users to perform subscription configuration. Subscribers can specify the target address information, key, user agreement and other configuration information for data reception through the graphical user interface, and can also choose the type of business data to be subscribed, and can apply to subscribe to multiple business data types under any tenant.

[0080] Specifically, the subscription configuration center receives the subscription information sent by any subscription user end, parses it, and obtains the target address information, user agreement, key and other information to create the corresponding subscription configuration information. For the convenience of storage, the subscription configuration information can be stored in the form of a subscription configuration table. The specific table structure is shown in Table 3 below:

[0081]

[0082] The subscriber ID is the ID of the subscriber, and the tenant ID is the tenant identifier.

[0083] The above subscription configuration table can be used to store information related to the subscriber's receiving address, support one subscriber to configure multiple receiving addresses or protocol types, and implement one-to-many push for each subscriber; the table structure can also add more fields to expand the subscription function, such as limiting the time period that can be pushed, specifying the number of retries, etc.; the protocol field is set to an enumeration value, and the value range is the type of protocol supported by the publisher, such as Internet HTTPS protocol call, intranet HTTPS protocol call, etc. Through different protocol types, more subscribers are supported to subscribe to the required data more flexibly.

[0084] S12: When the data push end monitors the business platform to generate business data or receives a scheduled task trigger instruction, it traverses the subscription configuration table through the subscription configuration center and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache;

[0085] In this embodiment, when the data push end monitors the business platform to generate business data or receives a scheduled task trigger instruction, that is, when the data push service is started, the subscription configuration center traverses the subscription configuration table and loads the existing and valid subscription configuration information into the first push configuration cache and the second push configuration cache as push configuration information to preheat the relevant cache data.

[0086] S13: When the subscription configuration center monitors the configuration update information, the second push configuration cache is updated according to the configuration update information.

[0087] Configuration update information refers to the change log in the configuration repository that stores subscription configuration information.

[0088] In this embodiment, since some subscribed users may need to add subscriptions during the operation of the data push service, the configuration update information can be monitored in real time through the subscription configuration center. By monitoring the binlog change log of the configuration library, the addition, deletion, and change operations of the subscription configuration information can be obtained in real time, and the cache-related data in the second push configuration cache can be updated in real time, thereby updating the push configuration and ensuring that the configuration takes effect immediately.

[0089] In an example of the present application, the subscription configuration center is further used to perform the following steps:

[0090] When the subscription configuration center listens to field mapping change information, it updates the field conversion cache according to the field mapping change information.

[0091] Field mapping change information includes the new storage field name and externally pushed field name. This configuration can be used to modify the externally pushed field information to ensure the consistency of the external display of the field.

[0092] In an embodiment of the present application, the subscription configuration center monitors the binlog log of the business data field mapping table in real time, and when the input field mapping change information is monitored, the field conversion cache is updated according to the field mapping change information.

[0093] In this embodiment, the subscription configuration center may also store a business data field mapping table, which is preloaded into the field conversion cache when the data push end is started. The business data field mapping table is shown in Table 4 below:

[0094]

[0095] The above business data field mapping table is traversed through the subscription configuration center, and the corresponding data type id, storage field name, external push field name and other data are loaded into the cache of the data push end as the field conversion cache, which can be stored in a hash structure. The cache data structure design is shown in Table 5 below:

[0096]

[0097] In an example of the present application, the data push end is also connected to the authority management center for communication, and the method further includes S21-S22:

[0098] S21, the authority control center responds to the authority control information sent by the management terminal, and adjusts the business data type management table, the subscribed data type management table and the authorized tenant range management table corresponding to each subscribing user terminal;

[0099] The authority control information refers to information instructions used to adjust the business data or the subscription authority of the subscribing user end, and the subscription authority may include but is not limited to the business data type management table, the subscribed data type management table and the authorized tenant range management table.

[0100] In this embodiment, the authority control center can receive the authority control information through the graphical user interface for management personnel to operate, or receive the authority control information sent by the management terminal connected to it, parse the authority control information to obtain the corresponding control instructions, and adjust the authority of the business data type management table, subscribed data type management table and authorized tenant range management table corresponding to each subscribing user terminal. Specifically, the administrator can choose whether to open the subscription rights for the subscriber and which business data to open, and can also choose to authorize the data range for the subscriber, which can be authorized by single tenant, region, organization account and other dimensions; the subscribing user terminal cannot receive unauthorized tenant business data or unauthorized business data types.

[0101] Specifically, for the service data type management table of the subscription user side, the service data type table can be adjusted through the authority control information, and the specific table structure can be shown in the following Table 6:

[0102]

[0103] In this embodiment, the service data type management table of the subscriber client is adjusted through the above service data type table. The table manages all service data types and data types that can be subscribed to by the subscriber client. The subscriber client can only subscribe to data types that are allowed to be subscribed. The subscription allowed field is set to an enumeration value, which can be set to 1, indicating that subscription is allowed, and set to 0, indicating that subscription is not allowed.

[0104] For the subscribed data type management table of the subscribing user side, after the subscribing user side submits the subscription application in the subscription configuration center, the administrator can review the subscription application through the authority management center, and manage the business data type and application review status of the subscription application applied by the subscribing user side. Specifically, it can be implemented by adjusting the subscribed data type management table through the authority management information, and the table structure is shown in Table 7 below:

[0105]

[0106] In this embodiment, a subscribing client can apply to subscribe to multiple business data types, and one business data type can be subscribed by multiple subscribing client terminals; the subscription review status field is set to an enumeration value, and the value range can be pending review, reviewed, failed review, etc., corresponding to 0, 1, 2 or other values ​​respectively. Only subscription applications that pass the review will push the corresponding business data type to the corresponding subscribing client terminal.

[0107] For the authorized tenant range management table of each subscription user end, the authorized tenant range management table can be adjusted through the following permission control information, and its table structure is shown in Table 8 below:

[0108]

[0109] In this embodiment, the above-mentioned authorized tenant range management table is used to manage the data scope authorized by the subscription user end, wherein the authorization type is set to an enumeration value, and the value range can be authorization by single tenant, authorization by region, authorization by organizational account, etc., corresponding to 0, 1, 2 or other values ​​respectively. Authorization can be performed in three dimensions at the same time for the same subscriber, thereby improving the flexibility of authorization; changes in tenant business data that are not authorized to the subscriber will not be pushed to the subscriber.

[0110] S22: When the subscribed data type management table or the authorized tenant range management table is adjusted, the authority management and control center updates the Bloom filter and / or the first push configuration cache according to the adjusted content.

[0111] In an embodiment of the present application, when the subscribed data type management table or the authorized tenant range management table is adjusted, that is, the binlog logs of the subscribed data type management table and the authorized tenant range management table are monitored. When data changes are monitored, the authority management center synchronously updates the tenant identification set in the Bloom filter to update the data type of the business data in real time.

[0112] In addition, since the subscribed data type management table or the authorized tenant range management table also involves contents such as the subscriber ID, the data type of business data and the subscription status, when the above tables are updated, the Bloom filter and / or the first push configuration cache are updated according to the adjusted contents.

[0113] For example, if only the business data type identifier is changed, that is, the data type is changed, the data type in the Bloom filter is changed. If there are changes in the subscriber ID, authorized tenant ID, etc., the first push configuration cache is updated.

[0114] Step 102, determining push configuration information according to the data type of the data to be pushed and the unique tenant identifier;

[0115] In this embodiment, after the business data is initially filtered using a Bloom filter and the data to be pushed is screened, the data type of the data to be pushed and the cache key value corresponding to the unique tenant identifier are obtained, and the push configuration cache that has been preheated and loaded in advance is further retrieved to determine whether the cache key value exists in the push configuration cache; if the cache key value does not exist in the push configuration cache, the push process of the data to be pushed is terminated; if it exists, the subscriber ID set to be pushed is obtained, thereby obtaining the target interface address, key, protocol and other information to be pushed.

[0116] In an example of the present application, step 102 may include the following sub-steps:

[0117] According to the data type and the unique tenant identifier of the data to be pushed, matching at least one corresponding cache key value from the preloaded first push configuration cache;

[0118] Use each cache key value to match the corresponding push key-value pair in the preloaded second push configuration cache;

[0119] If the match is successful, the push configuration information associated with the cache key value is extracted from the push key value pair;

[0120] If the match fails, the push of the data to be pushed will be stopped.

[0121] It should be noted that the existing and valid subscription configuration information can be loaded into the first push configuration cache and the second push configuration cache of the data push end by traversing the subscription configuration table. The hash structure can be used for storage. The first push configuration cache is shown in the following cache data structure design table 9:

[0122]

[0123] Among them, Table 9 can obtain the set of all business data types subscribed by a certain subscriber according to a subscriber ID, and then obtain the set of all tenant IDs authorized to the subscriber according to the subscriber ID, and load the push configuration information into the cache in turn, which can be stored in the set collection.

[0124] The second push configuration cache is shown in the cache data structure design table 10:

[0125]

[0126] In this embodiment, according to the data type identifier and unique tenant identifier of the data to be pushed, table 9 is retrieved to match all subscriber IDs as cache key values, and then table 10 is retrieved according to the cache key value to determine the subscription configuration information corresponding to each cache key value, that is, the configuration information entity class set in Table 10 (including receiving address, key, protocol, etc.).

[0127] In addition, you can also monitor the binlog logs of the subscription configuration table in real time. When data changes are monitored, update the cached data in Table 10, the binlog logs of the subscribed data type management table and the authorized tenant range management table. When data changes are monitored, update the cached data in Table 9.

[0128] Step 103, performing field conversion on the data to be pushed to generate field data;

[0129] In this embodiment, after determining the data to be pushed, field conversion is performed on each data to be pushed, so as to perform field and data format conversion processing according to the field conversion mapping configuration pre-loaded in advance, generate field data, and complete data preparation before pushing.

[0130] In an example of the present application, step 103 may include the following sub-steps:

[0131] According to the data type of the data to be pushed, the preloaded field conversion cache matches the corresponding field hash table;

[0132] Determine one by one whether the storage field name of each data to be pushed exists in the field hash table;

[0133] If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain the intermediate data;

[0134] If it does not exist, stop pushing the field corresponding to the storage field name;

[0135] The intermediate data are formatted according to the preset data format to generate field data.

[0136] Please refer to Table 5, which is a field conversion cache in an embodiment of the present invention. After the field conversion cache is preloaded to the data push end, the corresponding field hash table is matched from the preloaded field conversion cache according to the data type of the data to be pushed.

[0137] Then, by traversing the field hash table, determine one by one whether the storage field name of each data to be pushed exists in the field hash table. If so, it means that the data to be pushed can be pushed, and the storage field name of the data to be pushed can be converted into the externally pushed field name according to the field hash table to obtain the intermediate data. If it does not exist, it means that the data to be pushed is not configured with a field conversion mapping, that is, it is not recorded in the field hash table, and the data to be pushed is retained or fed back to the user for further processing. After completing the field conversion of all data to be pushed, the format of each intermediate data is converted according to the preset data format to obtain field data in a data format that meets the needs of each subscribing user end. Through this configuration, the field data pushed externally can be modified to ensure the consistency of the field display to the outside.

[0138] Step 104: Encrypt the field data according to the push configuration information and push it to the subscribing user end.

[0139] In this embodiment, after the mapping conversion of the field data is completed, the finally obtained field data is encrypted according to the push configuration information, and different specifications are called according to the protocol configuration to realize the final push of the data.

[0140] Among them, encryption can use AES symmetric encryption algorithm, and use thread pool technology to achieve high concurrent push. If data push fails, such as encountering network abnormalities, it can be retried according to the configured number of retries, and no retries will be made after exceeding the number of retries. Due to the stateless characteristics of the module executed by the push, the module can be quickly expanded horizontally, thereby ensuring the performance of the centralized push module.

[0141] In an example of the present application, the subscription configuration information includes a user agreement, a user key, and at least one target address information; step 104 may include the following sub-steps:

[0142] The field data is encrypted using the user key to obtain encrypted data;

[0143] Push encrypted data to the subscribed user end corresponding to each target address information according to the user agreement.

[0144] Among them, the subscription user end is used to:

[0145] When the encrypted data is received, the encrypted data is decrypted to obtain the decrypted data;

[0146] Determine whether the data identifier of the decrypted data is stored in the local database;

[0147] If so, perform data operations on the local data corresponding to the data identifier according to the decrypted data type and the data change type corresponding to the decrypted data;

[0148] If not, then local data is added to the local database and associated with the data identifier.

[0149] In this embodiment, after the subscriber receives the encrypted data sent by the data push end, the encrypted data is decrypted to obtain the decrypted data, and then the decrypted data is stored in the database according to the data type and change type. It is determined whether the data identifier of the decrypted data has been stored in the local database. If so, the local data corresponding to the data identifier is subjected to corresponding data operations according to the decrypted data type of the decrypted data and the specific data change type such as addition, deletion, modification, etc.

[0150] If the data identifier is not in the local database, local data is added in the local database using the decrypted data and associated with the data identifier.

[0151] like Figure 2 As shown, Figure 2 The present invention is a flowchart of a method for real-time push of business data under a multi-tenant architecture in an embodiment of the present invention.

[0152] The embodiments of the present application involve a data push terminal, a subscription user terminal, a business platform, a subscription configuration center, a permission management center, and a cache loading center. Before real-time push of business data, the subscription user terminal configures the subscription configuration table and the business data field mapping table in the configuration database through the graphical user interface of the subscription configuration center, and loads them into the cache in the cache loading center. The management terminal configures the business data type management table, the subscribed data type management table, and the authorized tenant range management table in the permission control database through the graphical user interface of the permission management center, so as to configure the business data type, the subscribable data range, and the audit status applied for by the subscription user terminal, and load them into the cache in the cache loading center in real time.

[0153] The cache loading center is connected to the data push end in communication, and is used as a real-time cache of the data push end. When the data push end is started, the cached key-value pairs are loaded into the Bloom filter, the first push configuration cache, and / or the second push configuration cache. At the same time, the cache loading center can also monitor the data changes of the configuration database and the authority control database in real time. If there is a data change in any database, the Bloom filter, the first push configuration cache, and / or the second push configuration cache are updated in real time.

[0154] After completing the cache configuration of the data push end, the data push end monitors the static business data of the business database of the business platform through the log binlog, and monitors the dynamic business data of the business data message queue through the topic, to determine whether to generate the business data required by the subscribing user end. When the generated business data is monitored, the built-in primary filtering module selects the Bloom filter to perform preliminary filtering on the business data to filter out the data to be pushed and transmit it to the secondary filtering module. The secondary filtering module performs secondary filtering on the push configuration information of the data to be pushed. At the same time as the secondary filtering is completed, the field conversion module performs field conversion on the data to be pushed, generates field data and sends it to the centralized push module together with the push configuration information determined by the cache. Finally, the centralized push module encrypts the field data according to the push configuration information and sends it to the subscribing user end.

[0155] Among them, the data that does not meet the filtering requirements screened in the primary filtering module or the secondary filtering module directly ends the data push process.

[0156] In an embodiment of the present application, the data push end stores the subscriber's subscription receiving address, key, protocol, etc.; manages and maintains the business data type that can be opened for subscription, and stores the business data type subscribed by the subscriber; stores the subscriber's data authorization dimension and authorization scope information; according to the business data type, preloads the tenant ID set that has been authorized to all subscribers and stores it in the corresponding Bloom filter; loads the subscription configuration into the cache (subscriber receiving address, push protocol, encryption key, data push scope and tenant ID set, business data table field mapping, etc.), and monitors the table record changes in the configuration library, and refreshes the cache in real time; monitors business data changes, and uses the Bloom filter to filter and pre-process the data according to the tenant ID to which the business data belongs, so as to reduce the amount of data; finally determines whether the data is pushed and the target address information (the target address can be multiple) according to the business data type and the tenant ID to which it belongs, and completes the conversion of fields and data formats; aggregates the filtered and converted data to be pushed and the destination information to the unified push processor module; the unified push processor finally pushes the business data changes to the subscriber receiving address according to different protocols.

[0157] As for the subscription user side, its functional responsibilities include providing and configuring the subscription receiving address, obtaining encryption and decryption keys, configuring protocols, etc.; selecting the type of business data that needs to be subscribed and applying for corresponding permissions; applying for the subscription data scope (by single tenant dimension, regional dimension, account dimension); upon receiving the data push, decrypt it and store and dispose of the local data. The overall process of incremental data subscription push and the functional design of each module, especially the design and implementation of the subscription configuration module, permission management module, configuration loading and monitoring update module, and data filtering and processing module. The advantages are that each module has a single function, isolated responsibilities, loose coupling between modules, and easy expansion.

[0158] In a specific implementation, the above embodiment can achieve the functional and performance requirements of data synchronization through the following distributed deployment scheme to ensure high availability of the service, as shown in Table 11:

[0159]

[0160] The middleware required are:

[0161] LVS / NG load balancing (requires 20M bandwidth) provides reverse proxy, load balancing and failover capabilities. Cache (Redis) is used to implement cache-based Bloom filters and store subscription push configuration data. Relational database (MySQL) is used to store business data, push configuration and authorization data. Message queue (Kafka) is used for decoupling, achieving high throughput, low latency, and improving system concurrency.

[0162] In the embodiment of the present application, when the business platform generates business data, the data type of the business data and the unique tenant identifier are matched with the Bloom filter to filter and obtain the data to be pushed; according to the data type of the data to be pushed and the unique tenant identifier, the push configuration information is determined; the field of the data to be pushed is converted to generate field data; the field data is encrypted according to the push configuration information and pushed to the subscribed user end. Through its table storage, cache data structure design and configuration change monitoring update strategy, the throughput and push efficiency of data push can be improved, and at the same time, when the subscription push configuration is updated, it can take effect in time, thereby improving the push flexibility of business data.

[0163] See also Figure 3 , Figure 3 A structural block diagram of a device for real-time push of business data under a multi-tenant architecture in an embodiment of the present application is shown.

[0164] The present invention provides a real-time push device for business data under a multi-tenant architecture, which is applied to a data push terminal, and the device includes:

[0165] The preliminary filtering module 301 is used to filter the business data generated by the business platform according to the data type and unique tenant identifier of the business data by matching the Bloom filter to obtain the data to be pushed;

[0166] Secondary filtering module 302, used to determine push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0167] A field conversion module 303 is used to perform field conversion on the data to be pushed and generate field data;

[0168] The centralized push module 304 is used to encrypt the field data according to the push configuration information and push it to the subscribing user end.

[0169] Optionally, the preliminary filtering module 301 is specifically used for:

[0170] When monitoring the business data generated by the business platform, the corresponding Bloom filter is matched according to the data type of the business data;

[0171] If the match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set;

[0172] If it does not exist, stop pushing business data;

[0173] If it exists or the Bloom filter fails to match, the business data is determined as the data to be pushed.

[0174] Optionally, the secondary filtering module 302 is specifically used for:

[0175] According to the data type and the unique tenant identifier of the data to be pushed, matching at least one corresponding cache key value from the preloaded first push configuration cache;

[0176] Use each cache key value to match the corresponding push key-value pair in the preloaded second push configuration cache;

[0177] If the match is successful, the push configuration information associated with the cache key value is extracted from the push key value pair;

[0178] If the match fails, the push of the data to be pushed will be stopped.

[0179] Optionally, the field conversion module 303 is specifically used for:

[0180] According to the data type of the data to be pushed, the preloaded field conversion cache matches the corresponding field hash table;

[0181] Determine one by one whether the storage field name of each data to be pushed exists in the field hash table;

[0182] If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain the intermediate data;

[0183] If it does not exist, stop pushing the field corresponding to the storage field name;

[0184] The intermediate data are formatted according to the preset data format to generate field data.

[0185] Optionally, the push configuration information includes a user agreement, a user key and at least one target address information; the centralized push module 304 is specifically used for:

[0186] The field data is encrypted using the user key to obtain encrypted data;

[0187] Push encrypted data to the subscribed user end corresponding to each target address information according to the user agreement.

[0188] Optionally, the subscriber client is used to:

[0189] When the encrypted data is received, the encrypted data is decrypted to obtain the decrypted data;

[0190] Determine whether the data identifier of the decrypted data is stored in the local database;

[0191] If so, perform data operations on the local data corresponding to the data identifier according to the decrypted data type and the data change type corresponding to the decrypted data;

[0192] If not, then local data is added to the local database and associated with the data identifier.

[0193] Optionally, the data push end is also connected to the subscription configuration center for communication.

[0194] Receive subscription information sent by any subscription client, and create corresponding subscription configuration information in the built-in subscription configuration table;

[0195] When the data push end monitors the business platform to generate business data or receives a scheduled task trigger instruction, it traverses the subscription configuration table and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache;

[0196] When the subscription configuration center monitors the configuration update information, the second push configuration cache is updated according to the configuration update information.

[0197] Optionally, the subscription configuration center is also used to:

[0198] When field mapping change information is monitored, the field conversion cache is updated according to the field mapping change information.

[0199] Optionally, the data push end is also connected to a permission control center for communication. The permission control center is specifically used for:

[0200] By responding to the permission control information sent by the management end, the business data type management table, subscribed data type management table and authorized tenant range management table corresponding to each subscribing user end are adjusted;

[0201] When the subscribed data type management table or the authorized tenant range management table is adjusted, the Bloom filter and / or the first push configuration cache is updated according to the adjusted content.

[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0203] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0204] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0206] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0207] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time push of business data under a multi-tenant architecture, characterized in that: Applied to the data push end, the method includes: When monitoring the business data generated by the business platform, the data type of the business data and the unique tenant identifier are matched with the Bloom filter to filter and obtain the data to be pushed; Determine push configuration information according to the data type and unique tenant identifier of the data to be pushed; Performing field conversion on the data to be pushed to generate field data; The field data is encrypted according to the push configuration information and pushed to the subscribing user end.

2. The method according to claim 1, characterized in that When the business data generated by the business platform is monitored, the data type of the business data and the unique tenant identifier are matched with the Bloom filter to filter and obtain the data to be pushed, including: When monitoring the business data generated by the business platform, matching the corresponding Bloom filter according to the data type of the business data; If the match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set; If it does not exist, stop pushing the business data; If it exists or the Bloom filter fails to match, the business data is determined as data to be pushed.

3. The method according to claim 1, characterized in that Determining the push configuration information according to the data type of the data to be pushed and the unique tenant identifier includes: According to the data type and the unique tenant identifier of the data to be pushed, matching at least one corresponding cache key value from the preloaded first push configuration cache; Using each of the cache key values ​​to match the corresponding push key-value pairs in the preloaded second push configuration cache; If the match is successful, extracting the push configuration information associated with the cache key value from the push key value pair; If the match fails, the pushing of the data to be pushed is stopped.

4. The method according to claim 3, characterized in that The step of performing field conversion on the data to be pushed to generate field data includes: According to the data type of the data to be pushed, a corresponding field hash table is matched from the preloaded field conversion cache; Determine one by one whether the storage field name of each of the data to be pushed exists in the field hash table; If it exists, convert the storage field name of the data to be pushed into the external push field name according to the field hash table to obtain the intermediate data; If it does not exist, stop pushing the field corresponding to the storage field name; The intermediate data are formatted according to a preset data format to generate field data.

5. The method according to claim 1, characterized in that The push configuration information includes a user agreement, a user key and at least one target address information; the encrypting the field data according to the push configuration information and pushing it to the subscribing user end includes: Encrypting the field data using the user key to obtain encrypted data; Push the encrypted data to the subscribing user terminals corresponding to the target address information according to the user agreement.

6. The method according to claim 5, characterized in that The subscription client is used to: When the encrypted data is received, decrypting the encrypted data to obtain decrypted data; Determining whether the data identifier of the decrypted data is stored in a local database; If so, performing data operations on the local data corresponding to the data identifier according to the decrypted data type and the data change type corresponding to the decrypted data; If not, local data is added to the local database and associated with the data identifier.

7. The method according to claim 4, characterized in that The data push end is also in communication connection with the subscription configuration center, and the method further includes: The subscription configuration center receives the subscription information sent by any subscription client, and creates corresponding subscription configuration information in the built-in subscription configuration table; When the data push end monitors that the business platform generates business data or receives a scheduled task trigger instruction, the subscription configuration table is traversed through the subscription configuration center, and the existing subscription configuration information is loaded into the first push configuration cache and the second push configuration cache; When the subscription configuration center monitors configuration update information, the second push configuration cache is updated according to the configuration update information.

8. The method according to claim 7, characterized in that The method further comprises: When the subscription configuration center monitors the field mapping change information, the field conversion cache is updated according to the field mapping change information.

9. The method according to claim 1, characterized in that: The data push end is also in communication connection with the authority control center, and the method further includes: The authority control center responds to the authority control information sent by the management terminal, and adjusts the service data type management table, the subscribed data type management table and the authorized tenant range management table corresponding to each of the subscribing user terminals; When the subscribed data type management table or the authorized tenant range management table is adjusted, the Bloom filter and / or the first push configuration cache is updated according to the adjustment content by the authority management and control center.

10. A real-time push device for business data under a multi-tenant architecture, characterized in that: Applied to the data push end, the device includes: A preliminary filtering module is used to filter the business data generated by the business platform according to the data type and unique tenant identifier of the business data to obtain the data to be pushed; A secondary filtering module, used to determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed; A field conversion module, used to perform field conversion on the data to be pushed to generate field data; The centralized push module is used to encrypt the field data according to the push configuration information and push it to the subscribing user end.

Images