A business data real-time pushing method and device under a multi-tenant architecture

Abstract

The application discloses a service data real-time pushing method and device under a multi-tenant architecture, and the method comprises the following steps: when service data generated by a service platform is monitored, the service data is filtered according to the data type and unique tenant identifier of the service data, and filtered data is obtained; the pushing configuration information is determined according to the data type and unique tenant identifier of the filtered data; the field conversion is performed on the filtered data, and field data is generated; and the field data is encrypted according to the pushing configuration information and pushed to a subscription user end. Through the table storage, cache data structure design and configuration change listening update strategy, the throughput and pushing efficiency of the data pushing are improved, and the pushing flexibility of the service data is improved when the subscription pushing configuration is updated.

Description

Technical Field

[0001] This invention relates to the field of data push technology, and in particular to a method and apparatus for real-time push of business data under a multi-tenant architecture. Background Technology

[0002] Data subscription push, as a data distribution mechanism, is typically used in scenarios such as data synchronization, data migration, and maintaining data consistency between internet IT systems. It pushes data based on user subscriptions and the data content the user is interested in. However, if data synchronization fails or errors occur, it can lead to a series of serious problems such as data corruption and production system failures.

[0003] In a multi-tenant scenario, all business data belongs to a single tenant. A tenant's business data may need to be synchronized to multiple subscribers, and a subscriber may also need to subscribe to multiple types of business data changes from multiple tenants. It is necessary to ensure that data is pushed to subscribers in real time and efficiently, as well as to have granular data access control and push logic processing to ensure data isolation and security and avoid unauthorized access.

[0004] Traditional data synchronization and push solutions suffer from reduced reliability when dealing with large volumes of data being pushed simultaneously because changes to data subscription configurations cannot take effect in real time. This results in a lack of accurate control over business data pushes. Summary of the Invention

[0005] This invention provides a method and apparatus for real-time push of business data under a multi-tenant architecture, which solves the technical problem that traditional data synchronization push schemes cannot guarantee accurate business data push control and reduce push reliability when dealing with large amounts of data synchronization push because changes in data subscription configuration cannot take effect in real time.

[0006] This invention provides a method for real-time push of business data under a multi-tenant architecture, applied to a data push terminal. The method includes:

[0007] When business data is detected from the business platform, it is filtered using a Bloom filter based on the data type and unique tenant identifier of the business data to obtain the data to be pushed.

[0008] Determine the push configuration information based on the data type and unique tenant identifier of the data to be pushed;

[0009] The data to be pushed is converted into fields to generate field data;

[0010] The field data is encrypted according to the push configuration information and pushed to the subscriber's terminal.

[0011] Optionally, when business data is detected generated by the business platform, the data to be pushed is obtained by filtering it using a Bloom filter based on the data type and unique tenant identifier of the business data, including:

[0012] When business data is detected from the business platform, a corresponding Bloom filter is matched according to the data type of the business data.

[0013] If a match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set;

[0014] If it does not exist, then stop pushing the aforementioned business data;

[0015] If the Bloom filter exists or fails to match, the business data is determined to be data to be pushed.

[0016] Optionally, determining the push configuration information according to the data type and unique tenant identifier of the data to be pushed includes:

[0017] Based on the data type and unique tenant identifier of the data to be pushed, match at least one corresponding cache key-value from the pre-loaded first push configuration cache;

[0018] Each of the aforementioned cache key-value pairs is used to match the corresponding push key-value pairs in the preloaded second push configuration cache;

[0019] If a match is successful, the push configuration information associated with the cache key-value pair is extracted from the push key-value pair;

[0020] If a match fails, the push of the data to be pushed will be stopped.

[0021] Optionally, the step of performing field transformation on the data to be pushed to generate field data includes:

[0022] According to the data type of the data to be pushed, the preloaded fields are converted and cached to match the corresponding field hash table;

[0023] Determine whether the storage field name of each of the data to be pushed exists in the field hash table;

[0024] If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain intermediate data;

[0025] If it does not exist, then stop pushing the field corresponding to the stored field name;

[0026] The intermediate data are converted according to a preset data format to generate field data.

[0027] Optionally, the push configuration information includes a user agreement, a user key, and at least one target address; the step of encrypting the field data according to the push configuration information and pushing it to the subscribing user includes:

[0028] The field data is encrypted using the user key to obtain encrypted data;

[0029] The encrypted data is pushed to the subscriber client corresponding to each of the target address information in accordance with the user agreement.

[0030] Optionally, the subscription client is used for:

[0031] When the encrypted data is received, it is decrypted to obtain the decrypted data;

[0032] Determine whether the data identifier of the decrypted data is stored in a local database;

[0033] If so, then perform data operations on the local data corresponding to the data identifier according to the decryption data type and data change type corresponding to the decryption data;

[0034] If not, add local data to the local database and associate it with the data identifier.

[0035] Optionally, the data push terminal also communicates with the subscription configuration center, and the method further includes:

[0036] The subscription configuration center receives subscription information sent by any subscriber client and creates corresponding subscription configuration information in the built-in subscription configuration table.

[0037] When the data push terminal detects that the business platform generates business data or receives a scheduled task trigger instruction, it traverses the subscription configuration table through the subscription configuration center and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache.

[0038] When the subscription configuration center detects configuration update information, it updates the second push configuration cache according to the configuration update information.

[0039] Optionally, the method further includes:

[0040] When the subscription configuration center detects a field mapping change, it updates the field conversion cache according to the field mapping change information.

[0041] Optionally, the data push terminal is also connected to the access control center, and the method further includes:

[0042] The permission control center responds to the permission control information sent by the management terminal and adjusts the business data type management table, subscribed data type management table and authorized tenant scope management table corresponding to each of the subscribed user terminals.

[0043] When the subscribed data type management table or the authorized tenant scope management table is adjusted, the Bloom filter and / or the first push configuration cache are updated through the permission control center according to the adjustment content.

[0044] This invention provides a real-time business data push device under a multi-tenant architecture, applied to a data push terminal, the device comprising:

[0045] The preliminary filtering module is used to filter business data generated by the business platform according to the data type and unique tenant identifier of the business data by matching a Bloom filter to obtain the data to be pushed.

[0046] The secondary filtering module is used to determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0047] The field conversion module is used to convert the fields of the data to be pushed and generate field data.

[0048] The centralized push module is used to encrypt the field data according to the push configuration information and push it to the subscriber's terminal.

[0049] As can be seen from the above technical solutions, the present invention has the following advantages:

[0050] When business data is detected from the business platform, it is filtered using a Bloom filter based on the data type and unique tenant identifier to obtain the data to be pushed. The push configuration information is then determined based on the data type and unique tenant identifier. Field transformations are performed on the data to be pushed to generate field data. Finally, the field data is encrypted according to the push configuration information and pushed to the subscribing user. Through its table storage, cached data structure design, and configuration change monitoring and update strategy, the throughput and efficiency of data push are improved, while ensuring timely application of push configuration updates to improve the flexibility of business data push. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1A flowchart illustrating the steps of a real-time business data push method under a multi-tenant architecture, as provided in this embodiment of the invention;

[0053] Figure 2 A flowchart illustrating the module implementation of a real-time business data push method under a multi-tenant architecture, as provided in this embodiment of the invention.

[0054] Figure 3 This is a structural block diagram of a real-time business data push device under a multi-tenant architecture, provided in an embodiment of the present invention. Detailed Implementation

[0055] This invention provides a method and apparatus for real-time business data push in a multi-tenant architecture. It addresses the technical problem of traditional data synchronization push schemes failing to guarantee accurate business data push control and reduced push reliability when dealing with large-scale data synchronization, due to the inability of data subscription configuration changes to take effect in real time. By monitoring real-time business data changes, and through methods such as data subscription configuration, data filtering preprocessing, and configuration data monitoring, it can handle scenarios with large-scale data synchronization pushes and ensure that subscription configuration changes take effect in real time. The data push scope can be controlled down to the field level, ensuring the accuracy and continuity of business data pushes, avoiding over- or under-pushing issues, improving push reliability, and resolving data consistency problems between multiple systems.

[0056] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0057] Please see Figure 1 , Figure 1 A flowchart illustrating the steps of a real-time business data push method under a multi-tenant architecture, as provided in this embodiment of the invention.

[0058] This invention provides a real-time business data push method under a multi-tenant architecture, applied to a data push terminal, the method comprising:

[0059] Step 101: When business data is detected generated by the business platform, the data is filtered by matching the data type and unique tenant identifier of the business data with a Bloom filter to obtain the data to be pushed.

[0060] Tenants refer to different customer groups using cloud computing platforms or SaaS software services. While these customers share the infrastructure of the cloud computing platform or the core functionality of the SaaS software, they are typically independent in terms of data, configuration, and permissions. Systems are categorized into multi-tenant systems and single-tenant systems. A multi-tenant system refers to a system that can support multiple tenants. A tenant is usually a group of users with similar access patterns and permissions; a typical tenant is several users from the same organization or company. Multi-tenant technology, or SaaS for short, is a software architecture technology designed to enable the use of the same program in a multi-user environment while ensuring data isolation between users.

[0061] A Bloom filter is a probabilistic data structure used to quickly determine whether an element belongs to a set. Its basic principle is based on a bit array and multiple hash functions. In this embodiment, the set is a tenant identifier set, which includes multiple unique tenant identifiers to be pushed to.

[0062] As the carrier of business activities, the business platform continuously generates various types of business data during operation, such as static and dynamic business data. This data storage can be achieved through business data tables in a relational database. In this embodiment, under a multi-tenant architecture based on shared tables, business data is stored in the same database table. However, a unique tenant identifier field is added to the table to distinguish data ownership. Change operations such as adding, deleting, and modifying data in the business data table can be obtained by middleware listening to binlog logs and sending the change records to a specified topic in the message queue. Topics can be distinguished by table name or partitioned based on primary key ID. The structure of the business data table is shown in Table 1 below:

[0063]

[0064] In this embodiment, the data push terminal communicates with the business platform and uses middleware to monitor and parse the business data generated by the platform in real time. For static business data, the binlog logs in the relational database can be monitored; for dynamic business data, changes in the business data message queue can be monitored in real time to monitor changes such as additions, updates, or deletions of business data. Each piece of business data is associated with a unique tenant identifier.

[0065] When the data push terminal detects business data generated by the business platform or receives a scheduled task trigger instruction, it obtains the data type and unique tenant identifier corresponding to the business data, matches it with the corresponding Bloom filter to filter it, and obtains the data to be pushed.

[0066] In one example of this application, step 101 may include the following sub-steps:

[0067] When business data is detected from the business platform, the corresponding Bloom filter is matched according to the data type of the business data.

[0068] If a match is found, a Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set.

[0069] If it does not exist, then stop pushing business data;

[0070] If the Bloom filter exists or fails to match, the business data will be identified as data to be pushed.

[0071] The tenant identifier set refers to the set of multiple unique tenant identifiers that subscribe to business data, and it is stored in the Bloom filter.

[0072] In this embodiment, when new business data is detected from the business platform, the data push terminal obtains the data type and unique tenant identifier corresponding to the business data. It then matches the data against a Bloom filter. If the match is successful, the Bloom filter further determines whether the unique tenant identifier exists in its built-in tenant identifier set. If the match fails, it indicates that there is no need to filter the business data at this time, or that the data type of the business data has not been recorded. In this case, the business data can be identified as data to be pushed and proceed directly to the next processing step.

[0073] After a successful match with the Bloom filter, if the unique tenant identifier does not exist in the built-in tenant identifier set, it indicates that the business data does not need to be pushed externally. In this case, the business data can be deleted or the push process can be terminated directly to stop pushing business data. If the unique tenant identifier exists in the built-in tenant identifier set, it indicates that the business data needs to be pushed. It is identified as data to be pushed and enters the next processing step. The Bloom filter mechanism can remove data that does not need to be pushed externally, reducing the amount of useless data.

[0074] The tenant identifier set can be designed using a cached data structure to store the set of all subscribers who have subscribed to a certain business data type. The set of all authorized tenant identifiers can be retrieved based on the subscriber set, and so on for other business data types. The specific cached data structure design is shown in Table 2 below:

[0075]

[0076] Optionally, before executing step 101, to facilitate subsequent field conversion and matching of subscription configuration information, the following steps S11-S13 can be executed through the subscription configuration center to create subscription configuration information and load it into the first push configuration cache and the second push configuration cache. The permission control center can then execute steps S21-S22 to create permission-related tables and load them into the Bloom filter, the first push configuration cache, or the second push configuration cache on the data push end.

[0077] In one example of this application, the data push client also communicates with the subscription configuration center, and the method further includes the following steps S11-S13:

[0078] S11. Receive subscription information sent by any subscriber client through the subscription configuration center, and create corresponding subscription configuration information in the built-in subscription configuration table;

[0079] In this embodiment, to facilitate users' subscription to business data, a graphical user interface (GUI) can be provided through the subscription configuration center for users to configure their subscriptions. Subscribers can specify the target address information, key, and user protocol for receiving data through the GUI, and can also select the type of business data to be subscribed to. They can subscribe to multiple business data types under any tenant.

[0080] Specifically, the subscription configuration center receives subscription information from any subscriber client, parses it to obtain target address information, user protocol, and key information, and creates corresponding subscription configuration information. For ease of storage, the subscription configuration information can be stored in the form of a subscription configuration table, the specific table structure of which is shown in Table 3 below:

[0081]

[0082] Among them, the subscriber ID is the ID of the subscriber user, and the tenant ID is the tenant identifier.

[0083] The subscription configuration table described above can store information related to the subscriber's receiving address, supporting multiple receiving addresses or protocol types for a single subscriber, enabling one-to-many push notifications for each subscriber. This table structure can also be expanded with more fields to enhance subscription functionality, such as limiting the pushable time period and specifying the number of retries. The protocol field is set as an enumeration value, with values ​​ranging from the types of protocols supported by the publisher, such as internet HTTPS calls and intranet HTTPS calls. By using different protocol types, more subscribers can flexibly subscribe to the data they need.

[0084] S12. When the data push terminal detects that the business platform generates business data or receives a scheduled task trigger instruction, it traverses the subscription configuration table through the subscription configuration center and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache.

[0085] In this embodiment, when the data push terminal detects that the business platform generates business data or receives a scheduled task trigger instruction, i.e. when the data push service starts, the subscription configuration center traverses the subscription configuration table and loads the existing and valid subscription configuration information into the first push configuration cache and the second push configuration cache as push configuration information to preheat the relevant cached data.

[0086] S13. When the subscription configuration center detects configuration update information, update the second push configuration cache according to the configuration update information.

[0087] Configuration update information refers to the change log in the configuration repository that stores subscription configuration information.

[0088] In this embodiment, since some subscribers may need to add subscriptions during the operation of the data push service, the subscription configuration center can monitor the configuration update information in real time. By monitoring the binlog change log of the configuration library, the center can obtain the addition, deletion, and modification operations of the subscription configuration information in real time and update the cache-related data in the second push configuration cache in real time, thereby updating the push configuration and ensuring that the configuration takes effect immediately.

[0089] In one example of this application, the subscription configuration center is also used to perform the following steps:

[0090] When the subscription configuration center detects a field mapping change, it updates the field conversion cache according to the field mapping change information.

[0091] The field mapping change information includes the new stored field name and the field name pushed to the outside world. This configuration allows you to modify the field information pushed to the outside world, ensuring the consistency of the field display.

[0092] In this embodiment, the subscription configuration center monitors the binlog of the business data field mapping table in real time. When it detects changes in the input field mapping, it updates the field conversion cache according to the changes in the field mapping.

[0093] In this embodiment, the subscription configuration center may also store a business data field mapping table, which is preloaded into the field conversion cache when the data push client starts. This business data field mapping table is shown in Table 4 below:

[0094]

[0095] By subscribing to the configuration center and traversing the above business data field mapping table, the corresponding data type id, storage field name, and external push field name are loaded into the data push cache as a field conversion cache. It can be stored using a hash structure, and its cache data structure design is shown in Table 5 below:

[0096]

[0097] In one example of this application, the data push client also communicates with the access control center, and the method further includes S21-S22:

[0098] S21. Respond to the permission control information sent by the management terminal through the permission control center, and adjust the business data type management table, subscribed data type management table and authorized tenant scope management table corresponding to each subscriber terminal;

[0099] Access control information refers to information instructions used to adjust the subscription permissions of business data or subscribers. These subscription permissions may include, but are not limited to, the business data type management table, the subscribed data type management table, and the authorized tenant scope management table.

[0100] In this embodiment, the access control center can receive access control information through a graphical user interface operated by administrators, or receive access control information sent by a management terminal with which it is connected. The center parses this access control information to obtain corresponding control instructions, and adjusts the permissions of the business data type management table, subscribed data type management table, and authorized tenant scope management table for each subscriber. Specifically, the administrator can choose whether to grant subscribers access and which business data subscription permissions to grant. The administrator can also choose the data scope to authorize subscribers, authorizing by single tenant, by region, by organization account, etc. Subscribers cannot receive unauthorized tenant business data or unauthorized business data types.

[0101] Specifically, the business data type management table for subscribers can be adjusted through access control information. The specific table structure is shown in Table 6 below:

[0102]

[0103] In this embodiment, the business data type management table for subscribers is adjusted using the aforementioned business data type table. This table manages all business data types and the data types that can be subscribed to by subscribers. Subscribers can only subscribe to the allowed data types. The "Allow Subscription" field is set to an enumeration value, which can be set to 1 to indicate that subscription is allowed, and a value of 0 to indicate that subscription is not allowed.

[0104] For the subscribed data type management table on the subscriber's end, after the subscriber submits a subscription application in the subscription configuration center, the administrator can review the application through the permission control center, managing the business data types subscribed by the subscriber and the review status of the application. Specifically, this can be achieved by adjusting the subscribed data type management table through permission control information. The table structure is shown in Table 7 below:

[0105]

[0106] In this embodiment, a subscriber client can apply to subscribe to multiple business data types, and a business data type can be applied to by multiple subscriber clients. The subscription review status field is set to an enumeration value, and the value range can be pending review, reviewed, review failed, etc., corresponding to 0, 1, 2 or other values ​​respectively. Only subscription applications that have passed the review will push the corresponding business data type to the corresponding subscriber client.

[0107] The authorized tenant scope management table for each subscriber client can be adjusted using the following permission control information. Its table structure is shown in Table 8 below:

[0108]

[0109] In this embodiment, the aforementioned authorized tenant scope management table is used to manage the data scope authorized by the subscriber client. The authorization type is set as an enumeration value, and the value range can be authorization by single tenant, authorization by region, authorization by organization account, etc., corresponding to 0, 1, 2 or other values ​​respectively. Authorization can be performed on the same subscriber at the same time according to three dimensions, improving the flexibility of authorization. Changes to tenant business data that have not been authorized to the subscriber will not be pushed to the subscriber.

[0110] S22. When the subscribed data type management table or the authorized tenant scope management table is adjusted, update the Bloom filter and / or the first push configuration cache through the permission control center according to the adjusted content.

[0111] In this embodiment of the application, when the subscribed data type management table or the authorized tenant scope management table is adjusted, the binlog logs of the subscribed data type management table and the authorized tenant scope management table are monitored. When a data change is detected, the permission control center synchronously updates the tenant identifier set in the Bloom filter to update the data type of the business data in real time.

[0112] In addition, since the subscribed data type management table or the authorized tenant scope management table also involve subscriber ID, data type of business data and subscription status, when the above tables are updated, the Bloom filter and / or the first push configuration cache should be updated according to the adjusted content.

[0113] For example, if only the business data type identifier is changed, i.e. the data type is changed, then the data type in the Bloom filter is changed. If there are changes such as subscriber ID, authorized tenant ID, etc., then the first push configuration cache is updated.

[0114] Step 102: Determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0115] In this embodiment, after using a Bloom filter to initially filter the business data and select the data to be pushed, the cache key value corresponding to the data type and unique tenant identifier of the data to be pushed is obtained. Then, the cache key value is further determined by searching the pre-warmed and loaded push configuration cache. If the cache key value does not exist in the push configuration cache, the push process for the data to be pushed is terminated. If it exists, the set of subscriber IDs to be pushed is obtained, thereby obtaining the target interface address, key and protocol information of the push.

[0116] In one example of this application, step 102 may include the following sub-steps:

[0117] Based on the data type and unique tenant identifier of the data to be pushed, match at least one corresponding cache key-value from the pre-loaded first push configuration cache;

[0118] Each cache key-value pair is matched with the corresponding push key-value pair in the preloaded second push configuration cache.

[0119] If a match is found, the push configuration information associated with the cached key-value pair is extracted from the push key-value pair.

[0120] If a match fails, the push of data to be pushed will be stopped.

[0121] It should be noted that existing and valid subscription configuration information can be loaded into the first and second push configuration caches on the data push end by traversing the subscription configuration table. A hash structure can be used for storage. The first push configuration cache is shown in Table 9 below for cache data structure design:

[0122]

[0123] Table 9 can retrieve the set of all business data types subscribed to by a subscriber based on a subscriber ID, and then retrieve the set of all tenant IDs authorized to the subscriber based on the subscriber ID. The push configuration information is then loaded into the cache in sequence and can be stored using a set collection.

[0124] The second push configuration cache is shown in Table 10 below, which illustrates the cache data structure design:

[0125]

[0126] In this embodiment, Table 9 is retrieved according to the data type identifier and unique tenant identifier of the data to be pushed, and all subscriber IDs are matched as cache key values. Then, Table 10 is retrieved according to the cache key values ​​to determine the subscription configuration information corresponding to each cache key value, that is, the set of configuration information entity classes in Table 10 (including receiving address, key, protocol, etc.).

[0127] In addition, it can monitor the binlog logs of the subscription configuration table in real time. When a data change is detected, the cached data in table 10 is updated. The binlog logs of the subscribed data type management table and the authorized tenant scope management table are also monitored. When a data change is detected, the cached data in table 9 is updated.

[0128] Step 103: Perform field transformation on the data to be pushed to generate field data;

[0129] In this embodiment, after determining the data to be pushed, the fields of each piece of data to be pushed are transformed in accordance with the pre-warmed and loaded field transformation mapping configuration to perform field and data format conversion processing, generate field data, and complete the data preparation before push.

[0130] In one example of this application, step 103 may include the following sub-steps:

[0131] The cached hash table matches the preloaded fields based on the data type of the data to be pushed;

[0132] Check whether the storage field name of each piece of data to be pushed exists in the field hash table;

[0133] If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain the intermediate data;

[0134] If it does not exist, then stop pushing the field corresponding to the storage field name;

[0135] The intermediate data is converted according to the preset data format to generate field data.

[0136] Please refer to Table 5, which is the field conversion cache in this embodiment of the invention. After the field conversion cache is preloaded to the data push end, the corresponding field hash table is matched from the preloaded field conversion cache according to the data type of the data to be pushed.

[0137] Next, by traversing the field hash table, it checks whether the storage field name of each piece of data to be pushed exists in the hash table. If it does, the data can be pushed, and the storage field name can be converted to the externally pushed field name according to the hash table to obtain intermediate data. If it does not exist, it means that the data to be pushed is not configured with a field conversion mapping, i.e., it is not recorded in the field hash table. The data to be pushed is retained or fed back to the user for further processing. After completing the field conversion of all data to be pushed, the intermediate data is formatted according to a preset data format to obtain field data that meets the needs of each subscriber's client. This configuration allows modification of the externally pushed field data to ensure consistency in the field display.

[0138] Step 104: Encrypt the field data according to the push configuration information and push it to the subscriber's terminal.

[0139] In this embodiment, after completing the mapping and conversion of field data, the final field data is encrypted according to the push configuration information, and different specifications are called according to the protocol configuration to realize the final push of data.

[0140] Encryption can employ the AES symmetric encryption algorithm, and thread pool technology is used to achieve high-concurrency push. If data push fails, such as due to network anomalies, it can be retried according to the configured number of retries, after which no further retries will be made. Due to the stateless nature of the module executing this push, it can be rapidly horizontally scaled, thereby ensuring the performance of the centralized push module.

[0141] In one example of this application, the subscription configuration information includes a user agreement, a user key, and at least one target address; step 104 may include the following sub-steps:

[0142] The user key is used to encrypt the field data to obtain encrypted data;

[0143] In accordance with the user agreement, encrypted data is pushed to the subscriber terminals corresponding to each target address.

[0144] The subscription client is used for:

[0145] When encrypted data is received, it is decrypted to obtain decrypted data;

[0146] Determine whether the data identifier of the decrypted data is stored in the local database;

[0147] If so, then perform data operations on the local data corresponding to the data identifier according to the decryption data type and data change type.

[0148] If not, add local data to the local database and associate it with a data identifier.

[0149] In this embodiment, after the subscriber receives encrypted data sent by the data push terminal, the encrypted data is decrypted to obtain decrypted data. Then, the data is stored in the database according to its data type and change type. It is determined whether the data identifier of the decrypted data is already stored in the local database. If so, the corresponding data operations are performed on the local data corresponding to the data identifier according to the decrypted data type and the specific data change type (e.g., add, delete, modify, or revise).

[0150] If the data identifier is not in the local database, then the local data is added using decrypted data and associated with the data identifier.

[0151] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a real-time business data push method under a multi-tenant architecture according to an embodiment of the present invention.

[0152] This application's embodiments involve a data push terminal, a subscriber terminal, a business platform, a subscription configuration center, a permission control center, and a cache loading center. Before real-time push of business data, the subscriber terminal configures the subscription configuration table and business data field mapping table in the configuration database through the graphical user interface of the subscription configuration center, and loads them into the cache in the cache loading center. The management terminal configures the business data type management table, subscribed data type management table, and authorized tenant scope management table in the permission control database through the graphical user interface of the permission control center, so as to configure the business data types applied for by the subscriber terminal, the range of subscribed data, and the review status, and loads them into the cache in the cache loading center in real time.

[0153] This cache loading center communicates with the data push client and acts as a real-time cache for the data push client. When the data push client starts, it loads the cached key-value pairs into the Bloom filter, the first push configuration cache, and / or the second push configuration cache. Simultaneously, this cache loading center can also monitor data changes in the configuration database and the access control database in real time. If any database experiences a data change, the Bloom filter, the first push configuration cache, and / or the second push configuration cache are updated in real time.

[0154] After configuring the data push client's cache, the data push client listens for binlog logs on the business database of the business platform and for topic listeners on the dynamic business data in the business data message queue to determine whether to generate the business data required by the subscribing client. When business data is detected, the built-in initial filtering module selects a Bloom filter to perform preliminary filtering of the business data to select the data to be pushed and send it to the secondary filtering module. The secondary filtering module performs secondary filtering on the push configuration information of the data to be pushed. At the same time as the secondary filtering is completed, the field transformation module transforms the fields of the data to be pushed, generates field data, and sends it along with the push configuration information determined by the cache to the centralized push module. Finally, the centralized push module encrypts the field data according to the push configuration information and sends it to the subscribing client.

[0155] Data that does not meet the filtering requirements and is selected in the initial or secondary filtering module will have the data push process terminated directly.

[0156] In this embodiment, the data push terminal stores subscribers' subscription receiving addresses, keys, protocols, etc.; manages and maintains business data types that can be subscribed to, and stores the business data types subscribed to by subscribers; stores subscribers' data authorization dimensions and authorization scope information; preloads a set of tenant IDs authorized to all subscribers according to the business data type and stores it in the corresponding Bloom filter; loads the subscription configuration into the cache (subscriber receiving address, push protocol, encryption key, data push scope and tenant ID set, business data table field mapping, etc.), and monitors changes in table records in the configuration library and refreshes the cache in real time; monitors changes in business data, and uses a Bloom filter to preprocess the data according to the tenant ID to which the business data belongs, reducing the amount of data; ultimately determines whether to push the data and the target address information (the target address can be multiple) based on the business data type and the tenant ID to which it belongs, and completes the conversion of fields and data formats; aggregates the filtered and converted data to be pushed and the destination information into a unified push processor module; the unified push processor pushes the business data changes to the subscriber receiving address according to different protocols.

[0157] For the subscriber client, its functional responsibilities include providing and configuring the subscription receiving address, obtaining encryption / decryption keys, configuring protocols, selecting the business data types to be subscribed to and requesting corresponding permissions, requesting the subscription data range (by single tenant, region, or account), and decrypting and storing and processing the data locally upon receiving the data push. The overall process of incremental data subscription push and the functional design of each module, especially the subscription configuration module, permission control module, configuration loading and monitoring update module, and data filtering and processing module, are presented. The advantages are that each module has a single function and isolated responsibilities, loose coupling between modules, and easy expansion.

[0158] In practical implementation, the above embodiments can achieve the data synchronization function and performance requirements through the following distributed deployment scheme, ensuring high availability of the service, as shown in Table 11:

[0159]

[0160] The middleware required includes:

[0161] LVS / NG load balancer (requires 20M bandwidth) provides reverse proxy, load balancing, and failover capabilities. A cache (Redis) is used to implement a cache-based Bloom filter and store subscription push configuration data. A relational database (MySQL) is used to store business data, push configurations, and authorization data. A message queue (Kafka) is used for decoupling, achieving high throughput, low latency, and improving system concurrency.

[0162] In this embodiment, when business data is detected from the business platform, it is filtered using a Bloom filter based on the data type and unique tenant identifier of the business data to obtain the data to be pushed. Push configuration information is then determined based on the data type and unique tenant identifier of the data to be pushed. Field transformation is performed on the data to be pushed to generate field data. The field data is then encrypted according to the push configuration information and pushed to the subscribing user's client. By utilizing its table storage, cached data structure design, and configuration change monitoring and update strategy, the throughput and efficiency of data push are improved. Simultaneously, updates to the subscription push configuration take effect promptly, enhancing the flexibility of business data push.

[0163] Please see Figure 3 , Figure 3 A structural block diagram of a real-time business data push device under a multi-tenant architecture, as shown in an embodiment of this application, is illustrated.

[0164] This invention provides a real-time business data push device under a multi-tenant architecture, applied to a data push terminal, the device comprising:

[0165] The preliminary filtering module 301 is used to filter business data generated by the business platform according to the data type and unique tenant identifier of the business data by matching a Bloom filter to obtain the data to be pushed.

[0166] The secondary filtering module 302 is used to determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed;

[0167] Field conversion module 303 is used to convert fields in the data to be pushed and generate field data;

[0168] The centralized push module 304 is used to encrypt field data according to the push configuration information and push it to the subscriber's terminal.

[0169] Optionally, the preliminary filtering module 301 is specifically used for:

[0170] When business data is detected from the business platform, the corresponding Bloom filter is matched according to the data type of the business data.

[0171] If a match is found, a Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set.

[0172] If it does not exist, then stop pushing business data;

[0173] If the Bloom filter exists or fails to match, the business data will be identified as data to be pushed.

[0174] Optionally, the secondary filtering module 302 is specifically used for:

[0175] Based on the data type and unique tenant identifier of the data to be pushed, match at least one corresponding cache key-value from the pre-loaded first push configuration cache;

[0176] Each cache key-value pair is matched with the corresponding push key-value pair in the preloaded second push configuration cache.

[0177] If a match is found, the push configuration information associated with the cached key-value pair is extracted from the push key-value pair.

[0178] If a match fails, the push of data to be pushed will be stopped.

[0179] Optionally, the field conversion module 303 is specifically used for:

[0180] The cached hash table matches the preloaded fields based on the data type of the data to be pushed;

[0181] Check whether the storage field name of each piece of data to be pushed exists in the field hash table;

[0182] If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain the intermediate data;

[0183] If it does not exist, then stop pushing the field corresponding to the storage field name;

[0184] The intermediate data is converted according to the preset data format to generate field data.

[0185] Optionally, the push configuration information includes the user agreement, user key, and at least one target address; the centralized push module 304 is specifically used for:

[0186] The user key is used to encrypt the field data to obtain encrypted data;

[0187] In accordance with the user agreement, encrypted data is pushed to the subscriber terminals corresponding to each target address.

[0188] Optionally, the subscription client is used for:

[0189] When encrypted data is received, it is decrypted to obtain decrypted data;

[0190] Determine whether the data identifier of the decrypted data is stored in the local database;

[0191] If so, then perform data operations on the local data corresponding to the data identifier according to the decryption data type and data change type.

[0192] If not, add local data to the local database and associate it with a data identifier.

[0193] Optionally, the data push terminal also communicates with the subscription configuration center, which is specifically used for...

[0194] Receive subscription information sent by any subscriber client and create corresponding subscription configuration information in the built-in subscription configuration table;

[0195] When the data push terminal detects business data generated by the business platform or receives a scheduled task trigger instruction, it iterates through the subscription configuration table and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache.

[0196] When the subscription configuration center detects configuration update information, it updates the second push configuration cache according to the configuration update information.

[0197] Optionally, the subscription configuration center is also used for:

[0198] When a field mapping change is detected, the field conversion cache is updated according to the field mapping change information.

[0199] Optionally, the data push terminal also communicates with the access control center, which is specifically used for:

[0200] By responding to the permission control information sent by the management terminal, adjust the business data type management table, subscribed data type management table and authorized tenant scope management table corresponding to each subscriber terminal;

[0201] When the subscribed data type management table or the authorized tenant scope management table is adjusted, update the Bloom filter and / or the first push configuration cache according to the adjustment.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0204] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0206] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time push of business data under a multi-tenant architecture, characterized in that, Applied to a data push terminal, the method includes: When business data is detected from the business platform, it is filtered using a Bloom filter based on the data type and unique tenant identifier of the business data to obtain the data to be pushed. Determine the push configuration information based on the data type and unique tenant identifier of the data to be pushed; The data to be pushed is converted into fields to generate field data; The field data is encrypted according to the push configuration information and pushed to the subscribing user's terminal; The step of determining the push configuration information according to the data type and unique tenant identifier of the data to be pushed includes: Based on the data type and unique tenant identifier of the data to be pushed, match at least one corresponding cache key-value from the pre-loaded first push configuration cache; Each of the aforementioned cache key-value pairs is used to match the corresponding push key-value pairs in the preloaded second push configuration cache; If a match is successful, the push configuration information associated with the cache key-value pair is extracted from the push key-value pair; If a match fails, the push of the data to be pushed will be stopped; The data push terminal is also connected to the access control center, and the method further includes: The permission control center responds to the permission control information sent by the management terminal and adjusts the business data type management table, subscribed data type management table and authorized tenant scope management table corresponding to each of the subscribed user terminals. When the subscribed data type management table or the authorized tenant scope management table is adjusted, the Bloom filter and / or the first push configuration cache are updated through the permission control center according to the adjustment content.

2. The method according to claim 1, characterized in that, When business data is detected generated by the business platform, the data is filtered using a Bloom filter based on the data type and unique tenant identifier to obtain the data to be pushed, including: When business data is detected from the business platform, a corresponding Bloom filter is matched according to the data type of the business data. If a match is successful, the Bloom filter is used to determine whether the unique tenant identifier of the business data exists in the built-in tenant identifier set; If it does not exist, then stop pushing the aforementioned business data; If the Bloom filter exists or fails to match, the business data is determined to be data to be pushed.

3. The method according to claim 1, characterized in that, The step of performing field transformation on the data to be pushed to generate field data includes: According to the data type of the data to be pushed, the preloaded fields are converted and cached to match the corresponding field hash table; Determine whether the storage field name of each of the data to be pushed exists in the field hash table; If it exists, the storage field name of the data to be pushed is converted into the external push field name according to the field hash table to obtain intermediate data; If it does not exist, then stop pushing the field corresponding to the stored field name; The intermediate data are converted according to a preset data format to generate field data.

4. The method according to claim 1, characterized in that, The push configuration information includes a user agreement, a user key, and at least one target address; encrypting the field data according to the push configuration information and pushing it to the subscribing user's terminal includes: The field data is encrypted using the user key to obtain encrypted data; The encrypted data is pushed to the subscriber client corresponding to each of the target address information in accordance with the user agreement.

5. The method according to claim 4, characterized in that, The subscription client is used for: When the encrypted data is received, it is decrypted to obtain the decrypted data; Determine whether the data identifier of the decrypted data is stored in a local database; If so, then perform data operations on the local data corresponding to the data identifier according to the decryption data type and data change type corresponding to the decryption data; If not, add local data to the local database and associate it with the data identifier.

6. The method according to claim 5, characterized in that, The data push terminal is also connected to the subscription configuration center, and the method further includes: The subscription configuration center receives subscription information sent by any subscriber client and creates corresponding subscription configuration information in the built-in subscription configuration table. When the data push terminal detects that the business platform generates business data or receives a scheduled task trigger instruction, it traverses the subscription configuration table through the subscription configuration center and loads the existing subscription configuration information into the first push configuration cache and the second push configuration cache. When the subscription configuration center detects configuration update information, it updates the second push configuration cache according to the configuration update information.

7. The method according to claim 6, characterized in that, The method further includes: When the subscription configuration center detects a field mapping change, it updates the field conversion cache according to the field mapping change information.

8. A real-time business data push device under a multi-tenant architecture, characterized in that, The device, used in data push applications, includes: The preliminary filtering module is used to filter business data generated by the business platform according to the data type and unique tenant identifier of the business data by matching a Bloom filter to obtain the data to be pushed. The secondary filtering module is used to determine the push configuration information according to the data type and unique tenant identifier of the data to be pushed; The field conversion module is used to convert the fields of the data to be pushed and generate field data. A centralized push module is used to encrypt the field data according to the push configuration information and push it to the subscriber's terminal; The secondary filtering module is specifically used for: Based on the data type and unique tenant identifier of the data to be pushed, match at least one corresponding cache key-value from the pre-loaded first push configuration cache; Each of the aforementioned cache key-value pairs is used to match the corresponding push key-value pairs in the preloaded second push configuration cache; If a match is successful, the push configuration information associated with the cache key-value pair is extracted from the push key-value pair; If a match fails, the push of the data to be pushed will be stopped; The data push terminal is also connected to the access control center, which is specifically used for: In response to the permission control information sent by the management terminal, adjust the business data type management table, subscribed data type management table and authorized tenant scope management table corresponding to each of the aforementioned subscription user terminals; When the subscribed data type management table or the authorized tenant scope management table is adjusted, update the Bloom filter and / or the first push configuration cache according to the adjustment content.

Images