Multi-tenant data transmission method, device and system

Abstract

The invention discloses a multi-tenant data transmission method, device and system, and relates to the technical field of data transmission. The method is applied to a SaaS platform constructed based on SpringBoot in a multi-tenant data transmission system, and comprises the following steps: acquiring task data of a data transmission task under the condition of detecting that a newly added data transmission task exists in a MySQL task table; using a business type and a tenant identifier in the task data as a task execution identifier, and registering the task execution identifier to an XXL-JOB; under the condition that a preset execution condition is met, calling all service nodes deployed on the SaaS platform, preempting a task execution identifier called back by the XXL-JOB, and taking the service node preempting the task execution identifier as a target service node; and calling the target service node, and executing the data transmission task. The purpose of reducing code redundancy is achieved, meanwhile, the situation that a plurality of service nodes repeatedly execute the data transmission task is avoided, and the flexibility and efficiency of executing the data transmission task are improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and more specifically, to a multi-tenant data transmission method, device and system. Background Art

[0002] In the tide of digital economy, data has become an indispensable core resource for enterprises and runs through every link of production. Usually, the enterprise's server receives data transmission tasks from multiple tenants, such as pulling data from the tenant and synchronizing it to the enterprise's database, or responding to the tenant's push request and packaging the data in the database and pushing it to the tenant.

[0003] Currently, XXL-JOB distributed scheduling tasks can be used to realize data transmission under multi-tenants and reduce the pressure of the server to realize data transmission on a single line. However, every time a new data transmission task is added, if the data transmission task is to be executed, it is necessary to configure the execution code corresponding to the data transmission task, or to configure a new executor in XXL-JOB. If the transmission task changes, a large amount of code needs to be rewritten or modified, which has poor flexibility. Moreover, when the differences between data transmission tasks are not large, a large amount of code with similar or repeated functions is configured, which increases the code redundancy of the server and makes it impossible to efficiently realize the data transmission task. Therefore, how to efficiently and flexibly realize the transmission tasks between multi-tenants and servers has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above problems, the present application provides a multi-tenant data transmission method, device and system to achieve the purpose of efficiently and flexibly realizing the transmission tasks between multi-tenants and servers. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a multi-tenant data transmission method, which is applied to a SaaS platform built on SpringBoot in a multi-tenant data transmission system, wherein the multi-tenant data transmission system at least further includes: XXL-JOB and MySQL, and the multi-tenant data transmission method includes:

[0006] When it is detected that there is a newly added data transmission task in the task table of the MySQL, task data of the data transmission task is obtained, where the task data at least includes: a business type and a tenant identifier;

[0007] Register the business type and the tenant identifier in the task data as task execution identifiers to the XXL-JOB;

[0008] When the preset execution conditions are met, all service nodes deployed on the SaaS platform are called to seize the task execution identifier of the XXL-JOB callback, and the service node that seizes the task execution identifier is used as the target service node;

[0009] The target service node is called to execute the data transmission task.

[0010] In a possible implementation, the calling the target service node to perform the data transmission task includes:

[0011] Calling the target service node to identify the request type of the data transmission request of the target address in the task data corresponding to the data transmission task;

[0012] In the case where the request type is identified as a push request, obtaining the data to be pushed corresponding to the push request of the target address, encrypting the data to be pushed according to a preset encryption protocol to obtain target push data, and pushing the target push data to the target address;

[0013] In the case where it is identified that the request type is a pull request, target pull data is obtained from the target address, the target pull data is decrypted according to the preset encryption protocol to obtain target data, and the target data is stored in the business table of MySQL.

[0014] In a possible implementation, obtaining the target pull data from the target address includes:

[0015] Identify a data pulling mode corresponding to the task configuration identifier of the data transmission task, the data pulling mode includes: a fixed number of pulling mode and a non-fixed cycle empty pulling mode, the fixed number of pulling mode is to pull data from the target address for a preset number of times, and the non-fixed cycle empty pulling mode is to continuously pull data from the target address until the pulling result is empty;

[0016] Based on the data pull mode, target pull data is acquired from the target address.

[0017] In a possible implementation, pushing the target push data to the target address includes:

[0018] Identify a data push mode corresponding to the task configuration identifier of the data transmission task, the data push mode including: a fixed number of push mode and an unfixed cycle push mode, the fixed number of push mode is to push the target push data to the target address for a preset number of times, the unfixed cycle push mode is to continuously push the target push data to the target address until the target push data is empty;

[0019] Based on the data push mode, the target push data is pushed to the target address.

[0020] In a possible implementation, the acquiring the to-be-pushed data corresponding to the push request of the target address includes:

[0021] Querying the business data whose push status is not pushed among all the business data corresponding to the business type as the data to be screened;

[0022] Converting the to-be-screened data into preset structure data;

[0023] The fields in the preset structure data that do not conform to the push request are filtered to obtain the data to be pushed.

[0024] In a possible implementation, the method further includes:

[0025] The push status of the data to be pushed in the business data is changed to a pushed status.

[0026] In a possible implementation, the calling of all service nodes deployed on the SaaS platform, seizing the task execution identifier of the XXL-JOB callback, and taking the service node that has seized the task execution identifier as the target service node includes:

[0027] Using the task execution identifier as a distributed lock for the data transmission task;

[0028] Call StringRedisTemplate to control all service nodes deployed on the SaaS platform to seize the distributed lock;

[0029] The service node that seizes the distributed lock is determined as the target service node.

[0030] In a possible implementation, the preset execution condition at least includes: the XXL-JOB meets the preset timing execution requirement.

[0031] The second aspect of the present application provides a multi-tenant data transmission device, which is applied to a SaaS platform built on SpringBoot in a multi-tenant data transmission system, wherein the multi-tenant data transmission system at least further includes: XXL-JOB and MySQL, and the multi-tenant data transmission method includes:

[0032] A task data acquisition unit, configured to acquire task data of the data transmission task when detecting that there is a newly added data transmission task in the task table of the MySQL, wherein the task data at least includes: a business type and a tenant identifier;

[0033] An identification registration unit, used to register the business type and the tenant identification in the task data as task execution identification to the XXL-JOB;

[0034] A service node determination unit is used to call all service nodes deployed on the SaaS platform, seize the task execution identifier of the XXL-JOB callback, and use the service node that seizes the task execution identifier as the target service node when the preset execution condition is met;

[0035] The service calling unit is used to call the target service node to execute the data transmission task.

[0036] The third aspect of the present application provides a multi-tenant data transmission system, including: a SaaS platform built on SpringBoot, XXL-JOB and MySQL, wherein the SaaS platform is used to implement each step of any of the above multi-tenant data transmission methods.

[0037] It can be seen from the above technical solution that the multi-tenant data transmission method provided by the embodiment of the present application is implemented based on XXL-JOB. When a newly added data transmission task is detected, the task to be executed is registered in XXL-JOB according to the business type and tenant identifier of the data transmission task. When the preset execution conditions for executing the data transmission task are met, the SaaS platform receives the data transmission task called back by XXL-JOB, and calls multiple service nodes deployed on the SaaS platform to preempt the data transmission task, ensuring that only one service node executes the data transmission task.

[0038] The embodiment of the present application schedules the service nodes of the SaaS platform to execute data transmission tasks according to the pending tasks registered in XXL-JOB by the business type and tenant identifier. Compared with the execution method of configuring new code or configuring executors in the prior art, it reduces the code redundancy of the server and improves the flexibility of executing data transmission tasks. In addition, by preempting the task, a target service node for executing the data transmission task is determined from multiple service nodes to ensure a one-to-one correspondence between the data transmission task and the service node, avoiding the situation where multiple service nodes repeatedly execute the data transmission task, and improving the execution efficiency of the data transmission task, thereby achieving efficient data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of a method for implementing multi-tenant data transmission provided in an embodiment of the present application;

[0041] Figure 2 A signaling diagram for implementing data pulling provided in an embodiment of the present application;

[0042] Figure 3 A signaling diagram for pushing internship data provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of a multi-tenant data transmission device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] In the field of communication networks, tenants usually refer to independent users or customer groups who register in a specific server or system and use the server or system. The tenants are provided with corresponding services through data transmission between the tenants and the server or system, for example, the tenant registration is completed by pulling the information of new tenants and the subscription data is pushed to the new tenants.

[0046] In the prior art, XXL-JOB is usually used to implement the scheduling of multi-tenant data transmission tasks, wherein XXL-JOB is a distributed task scheduling platform, which is used to uniformly manage the scheduling tasks on the task scheduling platform and is responsible for triggering the scheduling execution. If the scheduled data transmission task is to be executed, it is necessary to configure the code for implementing the execution logic of the data transmission task, and run the code to complete the data transmission task. Each time a data transmission task is executed, a large amount of new code needs to be inserted, and the flexibility of task execution is poor. In addition, for the operation of the XXL-JOB program, the code redundancy of the server is increased, and the data transmission task cannot be efficiently implemented.

[0047] In view of the deficiencies of the prior art, an embodiment of the present application provides a multi-tenant data transmission method, which is applied to a SaaS platform built on SpringBoot in a multi-tenant data transmission system. The multi-tenant data transmission system also includes at least: XXL-JOB and MySQL.

[0048] Among them, the multi-tenant data transmission system is deployed in the service system / server to respond to the tenant's data transmission tasks. The SaaS platform of the embodiment of the present application is developed through the Springboot open source technology framework. It is an actual service platform that realizes the pulling, pushing and transmission of tenant data and can schedule the use of various applications. MySQL is used to store tenant data and can be scheduled by SaaS; XXL-JOB is a distributed task management and scheduling platform that runs independently of the SaaS platform.

[0049] Specifically, refer to Figure 1 , a flowchart of a method for implementing multi-tenant data transmission provided in an embodiment of the present application is provided, which illustrates a multi-tenant data transmission method applied to a SaaS platform, and the specific process includes:

[0050] Step S110, when it is detected that there is a newly added data transmission task in the task table of MySQL, the task data of the data transmission task is obtained.

[0051] Step S120: register the business type and tenant identifier in the task data as the task execution identifier to XXL-JOB.

[0052] In the embodiment of the present application, a tenant or an administrator can create a public business data transmission task through a visual interface, and MySQL records the created public business data transmission task.

[0053] After the tenant issues a data transmission request, MySQL stores the data transmission request or data transmission task in the task table, waiting for the SaaS platform to execute the task. In addition, MySQL can also be used as a service system or server database to store data pulled from tenants and data to be pushed to tenants. Optionally, the SaaS platform can divide the data into multiple task tables according to the type of data transmission task to store related data in MySQL. For example, the task table is divided into a pull task table and a push task table, which store pull task data and push task data respectively.

[0054] Among them, the MySQL task table stores the task data contained in the data transmission task. The task data at least includes: the business type and tenant identifier of the transmitted data. If the data transmission task is a pull task, the task data may also include: pull request URL address, pull request APPID, insert the database INSERT SQL script after pulling the database, AES KEY data decryption, etc.; if the data transmission task is a push task, the task data may also include: push request URL address, push request APPID, push query data SELECT SQL script, AES KEY data encryption, etc.

[0055] Based on the program code of the SaaS platform, the SaaS platform will continuously scan the task table of MySQL, and when a new data transmission task is found, the business type and coordinate identifier in the task data of the data transmission task will be used as the task execution identifier of the new data transmission task and registered to XXL-JOB.

[0056] Step S130, when the preset execution conditions are met, all service nodes deployed on the SaaS platform are called to seize the task execution identifier of the XXL-JOB callback, and the service node that seizes the task execution identifier is used as the target service node.

[0057] Step S140: calling the target service node to execute the data transmission task.

[0058] After registration is completed, it can be determined whether to start executing the data transmission task corresponding to the registered task execution identifier according to the preset execution conditions. The preset execution conditions can be timing conditions, for example, the task data of the data transmission task requires to start execution 5 minutes after the registration is completed, or XXL-JOB is configured to query every 5 seconds whether there is a task execution identifier corresponding to the successfully registered data transmission task, and if the query is hit, the data transmission task is executed.

[0059] When the preset execution conditions are met, XXL-JOB calls back the task execution identifier and transmits it to the SaaS platform, and the service node of the SaaS platform executes the data transmission task corresponding to the task execution identifier. However, the SaaS platform usually deploys multiple service nodes for executing data transmission tasks. In order to avoid the service nodes from repeatedly executing data transmission tasks, in the embodiment of the present application, multiple service nodes simultaneously seize the callback task execution identifier, which can be based on the order of seizure or the matching degree between the service type pre-configured by the service node and the task execution identifier, so that only one service node can receive the task execution identifier, and this service node is used as the target service node to execute the data transmission task corresponding to the task execution identifier.

[0060] In summary, it can be seen from the above technical solutions that the multi-tenant data transmission method provided by the embodiment of the present application is implemented based on XXL-JOB. When a newly added data transmission task is detected, the task to be executed is registered in XXL-JOB according to the business type and tenant identifier of the data transmission task. When the preset execution conditions for executing the data transmission task are met, the SaaS platform receives the data transmission task called back by XXL-JOB, and calls multiple service nodes deployed on the SaaS platform to preempt the data transmission task, ensuring that only one service node executes the data transmission task.

[0061] The embodiment of the present application schedules the service nodes of the SaaS platform to execute data transmission tasks according to the pending tasks registered in XXL-JOB by the business type and tenant identifier. Compared with the execution method of configuring new code or configuring executors in the prior art, it reduces the code redundancy of the server and improves the flexibility of executing data transmission tasks. In addition, by preempting the task, a target service node for executing the data transmission task is determined from multiple service nodes to ensure a one-to-one correspondence between the data transmission task and the service node, avoiding the situation where multiple service nodes repeatedly execute the data transmission task, and improving the execution efficiency of the data transmission task, thereby achieving efficient data transmission.

[0062] Next, other possible implementations of the multi-tenant data transmission method provided in the embodiments of the present application are described.

[0063] In one possible implementation, implementing step S150, calling the target service node, and executing the data transmission task may include: calling the target service node, identifying the request type of the data transmission request of the target address in the task data corresponding to the data transmission task; when the request type is identified as a push request, obtaining the data to be pushed corresponding to the push request of the target address, encrypting the data to be pushed according to a preset encryption protocol, obtaining target push data, and pushing the target push data to the target address; when the request type is identified as a pull request, obtaining the target pull data from the target address, decrypting the target pull data according to the preset encryption protocol, obtaining the target data, and storing the target data in the business table of MySQL.

[0064] It can be understood that the multi-tenant data transmission system is used to provide services to data source partners, and to deal with the data transmission tasks faced by data source partners in two data transmission scenarios: pulling data from the tenant side and pushing data to the tenant side. Among them, the data source partner can be understood as a data source that provides data to tenants, or it can be a data platform that receives tenants' data storage and upload needs, such as: a multimodal digitalization provider. In the scenario of pulling data from the tenant side, the multi-tenant data transmission system is the data puller; in the scenario of pushing data to the tenant side, the multi-tenant data transmission system is the data pusher. The multi-tenant data transmission system has different roles, and the data transmission tasks performed are also different.

[0065] Optionally, in step S110, the transmission type of the data transmission task is determined according to the type of the MySQL task table where the data transmission task is located. Alternatively, according to the request type of the target address request in the task data, for example, the request type of the pull request URL address and the pull request APPID belongs to a pull request, and the request type of the push request URL address and the push request APPID belongs to a push request. The data transmission process for push data and pull data is different. Therefore, the execution process of the service node in the embodiment of the present application is different for different request types.

[0066] For data transmission tasks whose request type is a push request, the service node first responds to the push request in the task data, filters out the target push data from the stored data, encrypts the target push data, and sends the encrypted target push data to the tenant address. The encryption protocol can be encrypted according to the encryption algorithm predetermined between the tenant and the service node, so that the tenant can correctly decrypt the push data after receiving it.

[0067] In a possible implementation process, the data to be pushed corresponding to the push request of the target address is obtained, including: querying the business data with a push status of not pushed in all business data corresponding to the business type as the data to be filtered; converting the data to be filtered into preset structure data; filtering the fields in the preset structure data that do not meet the push request to obtain the data to be pushed.

[0068] For example, the multi-tenant data transmission method provided in the embodiment of the present application is applied to a public account subscription management system, and the multi-tenant data transmission system runs in the public account subscription management system. A tenant refers to a user who subscribes to a public account. In order to avoid repeatedly pushing subscription information to tenants, the embodiment of the present application performs a push status mark on the data stored in MySQL to distinguish between data that has been pushed and data that has not been pushed. Therefore, when the service node is executing a push-type data transmission task, it first obtains business data with a push status of a non-pushed state from the database. After the data that has not been pushed is successfully pushed, the push status of the successfully pushed data is modified to a pushed state, so as to update the data status and avoid pushing duplicate data next time.

[0069] Optionally, to obtain the business data in the unpushed state, you can configure SELECT SQL to query the unpushed List in the business table corresponding to the business type. <string>The collection structure data is processed in a loop. Furthermore, according to the configuration field in the task data, the JSON String field is filtered to obtain the push tenant agreement field, that is, the target push data. Furthermore, according to the agreement with the tenant, the AES KEY is configured to encrypt the target push data.

[0070] It is understandable that the data format, data structure, data type, field type, etc. of the target push data can be adaptively adjusted according to the tenant's needs. For example, if the push data structure that the current tenant can receive is non-collection structure data, the target push data needs to be converted into the corresponding non-collection structure data. Alternatively, the current platform data fields include: name, age, and city, but the tenant requests to push the name and age fields, then all "city" fields in the unpushed fields need to be filtered to ensure that there are only "name" and "age" fields in the target push fields.

[0071] Based on this, taking into account the differences in push requests, receivable data structures, data types, etc. of different tenants, the process of obtaining the target push data, the data format of the target push data, etc. are not uniquely limited to adapt to the needs of different tenants.

[0072] For data transmission tasks whose request type is a pull request, the service node first obtains data from the target address in the pull request to obtain the target pull data. Similar to the above-mentioned push data, the tenant encrypts the pulled data to ensure the security of the data during transmission. Therefore, after obtaining the target pull data, the service node needs to encrypt and decrypt it to obtain the target data and store it in the database.

[0073] In one possible implementation, obtaining target pulled data from a target address includes: identifying a data pulling mode corresponding to a task configuration identifier of a data transmission task, the data pulling modes including: a fixed number of pulling modes and a non-fixed cycle pulling mode, the fixed number of pulling modes is to pull data from the target address for a preset number of times, the non-fixed cycle pulling mode is to continuously pull data from the target address until the pulling result is empty; based on the data pulling mode, obtaining target pulled data from the target address.

[0074] It is understandable that the interface frequency of pulling or pushing business data interfaces may be different for different data partners, and the data pulling methods for different interface frequencies are different. In order to improve the flexibility of the SaaS platform and adapt to the data pulling methods of data transmission tasks with different interface frequencies, the service node can adjust the data pulling mode according to the interface frequency of the identified data transmission task when pulling data.

[0075] In an embodiment of the present application, the service node determines the data pulling mode of the current data pulling task by identifying the task configuration identifier in the task data, such as a page field in the data table of the task data, and pulls the tenant data in the data pulling mode.

[0076] Specifically, the data pulling mode may include: fixed number pulling mode and non-fixed cycle pulling mode. In the fixed number pulling mode, the number of pulls is a preset number during the execution of the data transmission task. For example, the tenant requests the server to pull 100 pieces of data, but under the fixed number pulling mode, it can only pull 3 times, 20 pieces of data each time, so this transmission task can only pull 60 pieces of data. In this data pulling mode, there is a situation where this data transmission task cannot pull all the tenant data.

[0077] The non-fixed cycle empty mode is to continuously request data partners to pull data during the execution of data transmission tasks until the data partner's interface response data is empty. For example, a tenant requests the server to pull 100 pieces of data, and under the requirements of the non-fixed cycle empty mode, the service node will continue to pull tenant data until the tenant data is emptied, and then the service node stops pulling data. In the non-fixed cycle empty mode, data can be pulled in a timely manner to avoid data omissions, but it also causes service pressure on tenants and data partners. The embodiment of the present application does not consider the advantages and disadvantages of each data pulling mode, but only switches to a data pulling mode adapted to the current data transmission task according to the task configuration identifier in the task data.

[0078] Based on this, when calling a service node to perform a pull-type data transmission task, first, the data pull mode of this pull task is determined according to the task configuration identifier in the task data, and data is requested from the target address according to the data pull mode. If the target address responds to the request abnormally or empty, or the service node sends an abnormal request, the current pull task is terminated. After receiving a normal response request from the target address, the data is pulled and converted to List according to the data source friendly agreement. <string>The data in the collection structure is the target data. <string>For the data of the collection structure, parse the JSON String according to the configuration fields in the task data, and configure AESKEY to parse and decrypt the data according to the agreement with the tenant to obtain the target data, and store the target data in the database. Optionally, dynamically insert and update the target data in the corresponding MySQL business table through SQL placeholders. In addition, the ID of each data in the target data can be extracted separately as a callback to the data source partner, that is, the server, to successfully identify the data pulled.

[0079] It is understandable that the interface frequencies of the interfaces of tenants and data partners used to receive data pushed by the service node also have the problem of different frequencies. Therefore, in order to adapt to different interface frequencies, the process of pushing the target push data to the target address may include: identifying the data push mode corresponding to the task configuration identifier of the data transmission task, the data push mode includes: a fixed number of push modes and a non-fixed cycle push mode. The fixed number of push modes is to push the target push data to the target address with a preset number of times, and the non-fixed cycle push mode is to continuously push the target push data to the target address until the target push data is empty; based on the data push mode, the target push data is pushed to the target address.

[0080] The data push mode can also be divided into: fixed number push mode and non-fixed cycle push mode. Among them, the fixed number push mode is that during the execution of the data transmission task, the number of data pushed to the tenant is the preset number; the non-fixed cycle push mode is that during the execution of the data transmission task, data is continuously pushed to the tenant until the push content is empty.

[0081] In another possible implementation, the process of implementing step S130 and determining the service node that matches the task execution identifier as the target service node may include: using the task execution identifier as the distributed lock of the data transmission task; calling StringRedisTemplate to control all service nodes deployed on the SaaS platform to preempt the distributed lock; and determining the service node that preempts the distributed lock as the target service node.

[0082] The embodiment of the present application uses the business type and tenant identifier in the task data as the distributed lock key of the data transmission task. XXL-JOB combines StringRedisTemplate to preempt the data transmission task, so that a data transmission task is only executed by one service node, avoiding multiple service nodes from executing multiple tasks. When each service node preempts a task, all service nodes are called, and each service node preempts the execution of the data transmission task corresponding to the task execution identifier through a distributed lock. The service node that preempts the data transmission task first is the target service node, and other service nodes cannot obtain the distributed lock, thereby achieving the purpose of one-to-one correspondence between service nodes and data transmission tasks.

[0083] Specifically, the following example illustrates the process of the above-mentioned service node preempting the data transmission task. When XXL-JOB meets the preset execution conditions, the SaaS platform will respond to the task execution identifier of the XXL-JOB callback, call all service nodes, and at the same time, the service node will obtain the distributed lock through StringRedisTemplate, that is, obtain the task execution identifier. In the embodiment of the present application, only one service node returns 1, and returning "1" proves that the distributed lock is preempted, while other service nodes return 0, proving that the distributed lock is not preempted. Based on this, the service node that preempts the distributed lock executes the data transmission task corresponding to the distributed lock.

[0084] In summary, the embodiment of the present application provides a multi-tenant data transmission method, involving technical fields such as XXL-JOB, Springboot, SaaS platform, MySQL, AES encryption and decryption, Redis, Kafka, etc., including multi-tenant data pulling tasks and data pushing tasks. The two task scenarios are based on SpringBoot reading MySQL task configuration data at the minute level, and dynamically registering the enabled task configuration to XxlJobExecutor according to "business type + tenant identification", combined with XXL-JOB's Job task scheduled execution. Pull data is dynamically stored in the MySQL database according to tenant, AES decryption, idempotence verification and other strategies; push data is dynamically encrypted with AES and pushed to the target tenant according to tenant identification, tenant interface address, whether the push is successful and other strategies. XxlJobExecutor is adapted to multi-tenants and multi-businesses. If there is a new tenant data transmission, no development is required, and it has the advantages of real-time monitoring, scalability, data isolation, and data security.

[0085] Combination Figure 2 as well as Figure 3 , the implementation of the above multi-tenant data transmission method is illustrated by example. First, refer to Figure 2 , explaining the execution process of the data transmission task of pulling data. The technical architecture can include: SaaS platform (2 SpringBoot service nodes), 1 XXL-JOB task scheduling management platform, AES encryption and decryption, MySQL database, Redis cache database and Kafka message middleware. Maintain and run the XXL-JOB service separately, and create a public business pull scheduled task through the visual interface, and use "Business: PullJobExecuteBean" as the identification Bean operation mode.

[0086] A new tenant is added to the SaaS platform system, and a new target tenant is added to the pull-job table of the MySQL database, which involves business pulling task data, and initializing automatic registration, decryption identification, inserting data into placeholder templates, pulling fields, etc.

[0087] SpringBoot creates a new Task service, and checks the MySQL database pull-job table every 5 seconds. If it is detected that the MySQL database pull-job table has started the initialization automatic registration task, the execution task PullJobExecuteBean is registered to XxlJobExecutor, and the constructor parameter is "business type + tenant identifier". The XXL-JOB service executes tasks regularly, and the callback template configures PullJobExecuteBean and executes it. Assuming that the current tenant transmission system is a public account subscription system, the "business type + tenant identifier" registered by PullJobExecuteBean can be understood as: the currently registered constructor A can pull the data of public account A from user end A to the server, where "public account A" represents the business type and user end A represents the tenant identifier. When the data pulling task is "pull the data corresponding to public account A generated by user end A to the server", XXL-JOB matches constructor A with the data pulling task, and constructor A executes the data pulling task.

[0088] PullJobExecuteBean listens to task scheduling and execution, and uses the "business type + tenant identifier" in the task information corresponding to the data pull task as the distributed lock key, combined with StringRedisTemplate to preempt tasks, to avoid multiple service nodes from executing multiple tasks. After confirming the matching service node, determine whether it is a fixed number of pulls or an unfixed cycle pull based on the task configuration identifier. According to the pull mode, request data according to the target address. If the request is abnormal, the response is abnormal, or the response result is empty, terminate the current cycle request pull task. If there is no abnormality in the response, the response will be converted to List according to the data source friendly agreement. <string>The collection structure is used to parse the List collection in a loop, parse the JSON String according to the configured fields, configure the AES KEY to parse and decrypt the data according to the agreement between the two parties, and dynamically insert the results into the corresponding MySQL business table in batches through the SQL placeholder. In addition, each data ID in the decrypted data is extracted separately and used as a successful identification to call back the data source to the friendly business.

[0089] Reference Figure 3 , a signaling diagram for implementing data pulling provided in the embodiment of the present application, the technical architecture can refer to the above Figure 2 Description

[0090] A new tenant is added to the SaaS platform system, and the target tenant is added to the push-job table of the MySQL database involving business push task data, and automatic registration, encryption identification, push query SELECT SQL, push fields, etc. are initialized.

[0091] SpringBoot creates a new Task service and checks the MySQL database pull-job table every 5 seconds. If it detects that the MySQL database push-job table has started the initialization automatic registration task, it will register the execution task PushJobExecuteBean to XxlJobExecutor, and the constructor parameters are "business type + tenant identifier". For specific dynamic registration of multi-tenant transmission JobExecuteBean executors, please refer to the following example:

[0092] public void refresh() {

[0093] / / Get all automatically registered Job tasks

[0094] List <syspushconfig>jobs = getAllPushJobs();

[0095] if(CollUtil.isEmpty(jobs)){

[0096] return;

[0097] }

[0098] jobs.forEach(job->{

[0099] / / Store push task information, unique key=businessType+customerId, value=job

[0100] pushJobMap.put(job.getBusinessType()+job.getCustomerId(),job);

[0101] / / Dynamically register PushJobHandler to adapt to multi-tenant and multi-business, and no need to add new transmission task code later

[0102] DynamicPushJobHandler.registerJobHandler(job);

[0103] if(ObjectUtil.isNull(XxlJobExecutor.loadJobHandler(getJobHandler(job)))){

[0104] XxlJobExecutor.registJobHandler(getJobHandler(job),newDynamicPushJobHandler(job.getBusinessType()));

[0105] }

[0106] });

[0107] }

[0108] XXL-JOB service executes data push tasks regularly, and the callback template configures PushJobExecuteBean and executes it. Specifically, the code for JobExecuteBean scheduling and executing data transmission can refer to the following example:

[0109] / / Get the current tenant information

[0110] CustomerInfocustomerInfo=getCustomerInfo();

[0111] / / Step 1: Get the tenant configuration SysPushConfig

[0112] SysPushConfigpushJob=PushJobHandlerHolder.getPushJob(jobName,customerInfo.getId());

[0113] / / Step 2: Multi-node distributed lock grabbing execution to avoid repeated execution on multiple nodes

[0114] StringRedisTemplateredisTemplate=SpringUtil.getBean(StringRedisTemplate.class);

[0115] StringredisLockKey=JobHandler.getRedisLockKey(DYNAMIC_PUSH_JOB_HANDLER_REDIS_LOCK,jobName,customerInfo.getId());

[0116] StringlocalhostStr=NetUtil.getLocalhostStr();

[0117] BooleanisLock=JobHandler.tryLock(redisTemplate,

[0118] DYNAMIC_PUSH_JOB_HANDLER_REDIS_LOCK_TIME_OUT,redisLockKey,localhostStr);

[0119] / / Failed to obtain the distributed lock, end this task scheduling early

[0120] if(Boolean.FALSE.equals(isLock)){

[0121] XxlJobContext.getXxlJobContext().setHandleMsg(String.format("%s failed to obtain the lock, jump out of the loop",localhostStr));

[0122] return;

[0123] }

[0124] / / Step 3: Start transmitting data

[0125] try{

[0126] / / Get the number of calls

[0127] intpages=ObjectUtil.isNotNull(pushJob.getPages())?pushJob.getPages():0;

[0128] if(pages<=0){

[0129] for(;;){ / / Unfixed number of times, loop to push empty data

[0130] if(doExecute(customerInfo,pushJob)){

[0131] break;

[0132] }

[0133] }

[0134] }else{

[0135] for(;pages>0;pages--){ / / The page field in the database sys_push_config table limits the number of calls

[0136] if(doExecute(customerInfo,pushJob)){

[0137] break;

[0138] }

[0139] }

[0140] }

[0141] }catch(Exceptione){

[0142] log.error("PushJobHandler.execute execution push method exception, errMsg:{}",JSON.toJSONString(e));

[0143] }finally{

[0144] / / Logic execution ends normally: release the distributed lock

[0145] redisTemplate.delete(redisLockKey);

[0146] }

[0147] Furthermore, PushJobExecuteBean monitors the task scheduling and execution, uses the "business type + tenant identifier" in the task information as the distributed lock key, and combines StringRedisTemplate to preempt the task, thus avoiding multiple service nodes from executing multiple tasks.

[0148] According to the task configuration flag, determine whether it is a fixed number of pushes or an unfixed cycle push; according to the target address request to push data, if the request is abnormal, the response is abnormal, or the response result is empty, terminate the current cycle request push task. According to the configuration, SELECT SQL query specifies that the business table is not pushed List <string>For collection structure data, loop through the List collection, filter the fields in the JSON String according to the configured fields, and only push the fields agreed upon by the tenants; configure AES KEY to encrypt the data according to the agreement between the two parties, and after the push result is successfully pushed, modify the corresponding data push status to "pushed" according to each data ID to avoid repeated data push.

[0149] Specifically, the execution code of the lock grabbing execution and data pushing process can refer to the following example:

[0150] / / Get the current execution tenant information

[0151] CustomerInfo customerInfo = getCustomerInfo();

[0152] / / Step 1: Get the tenant configuration SysPushConfig

[0153] SysPushConfig pushJob = PushJobHandlerHolder.getPushJob(jobName,customerInfo.getId());

[0154] / / Step 2: Multi-node distributed lock grabbing execution to avoid repeated execution on multiple nodes

[0155] StringRedisTemplate redisTemplate = SpringUtil.getBean(StringRedisTemplate.class);

[0156] String redisLockKey = JobHandler.getRedisLockKey(DYNAMIC_PUSH_JOB_HANDLER_REDIS_LOCK, jobName, customerInfo.getId());

[0157] String localhostStr = NetUtil.getLocalhostStr();

[0158] Boolean isLock = JobHandler.tryLock(redisTemplate, DYNAMIC_PUSH_JOB_HANDLER_REDIS_LOCK_TIME_OUT, redisLockKey, localhostStr);

[0159] / / Failed to obtain the distributed lock, end this task scheduling early

[0160] if (Boolean.FALSE.equals(isLock)) {

[0161] XxlJobContext.getXxlJobContext().setHandleMsg(String.format("%s failed to obtain the lock, exit the loop", localhostStr));

[0162] return;

[0163] }

[0164] / / Step 3: Start transmitting data

[0165] try {

[0166] / / Get the number of calls

[0167] int pages = ObjectUtil.isNotNull(pushJob.getPages()) ?pushJob.getPages() : 0;

[0168] if (pages <= 0) {

[0169] for (; ; ) { / / Unfixed number of times, loop to push empty data

[0170] if (doExecute(customerInfo, pushJob)) {

[0171] break;

[0172] }

[0173] }

[0174] } else {

[0175] for (; pages > 0; pages--) { / / Database sys_push_config table page field limits the number of calls

[0176] if (doExecute(customerInfo, pushJob)) {

[0177] break;

[0178] }

[0179] }

[0180] }

[0181] } catch (Exception e) {

[0182] log.error("PushJobHandler.execute push method exception, errMsg:{}",JSON.toJSONString(e));

[0183] finally

[0184] / / Logic execution ends normally: release the distributed lock

[0185] redisTemplate.delete(redisLockKey);

[0186] }

[0187] / / Get API information based on business type

[0188] String apiName = BgPushApiNameEnum.getApiNameByType(sysPushConfig.getBusinessType());

[0189] long timestamp = System.currentTimeMillis();

[0190] / / Determine whether to collect data

[0191] if (pushData instanceof Collection<?> ) {

[0192] Collection list = (Collection<?> ) pushData;

[0193] / / Process the pushed fields

[0194] if (CharSequenceUtil.isNotBlank(sysPushConfig.getPushField())) {

[0195] list = (Collection) dealPushField(list, sysPushConfig.getPushField());

[0196] }

[0197] / / Process the filter fields

[0198] if (CharSequenceUtil.isNotBlank(sysPushConfig.getFilterField())) {

[0199] list = (Collection) dealFilterField(list,sysPushConfig.getFilterField());

[0200] }

[0201] String data = JsonUtil.toJSONString(list);

[0202] log.info("Push original data = apiName:{}, timestamp:{}, data:{}", apiName,timestamp, data);

[0203] / / Both parties agree on AES KEY to encrypt data

[0204] reqContent.put(Constants.DATA, CharSequenceUtil.isBlank(customerInfo.getAesKey()) ?

[0205] list : AESUtil.encrypt(data, customerInfo.getAesKey()));

[0206] }

[0207] In summary, the multi-tenant data transmission method provided by the embodiment of the present application realizes dynamic registration of Job tasks based on SpringBoot and MySQL. New data transmission tasks do not require adding multiple JobExecuteBean codes. The code is highly flexible and has strong expansion capabilities, which greatly improves efficiency. In addition, during the push and pull process, the embodiment of the present application realizes data security based on SpringBoot and AES encryption and decryption to avoid data leakage.

[0208] The multi-tenant data transmission device provided in an embodiment of the present application is described below. The multi-tenant data transmission device described below and the multi-tenant data transmission method described above can be referenced to each other.

[0209] First, combine Figure 4 , introduces the multi-tenant data transmission device of the SaaS platform built on SpringBoot, which is applied to the multi-tenant data transmission system. Figure 4 As shown, the multi-tenant data transmission device may include:

[0210] The task data acquisition unit 100 is used to acquire task data of the data transmission task when detecting that there is a newly added data transmission task in the task table of the MySQL, wherein the task data at least includes: a business type and a tenant identifier;

[0211] An identification registration unit 200, used to register the business type and the tenant identification in the task data as task execution identification to the XXL-JOB;

[0212] The service node determination unit 300 is used to call all service nodes deployed on the SaaS platform, seize the task execution identifier of the XXL-JOB callback, and use the service node that seizes the task execution identifier as the target service node when the preset execution condition is met;

[0213] The service calling unit 400 is used to call the target service node to execute the data transmission task.

[0214] In a possible implementation, the service calling unit 400 includes:

[0215] A type identification subunit, used to call the target service node to identify the request type of the data transmission request of the target address in the task data corresponding to the data transmission task;

[0216] The data push subunit is used to, when identifying that the request type is a push request, obtain the data to be pushed corresponding to the push request of the target address, encrypt the data to be pushed according to a preset encryption protocol, obtain target push data, and push the target push data to the target address;

[0217] The data pulling subunit is used to obtain the target pulling data from the target address when identifying that the request type is a pull request, decrypt the target pulling data according to the preset encryption protocol to obtain the target data, and store the target data in the business table of MySQL.

[0218] In a possible implementation, the data pulling subunit includes:

[0219] A pull mode identification subunit, used to identify a data pull mode corresponding to the task configuration identifier of the data transmission task, wherein the data pull mode includes: a fixed number of pull modes and an unfixed cycle pull mode, wherein the fixed number of pull modes is to pull data from the target address at a preset number of times, and the unfixed cycle pull mode is to continuously pull data from the target address until the pull result is empty;

[0220] The data pulling subunit is used to obtain target pulling data from the target address based on the data pulling mode.

[0221] In a possible implementation, the data push subunit includes:

[0222] A push mode identification subunit is used to identify the data push mode corresponding to the task configuration identifier of the data transmission task, and the data push mode includes: a fixed number of push mode and an unfixed cycle push mode. The fixed number of push mode is to push the target push data to the target address for a preset number of times, and the unfixed cycle push mode is to continuously push the target push data to the target address until the target push data is empty;

[0223] The data push subunit is used to push the target push data to the target address based on the data push mode.

[0224] In a possible implementation, the data push subunit includes:

[0225] A data query subunit is used to query the business data whose push status is not pushed in all the business data corresponding to the business type as data to be screened;

[0226] A structure conversion subunit, used for converting the to-be-screened data into preset structure data;

[0227] The field filtering subunit is used to filter the fields in the preset structure data that do not conform to the push request to obtain the data to be pushed.

[0228] In a possible implementation, the method further includes:

[0229] The state modification subunit is used to modify the push state of the data to be pushed in the business data to a pushed state.

[0230] In a possible implementation, the service node determination unit includes:

[0231] A distributed lock determination subunit, configured to use the task execution identifier as a distributed lock for the data transmission task;

[0232] The node preemption subunit is used to call StringRedisTemplate to control all service nodes deployed on the SaaS platform to preempt the distributed lock;

[0233] The target node determination subunit is used to determine the service node that seizes the distributed lock as a target service node.

[0234] In a possible implementation, the preset execution condition at least includes: the XXL-JOB meets the preset timing execution requirement.

[0235] In summary, the multi-tenant data transmission method provided in the embodiment of the present application is implemented based on XXL-JOB. When a newly added data transmission task is detected, the task to be executed is registered in XXL-JOB according to the business type and tenant identifier of the data transmission task. When the preset execution conditions for executing the data transmission task are met, the SaaS platform receives the data transmission task called back by XXL-JOB, and calls multiple service nodes deployed on the SaaS platform to preempt the data transmission task, ensuring that only one service node executes the data transmission task.

[0236] The embodiment of the present application schedules the service nodes of the SaaS platform to execute data transmission tasks according to the pending tasks registered in XXL-JOB by the business type and tenant identifier. Compared with the execution method of configuring new code or configuring executors in the prior art, it reduces the code redundancy of the server and improves the flexibility of executing data transmission tasks. In addition, by preempting the task, a target service node for executing the data transmission task is determined from multiple service nodes to ensure a one-to-one correspondence between the data transmission task and the service node, avoiding the situation where multiple service nodes repeatedly execute the data transmission task, and improving the execution efficiency of the data transmission task, thereby achieving efficient data transmission.

[0237] It is understandable that before using the technical solutions disclosed in the embodiments of this disclosure, the types, scope of use, and usage scenarios of the personal information involved in this disclosure should be informed to the user in an appropriate manner in accordance with relevant laws and regulations and the user's authorization should be obtained. If the solution involves collecting user personal data (such as face, log data, etc.), the above paragraph can be added to the manual to avoid issuing notices due to infringement of user data privacy security.

[0238] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0239] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0240] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / string> < / syspushconfig> < / string> < / string> < / string> < / string>

Claims

1. A multi-tenant data transmission method, characterized in that: The SaaS platform based on SpringBoot is applied to a multi-tenant data transmission system, wherein the multi-tenant data transmission system at least includes: XXL-JOB and MySQL, and the multi-tenant data transmission method includes: When it is detected that there is a newly added data transmission task in the task table of the MySQL, task data of the data transmission task is obtained, where the task data at least includes: a business type and a tenant identifier; Register the business type and the tenant identifier in the task data as task execution identifiers to the XXL-JOB; When the preset execution conditions are met, all service nodes deployed on the SaaS platform are called to seize the task execution identifier of the XXL-JOB callback, and the service node that seizes the task execution identifier is used as the target service node; The target service node is called to execute the data transmission task.

2. The multi-tenant data transmission method according to claim 1, characterized in that: The calling the target service node to execute the data transmission task includes: Calling the target service node to identify the request type of the data transmission request of the target address in the task data corresponding to the data transmission task; In the case where the request type is identified as a push request, obtaining the data to be pushed corresponding to the push request of the target address, encrypting the data to be pushed according to a preset encryption protocol to obtain target push data, and pushing the target push data to the target address; In the case where it is identified that the request type is a pull request, target pull data is obtained from the target address, the target pull data is decrypted according to the preset encryption protocol to obtain target data, and the target data is stored in the business table of MySQL.

3. The multi-tenant data transmission method according to claim 2, characterized in that: The step of obtaining target pull data from the target address includes: Identify a data pulling mode corresponding to the task configuration identifier of the data transmission task, the data pulling mode includes: a fixed number of pulling mode and a non-fixed cycle empty pulling mode, the fixed number of pulling mode is to pull data from the target address for a preset number of times, and the non-fixed cycle empty pulling mode is to continuously pull data from the target address until the pulling result is empty; Based on the data pull mode, target pull data is acquired from the target address.

4. The multi-tenant data transmission method according to claim 2, characterized in that: The pushing the target push data to the target address includes: Identify a data push mode corresponding to the task configuration identifier of the data transmission task, the data push mode including: a fixed number of push mode and an unfixed cycle push mode, the fixed number of push mode is to push the target push data to the target address for a preset number of times, the unfixed cycle push mode is to continuously push the target push data to the target address until the target push data is empty; Based on the data push mode, the target push data is pushed to the target address.

5. The multi-tenant data transmission method according to claim 2, characterized in that: The acquiring the data to be pushed corresponding to the push request of the target address includes: Querying the business data whose push status is not pushed among all the business data corresponding to the business type as the data to be screened; Converting the to-be-screened data into preset structure data; The fields in the preset structure data that do not conform to the push request are filtered to obtain the data to be pushed.

6. The multi-tenant data transmission method according to claim 5, characterized in that: Also includes: The push status of the data to be pushed in the business data is changed to a pushed status.

7. The multi-tenant data transmission method according to claim 1, characterized in that: The calling of all service nodes deployed on the SaaS platform, seizing the task execution identifier of the XXL-JOB callback, and taking the service node that has seized the task execution identifier as the target service node, includes: Using the task execution identifier as a distributed lock for the data transmission task; Call StringRedisTemplate to control all service nodes deployed on the SaaS platform to seize the distributed lock; The service node that seizes the distributed lock is determined as the target service node.

8. The multi-tenant data transmission method according to claim 1, characterized in that: The preset execution condition at least includes: the XXL-JOB meets the preset timing execution requirement.

9. A multi-tenant data transmission device, characterized in that: The SaaS platform based on SpringBoot is applied to a multi-tenant data transmission system, wherein the multi-tenant data transmission system at least includes: XXL-JOB and MySQL, and the multi-tenant data transmission method includes: A task data acquisition unit, configured to acquire task data of the data transmission task when detecting that there is a newly added data transmission task in the task table of the MySQL, wherein the task data at least includes: a business type and a tenant identifier; An identification registration unit, used to register the business type and the tenant identification in the task data as task execution identification to the XXL-JOB; A service node determination unit is used to call all service nodes deployed on the SaaS platform, seize the task execution identifier of the XXL-JOB callback, and use the service node that seizes the task execution identifier as the target service node when the preset execution condition is met; The service calling unit is used to call the target service node to execute the data transmission task.

10. A multi-tenant data transmission system, characterized in that: include: A SaaS platform built on SpringBoot, XXL-JOB and MySQL, wherein the SaaS platform is used to implement each step of the multi-tenant data transmission method of any one of claims 1-8.

Images