A data collection method based on multi-tenant cache feedback mechanism

Abstract

The present invention relates to a data collection method based on a multi-tenant cache feedback mechanism. The method uses a cache to replace a database as a medium for data exchange. The data collection service stores the collected external real-time data in a format that can be recognized by the business after internal processing. The dataIsNull value in the cache, that is, the validity or expiration of the feedback value, can be used to flexibly control the opening and closing of different tenant data collection functions through a scheduling project, which not only ensures the stability of the system under multi-tenant use, but also adjusts the occupation of resources according to actual conditions. At the same time, due to the special relationship between the scheduling project and the collection project, when the data collection of the system reaches a performance bottleneck, the performance of data collection can be improved by dynamically adding collection projects.

Description

Technical Field

[0001] The invention relates to a data collection method based on a multi-tenant cache feedback mechanism. Background Art

[0002] In software development, there is such a scenario: some of the data in the system comes from external systems or external data sources, and most of this data is real-time data, such as the user's current status, the user's online time, and the user's real-time data statistics. This system needs to continuously poll the data interface of the external system through the interface at a certain frequency to obtain the latest real-time data and associate it with the data of this system for display. This data collection scenario needs to support multi-tenant form, that is, the data that different user organizations need to collect are isolated and different from each other. In this scenario, how to reduce the access pressure on the external system interface and reduce the processing pressure of the system on the collected data is the problem to be solved by the present invention.

[0003] There are two main existing data collection solutions:

[0004] 1. Collect data through client polling

[0005] This collection method mainly starts from the client's access to the data display page, such as Figure 1 .

[0006] When the user needs to view real-time data, the corresponding function page in the system will be opened. When the page is loaded, a timer with a certain frequency will be preset. The timer will repeatedly trigger http requests to the backend. In the http request, user information and other parameter identifiers used to distinguish different tenants and parameter identifiers of the requested data type will be carried. When the request reaches the backend business engineering, the backend business engineering service will send all the received parameter identifiers to the request input parameters of the external system interface to request the external interface. After obtaining the return output parameters of the external interface, they will be processed in the business engineering and returned to the front-end client for data display.

[0007] The advantages of this solution are that it is easy to implement and can distinguish the collected data of different tenants very well, but the disadvantages are also obvious, mainly including the following points.

[0008] Business engineering and data collection services are concentrated together: In this solution, data collection services and other business services in the system are concentrated in one project, which causes mutual influence between data collection and other business services. When the collection frequency is high and the data processing pressure increases, it is very likely to affect the normal operation of other business services in the system.

[0009] The concurrent pressure on the external system interface is relatively high: This solution takes the user opening the data viewing page as the starting point, and there are multiple tenants in the scenario. Assuming that each tenant has N users opening the data viewing page, the simultaneous polling volume of the back-end business project is equal to the number of tenants * N. Similarly, the simultaneous access volume to the external interface will reach the same level. In this way, as the number of tenants and the number of users opening the data viewing page increases, the back-end business service project and the external system interface of this system will eventually be overwhelmed or even crash.

[0010] 2. Independent collection project

[0011] The main idea of ​​this solution is to separate the business logic responsible for data collection from the back-end business engineering and provide independent services, such as Figure 2 .

[0012] The data collection service is established as a separate project, which is responsible for data polling of the external system interface to collect data in real time. This data collection project can be single-threaded or multi-threaded, and uses thread polling to collect all real-time data in the system. After data processing in this collection project, it is stored in the business database. The data collection project will continuously repeat this process so that the data stored in the business database is the latest real-time data.

[0013] The client still takes the user opening the data viewing page as the starting point, and continuously submits requests carrying parameters to the back-end service through the page timer. After the back-end service receives the request parameters, it queries the real-time data required by the user from the database based on the parameters, and returns it to the front-end page for display.

[0014] The main advantage of this solution is that it separates data collection projects from business projects, reducing the data processing pressure of business projects. At the same time, it makes data collection and business services independent of each other and reduces the access pressure to external system interfaces. However, this solution also has its shortcomings.

[0015] Frequent read and write operations on the database: This solution relies on the database as a medium to store and read data, and the data collection project initiates data collection requests to the external interface in a polling manner, so there will be continuous updates and storage operations of new real-time data in the database. At the same time, the client requests of the page will also continuously initiate read operation requests to the database to obtain the latest real-time user data. Therefore, in this case, the data reading and writing pressure is relatively high.

[0016] In multi-tenant scenarios, there is useless data collection: the user's demand for viewing real-time data often starts with the user opening the data viewing page. However, if no user opens the data viewing page in the current period under a tenant, that is, the tenant has no demand for viewing real-time data in the current period. However, in this solution, if the data collection project collects real-time data from the entire platform in a multi-threaded manner, there will be a situation where the data collected by some threads does not have the current display demand, that is, the collected data is useless in the current situation. Therefore, this solution wastes server computing resources to some extent. If the number of tenants on the platform is not large, then this waste can be accepted, but if the number of tenants on the platform reaches a certain level, computing resources become particularly precious.

[0017] Since the current traditional data collection methods have various drawbacks, the present invention proposes a cache-based data collection method in a multi-tenant scenario. The core of this method is to introduce scheduling engineering and perform dynamic scheduling of resources based on cache. Summary of the invention

[0018] The purpose of the present invention is to reduce the access pressure on the external system interface and reduce the processing pressure of the system on the collected data. Therefore, a data collection method based on a multi-tenant cache feedback mechanism is provided.

[0019] To achieve the above object, the technical solution of the present invention is: a data collection method based on a multi-tenant cache feedback mechanism, using cache instead of database as a medium for data exchange, the data collection service processes the collected external real-time data internally and stores it in the cache in a format that can be recognized by the business;

[0020] The user starts to view real-time data by opening the data viewing page, and polls the backend business project through the page timer. The destination accessed by the backend business project at this time is the cache. During the first access, there is no real-time data value required by the user in the cache, which triggers the storage of an identifier indicating that the data is empty in the cache. When the scheduling project queries the cache and finds that an identifier indicating that the data is empty is generated, it starts the data collection service according to the tenant parameters attached to the identifier to collect data for this tenant.

[0021] When the tenant's data collection service is started by the scheduling project, the data collection will poll the external data interface, continuously collect the real-time data of this tenant, and store it in the cache after internal processing. At this time, the front-end page that accesses the real-time data through page polling can obtain the first batch of data from the cache, and subsequent real-time data updates are also obtained through the cache;

[0022] When the user closes the data viewing page for a period of time, the mark that the data in the cache is empty will become invalid. After the scheduling project queries it, it will schedule the closure of the data collection service under this tenant to dynamically save server computing resources.

[0023] In one embodiment of the present invention, the cache component used by the cache is Redis that supports Key-Value storage.

[0024] In one embodiment of the present invention, the method is specifically implemented as follows:

[0025] S1. Initialization of acquisition project:

[0026] The number of collection projects is flexibly deployed according to the number of tenants and the amount of real-time data collection. The collection project is configured with the dispatching project address, which is used to register with the dispatching project when the collection project is initialized. At the same time, the collection project also provides a dispatching interface for the dispatching project to call. Through the dispatching interface, the opening and closing of the data collection thread in the corresponding collection project is controlled, and the working status data of the collection project is returned to the dispatching project for monitoring and display.

[0027] S2. Scheduling project initialization:

[0028] A registration interface is provided in the scheduling project, which is used to automatically register the data collection project to the scheduling project when adding the data collection project, so that it can be scheduled by the scheduling project;

[0029] When the scheduling project is started, the registered collection projects are read from the database into the memory, and the configuration information in the database is maintained according to the actual registration status when the collection project is started;

[0030] S3. Data collection:

[0031] S31. When a user opens a page for viewing real-time data, a timer of the page is loaded in JavaScript during the page loading process. The timer initiates a polling request to the backend business engineering at the data update frequency required by the user. The parameters carried in the request include: company_id, staff_id;

[0032] S32. After receiving the request from the page, the business engineering end first queries whether there is data with company_id as the key value in Redis. If there is, the data is directly returned to the data viewing page for parsing and display, and the expiration time of the key named dataIsNull+company_id is reset to T1; if no data with company_id as the key value is found in Redis, dataIsNull+company_id is used as the key value and company_id is the value value, and stored in the Redis cache. At the same time, the expiration time of the key is set to T1;

[0033] S33, the scheduling project will query Redis at frequency T2 to obtain a data list whose key is dataIsNull*, and compare it with the dataIsNull list stored in the memory to find out the data differences;

[0034] S34. After the scheduling project compares the dataIsNull feedback value retrieved from Redis with the dataIsNull value in its memory, if it is found that the newly retrieved dataIsNull value has increased, it selects a collection project to run the new data collection requirement according to the allocation algorithm based on the information of the collection project address and number of threads maintained in its memory. When a collection project is selected, the scheduling interface provided by this collection project is used to control the collection thread to be started inside the collection project, thereby continuously polling the external interface to obtain the latest real-time data under the current tenant; after the scheduling project successfully calls the scheduling interface provided by the collection project, it updates the corresponding internal collection project real-time data. The updated data includes: the current collection project The number of used threads and tenant thread information; if the retrieved dataIsNull value is found to be reduced, that is, the Key value of dataIsNull+company_id in Redis has expired, it means that the user has closed or exited the data viewing page, which interrupts the page's polling of Redis, and the expired value of the corresponding Key cannot be maintained, and then the feedback value of this tenant in Redis disappears; at this time, the scheduling project controls the collection project to close the data collection thread of the current tenant through the scheduling interface provided by the collection project to save server computing resources. After the scheduling interface is successfully called, the scheduling project will update the corresponding internal collection project real-time data. The updated data includes: the number of used threads of the current collection project and tenant thread information;

[0035] S35. After the scheduling of step S34, the corresponding data collection thread will be started or closed. The started collection thread will continuously poll the external interface to obtain real-time data. After its internal processing, the collection thread will store the processed data under the Key value corresponding to the company_id in step S31 for real-time dynamic display after the client page obtains it.

[0036] In one embodiment of the present invention, the collection project registration content includes: the collection project name, the initial total number of threads, and the number of used threads.

[0037] In one embodiment of the present invention, after the collection project is deployed, when the collection project is started, the initial information is registered in the scheduling project through the configured scheduling project address and the registration interface provided by the scheduling project, so that it can be scheduled by the scheduling project.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. Fast data access

[0040] Using cache as a medium for data exchange has the advantage of fast access speed compared to database. At the same time, exchanging data in cache can greatly reduce the IO pressure of data. In multi-tenant scenarios, when the number of tenants is large, if the database is used directly for data exchange, the database will have more frequent IO operations. If the concurrency is high, the database may even be blocked. Since the operating principles of cache and database are different, the data is accessed in memory. Therefore, in the face of high-concurrency multi-tenant scenarios, cache can better adapt.

[0041] 2. Dynamic resource scheduling

[0042] Since the scheduling project has been introduced, for tenants who currently do not need to view real-time data, the scheduling project will query Redis at a certain frequency and synchronize the data viewing identifier. If it is found that the user currently has no need to view real-time data, the data collection project will be scheduled to stop collecting real-time data for this tenant to save server computing resources. Similarly, if it is found that a tenant has the need to view real-time data at this time, the data collection service related to this tenant will be started to provide real-time data collection services for this tenant. In this way, dynamic scheduling of server resources is realized and computing resources are used rationally.

[0043] 3. Reduce external interface access pressure

[0044] According to the idea of ​​this method, since the access to the external interface is performed on a tenant basis, the access pressure on the external interface has been greatly reduced compared to the traditional solution 1, which performs access on a user basis.

[0045] 4. The data acquisition module can be dynamically expanded and reduced

[0046] In this method, data collection projects can support the deployment of multiple data collection projects, and the address of the scheduling project is configured in multiple data collection projects. At the same time, the address of each data collection project is also configured in the scheduling project. In this way, when the number of tenants in the platform increases and the original data collection module exceeds the load, the number of data collection modules can be increased by copying the project to improve the data collection capability. On the contrary, if the number of tenants decreases, the server resources can be avoided by removing the redundant data collection modules.

[0047] 5. The collection situation can be monitored

[0048] In this method, since there is a dynamic scheduling relationship between the scheduling project and the data acquisition module, the scheduling project can obtain the internal data of the data acquisition module through the interaction between the scheduling project and the data acquisition module, thereby realizing the monitoring of the data acquisition module. The monitored data mainly include the number of threads, the amount of collected data, the collection time, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a system architecture diagram for existing collection through client polling.

[0050] Figure 2 This is the system architecture diagram of the existing independent acquisition project.

[0051] Figure 3 This is a system architecture diagram of the data collection method based on the multi-tenant cache feedback mechanism of the present invention. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0053] like Figure 3 As shown, the present invention provides a data collection method based on a multi-tenant cache feedback mechanism, in which a cache is used instead of a database as a medium for data exchange. The data collection service processes the collected external real-time data internally and stores the data in a format that can be recognized by the business.

[0054] The user starts to view real-time data by opening the data viewing page, and polls the backend business project through the page timer. The destination accessed by the backend business project at this time is the cache. During the first access, there is no real-time data value required by the user in the cache, which triggers the storage of an identifier indicating that the data is empty in the cache. When the scheduling project queries the cache and finds that an identifier indicating that the data is empty is generated, it starts the data collection service according to the tenant parameters attached to the identifier to collect data for this tenant.

[0055] When the tenant's data collection service is started by the scheduling project, the data collection will poll the external data interface, continuously collect the real-time data of this tenant, and store it in the cache after internal processing. At this time, the front-end page that accesses the real-time data through page polling can obtain the first batch of data from the cache, and subsequent real-time data updates are also obtained through the cache;

[0056] When the user closes the data viewing page for a period of time, the mark that the data in the cache is empty will become invalid. After the scheduling project queries it, it will schedule the closure of the data collection service under this tenant to dynamically save server computing resources.

[0057] The following is the specific implementation process of the present invention.

[0058] The present invention provides a data collection method based on a multi-tenant cache feedback mechanism, which is specifically implemented as follows:

[0059] Initialize the acquisition project

[0060] The number of collection projects can be flexibly deployed according to the number of tenants and the amount of real-time data collection. The fixed address of the scheduling project is configured in the collection project, which is used to register with the scheduling project when the collection project is initialized. The main contents of the registration are: the name of the collection project, the initial total number of threads, the number of used threads, etc. At the same time, the collection project also provides a scheduling interface for the scheduling project to call. Through the scheduling interface, the opening and closing of the data collection thread in this collection project can be controlled, and the working status data of the collection project can be returned to the scheduling project for monitoring and display.

[0061] After the collection project is deployed, when the project is started, its initial information will be registered to the scheduling project through the scheduling project address configured by itself and the registration interface provided by the scheduling project, so that the scheduling project knows its existence and can be scheduled by the scheduling project.

[0062] Scheduling project initialization

[0063] A registration interface is provided in the scheduling project, which is used to automatically register the data collection project to the scheduling project when adding the data collection project, so that it can be scheduled by the scheduling project.

[0064] When the scheduling project is started, the registered data collection project is read from the database into the memory, and the configuration information in the database is maintained according to the actual registration status when the data collection project is started.

[0065] After the collection project and the scheduling project are initialized, the main steps to achieve are as follows:

[0066] 1. When a user opens a page for viewing real-time data, the page timer will be loaded in JavaScript during the page loading process. The timer will initiate a polling request to the backend business project at the data update frequency required by the user. The main parameters carried in the request include: company_id, staff_id, etc.

[0067] 2. After receiving the request from the page, the business engineering end first queries whether there is data with its company_id as the key value in Redis. If there is, the data is directly returned to the front-end page for parsing and display, and the expiration time of the key named dataIsNull+company_id is reset to T1. If no data with company_id as the key value is found in Redis, dataIsNull+company_id is used as the key value, and company_id is the value value, and stored in the Redis cache. At the same time, the expiration time of the key is set to T1.

[0068] 3. The scheduling project will query Redis at frequency T2 to obtain a data list whose key is dataIsNull*, and compare it with the dataIsNull list stored in the memory to find out the data differences.

[0069] 4. The scheduling project compares the dataIsNull feedback value retrieved from Redis with the dataIsNull value in its memory. If the newly retrieved dataIsNull value is found to have increased, it selects a collection project to run the new data collection requirement based on the collection project address, number of threads and other information maintained in its memory, and according to the allocation algorithm. When a collection project is selected, the scheduling interface provided by this collection project is used to control the collection thread to be started inside the collection project, so as to continuously poll the external interface to obtain the latest real-time data under the current tenant. After the scheduling project successfully calls the scheduling interface provided by the collection project, it updates the corresponding real-time data of the internal collection project. The updated data mainly includes: the number of threads used by the current collection project, tenant thread information, etc.

[0070] 5. Similar to the fourth point, if the retrieved dataIsNull value is found to be reduced, that is, the Key value of dataIsNull+company_id in Redis has expired, in this case, it indicates that the user has closed or exited the data viewing page, which interrupts the page's polling of Redis, and the expired value of the corresponding Key cannot be maintained, and then the feedback value of this tenant in Redis disappears. At this time, the scheduling project controls the collection project to close the data collection thread of the current tenant through the scheduling interface provided by the collection project to save server computing resources. After the scheduling interface is successfully called, the scheduling project will update the corresponding internal collection project real-time data. The updated data mainly includes the number of threads used by the current collection project, tenant thread information, etc.

[0071] 6. After the scheduling of points 4 and 5, the corresponding data collection thread will be started or closed. The started collection thread will continuously poll the external interface to obtain real-time data. After its internal processing, the collection thread will store the processed data under the Key value corresponding to company_id in point 1 for real-time dynamic display after the client page obtains it.

[0072] The above are the main implementation steps of a data collection method based on a multi-tenant cache feedback mechanism. Through this method, the dataIsNull value in the cache, that is, the validity or expiration of the feedback value, can be used to flexibly control the opening and closing of different tenant data collection functions through the scheduling project, which not only ensures the stability of the system under multi-tenant use, but also adjusts the resource occupancy according to actual conditions. At the same time, due to the special relationship between the scheduling project and the collection project, when the system data collection reaches a performance bottleneck, the data collection performance can be improved by dynamically adding collection projects.

[0073] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A data collection method based on a multi-tenant cache feedback mechanism, characterized in that: Use cache instead of database as the medium for data exchange. The data collection service processes the collected external real-time data internally and stores it in the cache in a format that the business can recognize. The user starts to view real-time data by opening the data viewing page, and polls the backend business project through the page timer. The destination accessed by the backend business project at this time is the cache. During the first access, there is no real-time data value required by the user in the cache, which triggers the storage of an identifier indicating that the data is empty in the cache. When the scheduling project queries the cache and finds that an identifier indicating that the data is empty is generated, it starts the data collection service according to the tenant parameters attached to the identifier to collect data for this tenant. When the tenant's data collection service is started by the scheduling project, the data collection will poll the external data interface, continuously collect the real-time data of this tenant, and store it in the cache after internal processing. At this time, the front-end page that accesses the real-time data through page polling can obtain the first batch of data from the cache, and subsequent real-time data updates are also obtained through the cache; When the user closes the data viewing page for a period of time, the mark that the data in the cache is empty will become invalid. After the scheduling project queries it, it will schedule the closure of the data collection service under this tenant to dynamically save server computing resources.

2. According to the data collection method based on the multi-tenant cache feedback mechanism of claim 1, it is characterized in that: The cache component used by the cache is Redis which supports Key-Value storage.

3. According to the data collection method based on the multi-tenant cache feedback mechanism of claim 1, it is characterized in that: The specific implementation of this method is as follows: S1. Initialization of acquisition project: The number of collection projects is flexibly deployed according to the number of tenants and the amount of real-time data collection. The collection project is configured with the dispatching project address, which is used to register with the dispatching project when the collection project is initialized. At the same time, the collection project also provides a dispatching interface for the dispatching project to call. Through the dispatching interface, the opening and closing of the data collection thread in the corresponding collection project is controlled, and the working status data of the collection project is returned to the dispatching project for monitoring and display. S2. Scheduling project initialization: A registration interface is provided in the scheduling project, which is used to automatically register the data collection project to the scheduling project when adding the data collection project, so that it can be scheduled by the scheduling project; When the scheduling project is started, the registered collection projects are read from the database into the memory, and the configuration information in the database is maintained according to the actual registration status when the collection project is started; S3. Data collection: S31. When a user opens a page for viewing real-time data, a timer of the page is loaded in JavaScript during the page loading process. The timer initiates a polling request to the backend business engineering at the data update frequency required by the user. The parameters carried in the request include: company_id, staff_id; S32. After receiving the request from the page, the business engineering end first queries whether there is data with company_id as the key value in Redis. If there is, the data is directly returned to the data viewing page for parsing and display, and the expiration time of the key named dataIsNull+company_id is reset to T1; if no data with company_id as the key value is found in Redis, dataIsNull+company_id is used as the key value and company_id is the value value, and stored in the Redis cache. At the same time, the expiration time of the key is set to T1; S33, the scheduling project will query Redis at frequency T2 to obtain a data list whose key is dataIsNull*, and compare it with the dataIsNull list stored in the memory to find out the data differences; S34. After the scheduling project compares the dataIsNull feedback value retrieved from Redis with the dataIsNull value in its memory, if it is found that the newly retrieved dataIsNull value has increased, it selects a collection project to run the new data collection requirement according to the allocation algorithm based on the information of the collection project address and number of threads maintained in its memory. When a collection project is selected, the scheduling interface provided by this collection project is used to control the collection thread to be started inside the collection project, thereby continuously polling the external interface to obtain the latest real-time data under the current tenant; after the scheduling project successfully calls the scheduling interface provided by the collection project, it updates the corresponding internal collection project real-time data. The updated data includes: the current collection project The number of used threads and tenant thread information; if the retrieved dataIsNull value is found to be reduced, that is, the Key value of dataIsNull+company_id in Redis has expired, it means that the user has closed or exited the data viewing page, which interrupts the page's polling of Redis, and the expired value of the corresponding Key cannot be maintained, and then the feedback value of this tenant in Redis disappears; at this time, the scheduling project controls the collection project to close the data collection thread of the current tenant through the scheduling interface provided by the collection project to save server computing resources. After the scheduling interface is successfully called, the scheduling project will update the corresponding internal collection project real-time data. The updated data includes: the number of used threads of the current collection project and tenant thread information; S35. After the scheduling of step S34, the corresponding data collection thread will be started or closed. The started collection thread will continuously poll the external interface to obtain real-time data. After its internal processing, the collection thread will store the processed data under the Key value corresponding to the company_id in step S31 for real-time dynamic display after the client page obtains it.

4. The data collection method based on the multi-tenant cache feedback mechanism according to claim 3 is characterized in that: The registration content of the collection project includes: collection project name, initial total number of threads, and number of used threads.

5. The data collection method based on the multi-tenant cache feedback mechanism according to claim 3 is characterized in that: After the collection project is deployed, when the collection project is started, the initial information will be registered to the scheduling project through the configured scheduling project address and the registration interface provided by the scheduling project so that it can be scheduled by the scheduling project.

Images