A method and device for distributing monitoring data based on ElasticJob

Abstract

The application discloses a kind of based on ElasticJob's distributed monitoring data method and device, the method includes: in Activiti process engine in calling Zookeeper distributed service, different tasks are assigned to different Zookeeper node, in Zookeeper node, Activiti process engine will create an asynchronous thread to obtain API interface information, in ElasticJob listener, Activiti engine parses API configuration cron expression, obtains the time and the number of cycles specifically required to send network communication, after specified execution time, Activiti engine utilizes timing task job framework Quartz to initiate network communication, call external system service, after obtaining the return data of external system, Activiti engine parses API interface's custom data receiving JavaScript format in ElasticJob listener, and verifies whether the data format and content of external system meet configuration, if data meet the format and content, then directly return result;Otherwise, Activiti engine initiates network communication again, call external system service, until data meet the rule.

Description

Technical Field

[0001] This invention relates to the field of data processing and transmission technology, and in particular to a method and apparatus for distributed data monitoring based on ElasticJob. Background Technology

[0002] When cloud PC users across China use YiFei to apply for additional cloud PC features or collect private cloud deployment requirements, YiFei sends online business data to a third-party work order maintenance system via an API and waits for the third-party system's processing results. When there are many users and data retrieval has a certain latency, this can easily cause system lag. To improve system performance, YiFei needs to distribute its monitoring of external systems and update the business application forms in its own system with the final results.

[0003] The conventional approach to retrieving data from external system services involves obtaining the API interface configuration via an asynchronous thread and then using network communication to call the external system platform. The returned data from the external system platform is then used as the final result. This is a relatively primitive and simple solution, but it cannot handle situations where the returned data does not conform to the API interface format. Another data processing method involves adding a Zookeeper distributed service after the task starts. Zookeeper can shard large batches of tasks, avoiding system lag caused by different tasks executing simultaneously in the same service, effectively improving system performance. However, this method, like the one mentioned above, only calls the external system platform once; once the data is successfully returned, the external system service is considered complete, without considering the latency and variability of external system processing and data. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for distributed data monitoring based on ElasticJob, comprising the following steps:

[0006] Step 1: First, the Zookeeper distributed service is called in the Activiti process engine. Different tasks are assigned to different Zookeeper nodes, which avoids system lag caused by different tasks being executed at the same time in the same service and effectively improves the system's running performance.

[0007] Step 2: In the Zookeeper node, the Activiti process engine will create an asynchronous thread to obtain API interface information. This is done so that the subsequent process of calling external system services will not affect or occupy the existing thread, making full use of server resources. After obtaining the API configuration information, the ElasticJob listener is enabled.

[0008] Step 3: In the ElasticJob listener, the Activiti engine parses the cron expression configured in the API to obtain the specific time and number of loops for sending network communication.

[0009] Step 4: When the specified execution time arrives, the Activiti engine uses the Quartz scheduled task framework to initiate network communication and call external system services.

[0010] Step 5: After receiving the data returned by the external system, the Activiti engine parses the custom data receiving JavaScript format of the API interface in the ElasticJob listener and verifies whether the data format and content of the external system conform to the configuration.

[0011] Step Six: If the data conforms to the predetermined format and content, the result is returned directly; otherwise, the Activiti engine initiates network communication again to call external system services until the data conforms to the rules.

[0012] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, in step four, the Activiti engine makes network communication requests through HttpClient.

[0013] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the step of obtaining API interface information through the asynchronous thread in step two is as follows:

[0014] S1: Send a request to the server to obtain interface information via an asynchronous thread;

[0015] S2: The server retrieves the data stream from the target API;

[0016] S3: Based on the preset correspondence between interface information and data stream fields, extract the target API interface information from specific fields in the target API's data stream.

[0017] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the method for obtaining the data stream of the target API includes:

[0018] Data streams from multiple APIs are obtained through multiple packet capture proxies installed on the devices to which the APIs belong. The data streams of the target API are filtered from the data streams of the multiple APIs based on the address information of the target API.

[0019] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the method for obtaining the data stream of the target API further includes:

[0020] Obtain the interface survey instruction for the target API, wherein the interface survey instruction carries the address information of the target API;

[0021] Based on the address information of the target API, a packet capture proxy program is sent to the device to which the target API belongs, so that the packet capture proxy program is installed on the device.

[0022] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the core elements of the Quartz scheduled task framework mentioned in step four include Scheduler, trigger, and Job.

[0023] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the Tigger element is used to define the scheduling time and execute tasks according to the time rules.

[0024] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the Job is a scheduled task.

[0025] As a preferred embodiment of the distributed data monitoring method based on ElasticJob described in this invention, the Scheduler represents a task scheduler.

[0026] The present invention also provides a distributed data monitoring device based on ElasticJob, comprising:

[0027] A creation unit is used to create process instances and asynchronous threads in the Activiti engine;

[0028] The acquisition unit is used to acquire API interface information through an asynchronous thread;

[0029] The parsing unit is used to parse the cron expression configured by the API to obtain the specific time for sending network communication and the number of loops.

[0030] The judgment unit is used to verify whether the data format and content of the external system conform to the configuration.

[0031] The beneficial effects of this invention are:

[0032] 1. This invention adds a distributed listening mechanism based on ElasticJob. The combination of ElasticJob listener and distributed service avoids system lag caused by different tasks executing simultaneously in the same service, effectively improving the system's operating performance.

[0033] 2. This invention can guarantee the accuracy and timeliness of the final data. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0035] Figure 1 This is a flowchart of a distributed data monitoring method based on ElasticJob according to the present invention. Detailed Implementation

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0038] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0039] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0040] Example 1

[0041] Reference Figure 1 As an embodiment of the present invention, a method for distributed data monitoring based on ElasticJob is provided, which mainly includes the following steps:

[0042] Step 1: When a cloud desktop user applies for additional cloud PC functions, they initiate a work order through the Yifei Low-Code Platform. This work order creates a process instance in the Activiti engine and, based on the sharding results, enters the distributed service Zookeeper. This avoids system lag caused by different tasks executing simultaneously in the same service, effectively improving system performance.

[0043] Step 2: The Activiti process engine starts an asynchronous thread in the Zookeeper node to obtain the configuration information of the API interface and enables the ElasticJob listener, which then obtains the specific configuration information.

[0044] Step 3: In the ElasticJob listener, the Activiti engine parses the cron expression configured in the ElasticJob API to obtain the specific time for sending network communication and the loop condition;

[0045] Step 4: After the specified execution time is reached, the Activiti process engine uses the mature scheduled task framework Quartz to execute the API interface scheduling task, and uses HttpClient to make network communication requests, sending the cloud desktop application data in combination with the API interface to the work order maintenance system.

[0046] Step 5: After receiving the data returned by the work order maintenance system, the Activiti engine parses the custom data receiving JavaScript format of the API interface in the ElasticJob listener and verifies whether the data format and content of the work order maintenance system conform to the configuration.

[0047] Step Six: After receiving the data returned by the work order maintenance system, the Activiti engine parses the custom data receiving JavaScript format of the API interface in the ElasticJob listener and verifies whether the data format and content of the work order maintenance system conform to the configuration.

[0048] Specifically, in step four, the Activiti engine makes network communication requests through HttpClient.

[0049] Specifically, the step in step two, obtaining API interface information through the asynchronous thread, is as follows:

[0050] S1: Send a request to the server to obtain interface information via an asynchronous thread;

[0051] S2: The server retrieves the data stream from the target API;

[0052] S3: Based on the preset correspondence between interface information and data stream fields, extract the target API interface information from specific fields in the target API's data stream.

[0053] The method for obtaining the data stream of the target API includes:

[0054] Data streams from multiple APIs are obtained through multiple packet capture proxies installed on the devices to which the APIs belong. The data streams of the target API are filtered from the data streams of the multiple APIs based on the address information of the target API.

[0055] The method for obtaining the data stream of the target API also includes:

[0056] Obtain the interface survey instruction for the target API, wherein the interface survey instruction carries the address information of the target API;

[0057] Based on the address information of the target API, a packet capture proxy program is sent to the device to which the target API belongs, so that the packet capture proxy program is installed on the device.

[0058] Specifically, the core elements of the Quartz timed task framework mentioned in step four include Scheduler, trigger, and Job. The trigger element is used to define the scheduling time and execute tasks according to the time rules. The Job is the scheduled task, and the Scheduler represents the task scheduler.

[0059] In summary, this invention can be applied to scenarios where cloud PC users in various regions of China apply for additional cloud PC functions using YiFei, and YiFei sends online business data to a third-party work order maintenance system via an interface, awaiting the processing results from the third-party system. This invention allows YiFei to continuously query external system services after a specific time, updating the final result data to the business application form within the system, making it perfectly suitable for business scenarios with high data latency. Furthermore, this invention utilizes distributed services to solve the high concurrency problem of large-scale tasks, avoiding system lag caused by different tasks executing simultaneously in the same service, and effectively improving system performance.

[0060] The present invention also provides a distributed data monitoring device based on ElasticJob, comprising:

[0061] A creation unit is used to create process instances and asynchronous threads in the Activiti engine;

[0062] The acquisition unit is used to acquire API interface information through an asynchronous thread;

[0063] The parsing unit is used to parse the cron expression configured by the API to obtain the specific time for sending network communication and the number of loops.

[0064] The judgment unit is used to verify whether the data format and content of the external system conform to the configuration.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for distributed data monitoring based on ElasticJob, characterized in that, Includes the following steps: Step 1: First, the Zookeeper distributed service is called in the Activiti process engine, and different tasks are assigned to different Zookeeper nodes; Step 2: In the Zookeeper node, the Activiti process engine will create an asynchronous thread to obtain API interface information. After obtaining the API configuration information, it will enable the ElasticJob listener. Step 3: In the ElasticJob listener, the Activiti engine parses the cron expression configured in the API to obtain the specific time and number of loops for sending network communication. Step 4: When the specified execution time arrives, the Activiti engine uses the Quartz scheduled task framework to initiate network communication and call external system services. Step 5: After receiving the data returned by the external system, the Activiti engine parses the custom data receiving JavaScript format of the API interface in the ElasticJob listener and verifies whether the data format and content of the external system conform to the configuration. Step Six: If the data conforms to the predetermined format and content, the result is returned directly; otherwise, the Activiti engine initiates network communication again to call external system services until the data conforms to the rules.

2. The method for distributed data monitoring based on ElasticJob as described in claim 1, characterized in that: In step four, the Activiti engine makes network communication requests through HttpClient.

3. The method for distributed data monitoring based on ElasticJob as described in claim 1, characterized in that: The step in step two, which involves obtaining API interface information through the asynchronous thread, is as follows: S1: Send a request to the server to obtain interface information via an asynchronous thread; S2: The server retrieves the data stream from the target API; S3: Based on the preset correspondence between interface information and data stream fields, extract the target API interface information from specific fields in the target API's data stream.

4. The method for distributed data monitoring based on ElasticJob as described in claim 3, characterized in that: The method for obtaining the data stream of the target API includes: Data streams from multiple APIs are obtained through multiple packet capture proxies installed on the devices to which the APIs belong. The data streams of the target API are filtered from the data streams of the multiple APIs based on the address information of the target API.

5. The method for distributed data monitoring based on ElasticJob as described in claim 4, characterized in that: The method for obtaining the data stream of the target API also includes: Obtain the interface survey instruction for the target API, wherein the interface survey instruction carries the address information of the target API; Based on the address information of the target API, a packet capture proxy program is sent to the device to which the target API belongs, so that the packet capture proxy program is installed on the device.

6. The method for distributed data monitoring based on ElasticJob as described in claim 1, characterized in that: The core elements of the Quartz scheduled task framework described in step four include Scheduler, trigger, and Job.

7. The method for distributed data monitoring based on ElasticJob as described in claim 6, characterized in that: The Tigger element is used to define the scheduling time and execute tasks according to the time rules.

8. The method for distributed data monitoring based on ElasticJob as described in claim 6, characterized in that: The Job is the scheduled task.

9. The method for distributed data monitoring based on ElasticJob as described in claim 6, characterized in that: The Scheduler refers to the task scheduler.

10. A distributed data monitoring device based on ElasticJob as described in any one of claims 1-9, characterized in that, include: A creation unit is used to create process instances and asynchronous threads in the Activiti engine; The acquisition unit is used to acquire API interface information through an asynchronous thread; The parsing unit is used to parse the cron expression configured by the API to obtain the specific time for sending network communication and the number of loops. The judgment unit is used to verify whether the data format and content of the external system conform to the configuration.

Images