A data subscription registration and cancellation system and method based on distributed images

Abstract

The present invention discloses a data subscription registration and cancellation system and method based on distributed screens. The screen subscription adopts a distributed registration and cancellation subscription scheme, which improves the subscription performance and efficiency of the screens to cope with high-concurrency scenarios. Then, filtering is performed for the same query conditions to reduce the duplicate BI analysis calculation resources, avoid duplicate data pushing, and improve the pushing efficiency of job information. It relies on the message queue mechanism for data interaction and synchronization, including registration messages, job dispatching messages, and job pushing data; through a weakly centralized scheduler cluster, the scheduling work of other schedulers is not affected after the master node fails; the job data is stored in a decentralized and distributed manner in the JVM memory of each node to avoid the performance bottleneck of database access.

Description

Technical Field

[0001] The present invention relates to the technical field of data cloud storage, and in particular to a data subscription registration and cancellation system and method based on distributed screens. Background Art

[0002] Improved based on the existing visualization system, the main defect of the existing visualization system is that the existing technology only supports large-screen clients, does not support multiple terminals, and there is no filtering device for the data set, resulting in low efficiency and performance of visualization. Summary of the Invention

[0003] The purpose of the present invention is to provide a data subscription registration and cancellation system and method based on distributed screens, so as to solve the foregoing problems existing in the prior art.

[0004] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] A data subscription registration and cancellation system and method based on distributed screens, including a plurality of data visualization clients and / or three-dimensional visualization clients, a task scheduling cluster, a distributed message queue, and a data integration server,

[0006] The data visualization client and the three-dimensional visualization client can initiate screen subscription registration or cancellation information and then transmit it to the task scheduling cluster;

[0007] The task scheduling cluster includes a plurality of schedulers and executors, which regularly send information to the distributed message queue to inform its status; obtain job information, and send the obtained job information to the data integration server;

[0008] The data integration server receives the job information transmitted by the task scheduling cluster, and after initializing and conditionally filtering the job information, transmits it to the data visualization server and / or three-dimensional visualization client through the distributed message queue;

[0009] The data visualization client is used to receive the job information transmitted by the distributed message queue, and design and display the screen subscription registration and cancellation functions of the job information;

[0010] The three-dimensional visualization client is used to receive the job information transmitted by the distributed message queue and display the screen registration subscription content.

[0011] Preferably, the system further includes a memory cache and a database. The memory cache is an open-source key-value storage system, mostly used as a cache, and can be deployed in a cluster mode; the database is used to store the logical relationships of several visualization clients.

[0012] Preferably, the data visualization client includes a DVP micro front-end, a DVP microservice layer, and a storage layer. The DVP micro front-end corresponds and connects the screen designer and the display to the API interface in the DVP microservice layer through a logic layer. After the object processing of the data screen by the API interface layer, the data is stored in the storage layer.

[0013] Preferably, the data integration server includes a data query and mapping module, a data push module, and a data initialization module. The data query and mapping module is connected to the data initialization module and the data push module respectively. The data query and mapping module includes a condition filter to query and obtain the data information content, and uses the condition filter to filter out the redundant data information conditions, and then transmits them to the data push module and the data initialization module. The data initialization module initializes the queried data and stores it in the cache. The data push module can push the data from the data query and mapping module and the cache to the distributed message queue.

[0014] Another object of the present invention is to provide a method for data subscription registration and cancellation based on distributed screens, including the following steps:

[0015] S1, based on the subscriber forwarding center on several data visualization clients and / or 3D visualization clients, send the job information of the subscriber to the distributed message queue through the subscriber data push center;

[0016] S2, the distributed message queue autonomously distributes the received job information to the schedulable executors in the task scheduling cluster;

[0017] S3, after receiving the allocated job information, the schedulable executor immediately informs other scheduling executors of its own status through the distributed message queue, and at the same time sends the job information to the data integration server and pushes the screen data to the corresponding screen.

[0018] Preferably, the job information includes registration information and cancellation information, and the registration information and cancellation information include data screen information, control information, model information, data set information, and subscription frequency information;

[0019] Step S1 further includes condition filtering: according to the data screen information, control information, model information, data set information, and subscription frequency information in the job information, de-duplicate and filter the job information.

[0020] Preferably, between steps S1 and S2, it further includes: all schedulers and executors need to send their registration messages to the specified queue immediately when joining the task scheduling cluster to inform other nodes of their existence; specifically: after the scheduler is successfully registered, the main scheduler will dispatch scheduling tasks and synchronize jobs; after the executor is successfully registered, the main scheduler synchronizes all jobs, and all schedulers add information about schedulable executors.

[0021] Preferably, in the scheduler cluster, there is one master and multiple slaves, and the election method of the master node is preemptive. It is necessary to regularly reset the Leader lock identifier of Redis. The Leader lock identifier expires within 10 seconds, and its value is the unique Hash identifier of the scheduler node. The scheduler that preempts this resource lock first is the main scheduler.

[0022] Preferably, when there are operations of adding, deleting, modifying, or querying jobs, the scheduler will send a message to the specified message queue, and all nodes will consume and synchronize the incremental jobs to achieve job consistency.

[0023] Preferably, the scheduler determines which message queue to send the message to and triggers the timing of the decision to send the message; the "all nodes" refers to the executor nodes and schedulers that cooperate with each other to complete the entire scheduling execution process.

[0024] Preferably, the data pusher numbers the query conditions provided by the data visualization client, numbers the duplicate query conditions the same, and differentiates them according to the screen instance id; then after querying the results according to the query conditions, the query results are pushed to the data visualization client screen corresponding to the corresponding screen instance id.

[0025] The beneficial effects of the present invention are:

[0026] The present invention discloses a data subscription registration and cancellation system and method based on distributed screens. For the screen subscriptions of multiple clients, a distributed registration and cancellation subscription scheme is adopted to improve the subscription performance and efficiency of the data visualization client screens to cope with the high-concurrency scenarios of multiple clients. Then, filtering is performed on the same query conditions to avoid duplicate data pushing and improve the pushing efficiency of job information. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is the system architecture diagram provided in Embodiment 1;

[0028] Figure 2 is the DVP composition architecture diagram provided in Embodiment 1;

[0029] Figure 3 is the schematic diagram of the data subscription and publishing architecture design;

[0030] Figure 4It is a schematic diagram of the principle of the conditional filter;

[0031] Figure 5 It is a schematic diagram of the principle of the method for screen subscription registration and cancellation of the data subscription device provided in Embodiment 2. Detailed implementation manners

[0032] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] Term explanation:

[0034] Screen: The control that requires data;

[0035] Query condition: A series of data filtering parameters that can be set on the screen;

[0036] Forwarding center: Receives the data from the data pusher and then forwards it to the required screen;

[0037] Pusher: Queries data from the data center according to the query conditions provided by the query device, and pushes the results to the forwarding center;

[0038] Query device: Mainly caches the query conditions and scheduling plans, and then triggers the pusher to work through the scheduler;

[0039] Scheduler: A system responsible for controlling job dispatching and timed execution;

[0040] Executor: A system responsible for executing jobs, including multiple job processors, corresponding to the pusher;

[0041] CRUD (Create, Read, Update, Delete): The four basic functions of adding, deleting, querying, and modifying job tasks;

[0042] IO: Input / Output

[0043] JVM: Java Virtual Machine

[0044] DVP: Data Visualization Client

[0045] DIP: Data Integration Server

[0046] 3D: 3D Visualization Client

[0047] BI: Business Computing Analysis Service

[0048] Embodiment 1

[0049] This embodiment provides a data subscription registration and cancellation system based on distributed screens, as Figure 1 shown, which includes several data visualization clients and / or 3D visualization clients, a task scheduling cluster, a distributed message queue, and a data integration server.

[0050] The task scheduling cluster includes several schedulers and executors, which regularly send information to the distributed message queue to inform its status; obtain job information, and send the obtained job information to the data integration server.

[0051] The distributed message queue in this embodiment is used to implement data interaction including heartbeat, full job loading, full job synchronization, job dispatching, and job scheduling.

[0052] The schedulers and executors of the task scheduling cluster can be used as nodes, which regularly send information to the distributed message queue to inform its status; obtain job information, and sending the obtained job information to the data integration server specifically includes: the node regularly sends a heartbeat to the specified distributed message queue to inform other nodes of its survival status; when the node joins the task scheduling cluster, it sends a registration message to the specified distributed message queue immediately to inform other nodes of its existence; after the scheduler registration is successful, the main scheduler dispatches scheduling tasks and synchronizes jobs; after the executor registration is successful, the main scheduler synchronizes the full amount of jobs, and all schedulers add information about the schedulable executors.

[0053] The data integration server in this embodiment includes a data query mapping module, a data push module, and a data initialization module. The data query mapping module is respectively connected to the data initialization module and the data push module. The data query mapping module includes a condition filter, queries and obtains the data information content, filters the redundant data information conditions using the condition filter, and then passes them to the data push module and the data initialization module. The data initialization module initializes the queried data and stores it in the cache. The data push module can push the data from the data query mapping module and the cache to the distributed message queue.

[0054] The data integration server receives the job information transmitted by the task scheduling cluster, initializes and conditionally filters the job information, and then transmits it to the data visualization server and / or 3D visualization client through the distributed message queue; in addition, DIP caches the message from the data subscription end passed by MQ into the REDIS cache. At the same time, the task scheduling adds a scheduling task, triggers the execution of the BI query interface to obtain data according to the task time, and pushes the data to MQ; it should be noted that the cancellation action is the opposite of the registration action, clearing the subscription cache and scheduling tasks.

[0055] The data visualization client includes a screen designer and a player, which are used to receive job information transmitted by a distributed message queue and design and display the screen subscription registration and cancellation functions of the job information; the screen designer mainly designs and configures the screen, including which controls are selected for the screen and which models are bound to the controls, etc. The screen player includes register subscription / cancel subscription, receives data from the distributed message queue and forwards the data to the data screen.

[0056] The 3D visualization client includes a screen player, which is used to receive job information transmitted by a distributed message queue and display the screen subscription registration content. The screen player includes register subscription / cancel subscription, receives MQ data and forwards the data to the data screen.

[0057] The system further includes a memory cache and a database. The memory cache is an open-source key-value storage system, mostly used as a cache, and can be deployed in a cluster mode; the database is used to store the logical relationships of several visualization clients.

[0058] The data visualization client in this embodiment includes a DVP micro front-end, a DVP microservice layer and a storage layer. The DVP micro front-end corresponds and connects the screen designer and the display with the API interfaces in the DVP microservice layer through a logic layer, and the API interface layer stores the data in the storage layer after processing the objects of the data screen.

[0059] As Figure 2 shown, the DVP front-end includes a process diagram of calling the DVP interface layer to save and update data such as content, model list, screen opening, saving, resource uploading / downloading, and resource obtaining connection.

[0060] The DVP interface layer includes a model list interface, an uploading / downloading interface, an external link interface, a screen content and a temporary saving interface.

[0061] Embodiment 2

[0062] This embodiment provides a method for data subscription registration and cancellation based on distributed screens. As Figure 5 shown, it includes the following steps:

[0063] S1, based on the subscription center on several data visualization clients and / or 3D visualization clients, the job information of the subscription center is sent to the distributed message queue through the subscription data push center of the subscription center;

[0064] The job information includes registration information and cancellation information, and the registration information and cancellation information include data screen information, control information, model information, data set information, and subscription frequency information;

[0065] Step S1 further includes conditional filtering: according to the data screen information, control information, model information, data set information, and subscription frequency information in the job information, duplicate filtering is performed on the job information, and then it is passed into the distributed message queue.

[0066] S2. The distributed message queue autonomously distributes the received job information to the schedulable executors in the task scheduling cluster respectively;

[0067] When all schedulers and executors join the task scheduling cluster, they need to send their registration messages to the specified queue immediately to inform other nodes of their existence; specifically: after the scheduler is successfully registered, the main scheduler will dispatch scheduling tasks and synchronize jobs; after the executor is successfully registered, the main scheduler synchronizes all jobs, and all schedulers add schedulable executor information.

[0068] S3. After receiving the allocated job information, the schedulable executor immediately informs other scheduling executors of its own status through the distributed message queue, and at the same time sends the job information to the data integration server, and pushes the screen data to the corresponding screen.

[0069] The job information in this embodiment may also include create, update, delete, and query. When there are create, update, delete, and query operations in the job, the scheduler will send a message to the specified message queue. When a node receives the message, it will judge the operation type of the message, and the operation type will clearly identify create, query, update, and delete; different actions will be triggered for different types, and all nodes consume and synchronize incremental jobs to achieve job consistency.

[0070] The data pusher numbers the query conditions provided by the data visualization client, numbers the duplicate query conditions with the same number, and at the same time differentiates according to the screen instance id; multiple instances of dvp subscribe and cancel subscriptions for data, and the backend dip receives subscription registrations / cancellations and pushes / stops the data pushing process, as Figure 3 shown; then after querying the result according to the query condition, the query result is pushed to the data visualization client screen corresponding to the corresponding screen instance id. For the detailed description of the query mapping, there is a conditional filter in the query mapping process to filter out redundant conditions, and dip only pushes the required data.

[0071] The working principle of the filter is as Figure 4As shown, each client corresponds to a subscription end, and each subscription end corresponds to a query condition. The query condition includes data screen information, control information, model information, data set information, and subscription frequency information. Condition filtering mainly performs deduplication filtering based on the model information, data combination information, and subscription frequency in the query condition. Different clients are very likely to include the same model information, data set information, and subscription frequency. The model information includes a model unique identifier and model-corresponding data fields. The data set information is mainly a set of conditions for filtering on the control according to multiple data fields of the model.

[0072] By adopting the above technical solutions disclosed in the present invention, the following beneficial effects are obtained:

[0073] The present invention discloses a data subscription registration and cancellation system and method based on a distributed screen. The screen subscription adopts a distributed registration and cancellation subscription scheme, which improves the subscription performance and efficiency of the screen to cope with high-concurrency scenarios, and then filters for the same query conditions to avoid duplicate data pushing and improve the pushing efficiency of job information.

[0074] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A data subscription registration and cancellation system based on distributed images, characterized in that, It includes several data visualization clients and / or 3D visualization clients, a task scheduling cluster, a distributed message queue, and a data integration server. The data visualization client and the 3D visualization client can initiate screen subscription registration or cancellation information and send it to the task scheduling cluster. The task scheduling cluster includes several schedulers and executors, and regularly sends information to the distributed message queue to inform its status. Obtain job information and send the obtained job information to the data integration server. The data integration server receives the job information transmitted by the task scheduling cluster, initializes and conditionally filters the job information, and then transmits it to the data visualization server and / or 3D visualization client through the distributed message queue. The data visualization client is used to receive the job information transmitted by the distributed message queue and design and display the screen subscription registration and cancellation functions of the job information. The 3D visualization client is used to receive the job information transmitted by the distributed message queue and display the screen registration subscription content. The system also includes a memory cache and a database. The memory cache is an open-source key-value storage system, mostly used as a cache, and can be deployed in a cluster mode. The database is used to store the logical relationships of several visualization clients. The data visualization client includes a micro front-end, a DVP microservice layer, and a storage layer. The DVP micro front-end connects the screen designer and the display to the API interfaces in the DVP microservice layer through the logic layer. After the object processing process of the data screen, the data is stored in the storage layer through the API interface layer. The data integration server includes a data query mapping module, a data push module, and a data initialization module. The data query mapping module is connected to the data initialization module and the data push module respectively. The data query mapping module includes a condition filter to query and obtain the data information content, and uses the condition filter to filter out the redundant data information conditions, and then transmits them to the data push module and the data initialization module. The data initialization module initializes the queried data and stores it in the cache. The data push module can push the data from the data query mapping module and the cache to the distributed message queue.

2. A method for data subscription registration and cancellation based on distributed images, implemented based on the system described in claim 1, characterized in that, It includes the following steps: S1, Based on the subscription center on several data visualization clients and / or 3D visualization clients, forward the job information of the subscription center to the distributed message queue through the subscription data push center. S2, The distributed message queue distributes the received job information to the schedulable executors in the task scheduling cluster respectively. S3, After receiving the allocated job information, the schedulable executor immediately informs other scheduling executors of its own status through the distributed message queue. At the same time, it needs to send the job information to the data integration server and push the screen data to the corresponding data visualization client and / or 3D visualization client for screen display.

3. The data subscription registration and cancellation method based on distributed images according to claim 2, characterized in that, The operation information in step S1 includes registration information and cancellation information, and the registration information and cancellation information include data screen information, control information, model information, data set information, and subscription frequency information; Step S1 further includes conditional filtering: deduping and filtering the operation information according to the data screen information, control information, model information, data set information, and subscription frequency information in the operation information.

4. The method for data subscription registration and cancellation based on distributed screens according to claim 3, characterized in that It is characterized in that Between step S1 and S2, it further includes: all schedulers and executors need to send their registration messages to the specified queue immediately when joining the task scheduling cluster to inform other nodes of their existence; specifically: after the scheduler is successfully registered, the main scheduler will dispatch scheduling tasks and synchronize jobs; after the executor is successfully registered, the main scheduler synchronizes all jobs, and all schedulers add schedulable executor information.

5. The data subscription registration and cancellation method based on distributed pictures according to claim 4, characterized in that When there are addition, deletion, modification, and query operations on the job, the scheduler will send a message to the specified message queue, and all nodes consume and synchronize the incremental job to achieve job consistency.

6. The method for data subscription registration and cancellation based on distributed images according to claim 5, characterized in that, The scheduler determines which message queue to send the message to and triggers the timing of the decision to send the message; the "all nodes" refers to the executor nodes and schedulers that cooperate with each other to complete the entire scheduling execution process.

7. The method for data subscription registration and cancellation based on distributed pictures according to claim 6, wherein The data pusher numbers the query conditions provided by the data visualization client, numbers the repeated query conditions with the same number, and at the same time differentiates according to the screen instance id; then after querying the result according to the query condition, the query result is pushed to the data visualization client screen corresponding to the corresponding screen instance id.

Images