Data distribution system of B / S architecture real-time data visualization system

Abstract

According to the data distribution system of the B / S architecture real-time data visualization system, the B / S architecture real-time data visualization system comprises a front end, a rear end and a persistence layer, and the data distribution system comprises a precise subscription module and a container cloud load balancing module. The problems of data delay and packet loss of a B / S architecture real-time data visualization system in a large-throughput and high-concurrency scene are solved, and the availability and user experience of the real-time data visualization system in a complex industrial scene are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of B / S architecture, and in particular relates to a data distribution system of a B / S architecture real-time data visualization system. Background Art

[0002] With the rapid development of informatization today, the B / S (Browser / Server) architecture is being used more and more widely. This architecture has unique advantages such as easy access, fast iteration, seamless upgrade and cross-platform availability, which coincides with the diversified front-end requirements of real-time data visualization systems such as distributed mobile terminals and centralized command centers, so it is widely used in this field. In industrial scenarios such as power, chemical industry, and aerospace with many measuring points (sensors) and high transmission frequency, due to the good performance of the intranet network, in order to ensure the real-time data transmission, the traditional C / S (Client / Server) system mostly uses UDP multicast messages for data distribution, and this data distribution method does not have concurrency problems. When the visualization system migrates from the C / S architecture to the B / S architecture, the reliable communication technology based on the connection, websocket, becomes the first choice because the browser does not support the UDP protocol. This requires the background to send data one by one to many front-ends, resulting in data delays or even data loss problems at high throughput and high concurrency, which seriously affects the availability of the visualization system.

[0003] Real-time data visualization system with B / S architecture Figure 1 As shown in the figure, it includes the front-end, back-end and persistence layer. In the past, polling and long polling technologies were usually used to achieve dynamic updates of website front-end data. Polling is to repeatedly initiate HTTP requests to the back-end at a specified time interval and return data through HTTP. This mode has two obvious disadvantages. First, if there is no data update on the server, the HTTP request is invalid. Second, the HTTP request may contain a long header, but the valid data returned is only a small part. Obviously, this will waste a lot of bandwidth and other resources. In long polling, the server will block the HTTP request and return it when the data is updated. The front-end will initiate the request again after completing the display of the returned data. This method avoids invalid HTTP requests, but there is still server resource consumption caused by repeated connections and low information density. In the scenario of high-frequency data refresh, the HTTP blocking time is short, and the performance degenerates to the polling mode. Based on the above application requirements, HTML5 defines the WebSocket protocol.

[0004] In the WebSocket protocol, the browser and the server only need to complete a handshake to create a persistent connection between them and perform two-way data transmission. Figure 2This allows the server to actively push data to the client without the client initiating polling, making data exchange between the client and the server much simpler.

[0005] The front-end needs to establish a stable websocket connection with the back-end for duplex data transmission. When the back-end distributes data, the data will be sent repeatedly to different front-ends. This brings two problems. First, the network bandwidth usage is proportional to the front-end access volume, which may lead to bandwidth exhaustion. Second, the back-end service uses high CPU resources through cyclic traversal transmission, which may cause data accumulation and data delay or loss.

[0006] For example, in a real industrial scenario, nearly 70,000 measurement points are deployed, and the frequency of data generated by each measurement point varies from 1Hz, 10Hz, 20Hz, and 50Hz. The average data frequency is about 20Hz. Data distribution adopts JSON format, considering Figure 3 The simplified data structure shown in the figure is 92 bytes in size. For the above industrial scenario, if no technical strategy is adopted, the bandwidth occupied by pushing all data to a single front end is as high as 982.7Mb. At the same time, the browser side will also have performance problems when processing such a large amount of data push. When the above traffic needs to be pushed to a large number of front ends at the same time, data delays, loss, and even system paralysis will inevitably occur due to bandwidth exhaustion and server resource exhaustion. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a data distribution system for a real-time data visualization system with a B / S architecture, and proposes solutions from two aspects: controlling network bandwidth and reducing server pressure.

[0008] The data distribution system of the B / S architecture real-time data visualization system of the present invention comprises a front end, a back end and a persistence layer, and the data distribution system comprises:

[0009] Precision subscription module, which is data subscription, including front-end data subscription and back-end data subscription; on the front-end, i.e., browser side, when the visualization component is loaded, it applies to the back-end for data subscription, and when the component is uninstalled, it reports data unsubscription to the back-end; on the back-end, i.e., server side, it receives data subscription information in real time, and distributes data accurately according to the subscription status of the front-end;

[0010] A container cloud load balancing module combines the high-performance fixed PCs and rack-mounted servers of a centralized command and control center into a container cloud for data distribution of the entire visualization system, and uses a corresponding operation and maintenance management platform for container orchestration and load balancing.

[0011] The front-end data subscription includes a display component, a data binding module, a channel management module and remote data;

[0012] The display component is responsible for information display;

[0013] The data binding module is used to record the data subscription information of all components, dynamically maintain the subscriber list for each data, and distribute the data to the subscribers when the data is received;

[0014] The channel management module is responsible for channel establishment and management, and for sending data subscription instructions;

[0015] Remote data, backend microservices responsible for data.

[0016] The display component is provided with a component data binding interface for providing the component with functions related to data subscription, as well as function calls for data reception and interface refresh.

[0017] The channel management module is responsible for establishing and managing the websocket channel. It is the final sender of subscription information and the direct receiver of data. When the websocket channel is abnormal, it is responsible for collecting data subscription requirements and implementing reconnection and resubscription.

[0018] The backend, namely the server, dynamically maintains a subscription information list and a subscriber list. The subscription information list is used to record the number of subscriptions to the data, and the subscriber list is used for data distribution.

[0019] The subscription information list is used to record the number of subscriptions to the data. When the subscription information is received, if there is subscription information for the data, the corresponding number of data subscriptions increases, otherwise a record is added to both the subscription information list and the subscriber list; if the parameter has not recorded the websocket subscriber in the subscriber list, the websocket is added to the relevant record; when the unsubscribe information is received, the corresponding number of subscriptions decreases. If the subscription number is reduced to zero, the corresponding records of the data in the subscription information list and the subscriber list are deleted at the same time.

[0020] The subscriber list is used for data distribution, which means that when the real-time data is received and parsed, the table is immediately queried. If there is a subscriber record for the data, the websocket list of the record is traversed to distribute the data.

[0021] In terms of the configuration strategy of the container cloud load balancing module, the rack-mounted server provides normal and stable data distribution services for mobile terminals and scattered users, and the command and control center uses microservices running in the PC terminal container for data distribution.

[0022] The front end includes visualization software, designer software and system management software. The visualization software is responsible for the final information presentation, the designer software is responsible for the development and production of visualization pages, and the system management software is responsible for basic information configuration and editing.

[0023] The backend includes page management services, basic information services, data receiving and distribution services, and data storage services; the persistence layer usually directly accesses or operates the disk through a database.

[0024] The beneficial effect of the present invention is that the present invention solves the data delay and packet loss problems of the B / S architecture real-time data visualization system in high-throughput and high-concurrency scenarios by proposing data precision subscription and container cloud load balancing solutions, effectively solves the problems of network bandwidth and front-end and back-end hardware resource exhaustion, and greatly improves the availability and user experience of the real-time data visualization system in complex industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a real-time data visualization system architecture.

[0026] Figure 2 It is the interaction sequence of three modes.

[0027] Figure 3 Is the data distribution format.

[0028] Figure 4 It is a front-end data subscription solution.

[0029] Figure 5 It is a backend data subscription solution.

[0030] Figure 6 It is a container cloud platform with heterogeneous hardware. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0032] like Figure 1As shown, the B / S architecture real-time data visualization system of the present invention includes a front end, a back end and a persistence layer. The front end includes visualization software, designer software and system management software. The visualization software is responsible for the final information presentation, the designer software is responsible for the development and production of visualization pages, and the system management software is responsible for basic information configuration and editing. The back-end core services are mainly page management services, basic information services, data receiving and distribution services, and data storage services. The persistence layer usually directly accesses or operates the disk through a database. The external interface is mainly various data sources such as sensors, interface machines, and decision-making systems. After massive high-frequency real-time data enters the visualization system, it is parsed and distributed to the front end by the data receiving and distribution service, and the data storage service performs data persistence storage to facilitate on-demand query by the front end. The front-end visualization software and designer software establish a stable connection with the data receiving and distribution service, and the data flows to the front end from this connection.

[0033] The data distribution system of the present invention includes a precise subscription module and a container cloud load balancing module.

[0034] The precise subscription module is for data subscription, including front-end data subscription and back-end data subscription. On the front-end, i.e. the browser, the visualization component applies for data subscription to the back-end when it is loaded, and reports data unsubscription to the back-end when the component is uninstalled. On the back-end, i.e. the server, data subscription information is received in real time, and precise data distribution is performed according to the subscription status of the front-end.

[0035] Front-end data subscription includes display component (component), data binding module (Datasource), channel management module (WSDatasource) and remote data (server);

[0036] The display component (component) is responsible for information display; the data binding module (Datasource) is used to record the data subscription information of all components, is responsible for subscribing data to WSDatasource, and when the data is received, is responsible for distributing the data to the subscribers.

[0037] The channel management module (WSDatasource) is responsible for establishing and managing the websocket channel and sending data subscription instructions;

[0038] Remote data (server), responsible for the backend microservices of the data.

[0039] A component data binding interface (DataBinding) is set in the display component to provide the component with data subscription related functions to facilitate component development.

[0040] The channel management module is responsible for establishing and managing the websocket channel. It is the final sender of subscription information and the direct receiver of data. When the websocket channel is abnormal, it is responsible for collecting data subscription requirements and implementing reconnection and resubscription.

[0041] like Figure 4 As shown, the present invention provides developers with a component development interface DataBinding, which provides relevant functions for data subscription, as well as function calls for data reception and interface refresh. Components only need to rewrite relevant functions to subscribe to and use real-time data. At the application level, a Datasource module is designed, which records the data subscription information of all components and dynamically maintains a list of subscribers (components) for each data. When data is received, it will be distributed to the components that subscribe to the data, thereby triggering visual rendering. At the network level, a WSDatasource channel management module is designed, which is responsible for establishing and managing the websocket channel, and is the final sender of subscription information and the direct recipient of data. When the websocket channel is abnormal, it is responsible for collecting data subscription requirements and implementing reconnection and resubscription.

[0042] The backend, i.e. the server, is responsible for dynamically maintaining the subscription information list and the subscriber list. The subscription information list is used to record the number of subscriptions to the data, and the subscriber list is used for data distribution.

[0043] like Figure 5 As shown, the subscription information list is used to record the number of subscriptions to the data. When the subscription information is received, if there is subscription information for the data, the corresponding number of data subscriptions increases, otherwise a record is added to both the subscription information list and the subscriber list; if the websocket subscriber has not been recorded in the subscriber list for this parameter, the websocket is added to the relevant records; when the unsubscribe information is received, the corresponding number of subscriptions decreases. If the number of subscriptions is reduced to zero, the corresponding records of the data in the subscription information list and the subscriber list are deleted at the same time.

[0044] The subscriber list is used for data distribution, which means that after receiving and parsing the real-time data, the table is immediately queried. If there is a subscriber record for the data, the websocket list of the record is traversed for data distribution.

[0045] It can be seen that the data precision subscription solution significantly reduces the amount of data pushed from the backend to the frontend. It not only effectively controls the network bandwidth usage, but also avoids unnecessary resource usage of the frontend web application and pressure on the backend data distribution service, ensuring that the entire system is lighter and more stable.

[0046] Due to the openness of the B / S architecture, even if precise data subscription is used, the system still faces performance problems caused by server pressure as the amount of data and concurrent access increase. At the same time, in special places such as centralized command and control centers, a large number of users tend to pay attention to overall situation information, which is strongly related to the workflow. This is the convergence of data needs of a large number of users. The convergence of needs requires the backend to traverse and send the same data to a large number of frontends, which weakens the performance improvement brought by precise data subscription and is extremely unfriendly to avoid data delay and loss caused by backend CPU resource occupation. Therefore, a load balancing solution based on container cloud is proposed.

[0047] like Figure 6 As shown in the figure, the container cloud load balancing module makes full use of the high-performance fixed computing resources of the centralized command and control center, and combines the high-performance fixed PCs of the centralized command and control center with rack servers to form a container cloud for data distribution of the entire visualization system, and uses the corresponding operation and maintenance management platform for container orchestration and load balancing. In terms of configuration strategy, rack servers provide normal and stable data distribution services for mobile terminals and scattered users, while a large number of users with similar needs such as the command and control center use microservices running in PC terminal containers for data distribution. When such users go offline, the corresponding microservices can also go offline, thus realizing a "self-sufficient" data service ( Figure 6 (shown by double solid lines).

[0048] It can be seen that this model can dynamically expand the data distribution service as the front-end access volume increases, ensuring balanced and controllable resource consumption of the entire cluster, and solving the problem of system performance degradation as concurrent access increases.

[0049] Example

[0050] The technical solution of the present invention is applied to a real industrial scenario described in the background technology, and a visualization system driven by 400 concurrent data and refreshed in real time is realized. The system has been put into use.

[0051] In a test, two control halls were used. All terminals in Hall 1 joined the container cloud, while Hall 2, as a scattered user, did not join the container cloud. The test results are shown in Table 1. In the table, "PC front-end number" refers to the number of users who use PC computing resources for data distribution, "scattered front-end number" refers to the number of users who use servers for data distribution, and "PC CPU" and "server CPU" are both the average CPU consumption of the data distribution service process. This test is performed on a specific display page.

[0052] Table 1 Concurrent access service resource consumption

[0053]

[0054] It can be seen that the actual operation results of the load balancing solution based on container cloud are consistent with the design assumptions. The data precision subscription solution solves the system availability, which is confirmed by the normal operation of the system.

[0055] Although the above embodiments have been shown and described, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. Changes, modifications, substitutions and variations of the above embodiments by those of ordinary skill in the art are all within the scope of protection of the present invention.

Claims

1. A data distribution system for a B / S architecture real-time data visualization system, characterized in that: The B / S architecture real-time data visualization system includes a front end, a back end and a persistence layer, and the data distribution system includes: Precision subscription module, which is data subscription, including front-end data subscription and back-end data subscription; on the front-end, i.e., browser side, when the visualization component is loaded, it applies to the back-end for data subscription, and when the component is uninstalled, it reports data unsubscription to the back-end; on the back-end, i.e., server side, it receives data subscription information in real time, and distributes data accurately according to the subscription status of the front-end; A container cloud load balancing module combines the high-performance fixed PCs and rack-mounted servers of a centralized command and control center into a container cloud for data distribution of the entire visualization system, and uses a corresponding operation and maintenance management platform for container orchestration and load balancing.

2. According to the data distribution system of the B / S architecture real-time data visualization system of claim 1, it is characterized in that: The front-end data subscription includes a display component, a data binding module, a channel management module and remote data; The display component is responsible for information display; The data binding module is used to record the data subscription information of all components, dynamically maintain the subscriber list for each data, and distribute the data to the subscribers when the data is received; The channel management module is responsible for channel establishment and management, and for sending data subscription instructions; Remote data, backend microservices responsible for data.

3. The data distribution system of the B / S architecture real-time data visualization system according to claim 2 is characterized in that: The display component is provided with a component data binding interface for providing the component with functions related to data subscription, as well as function calls for data reception and interface refresh.

4. The data distribution system of the B / S architecture real-time data visualization system according to claim 2 is characterized in that: The channel management module is responsible for establishing and managing the websocket channel. It is the final sender of subscription information and the direct receiver of data. When the websocket channel is abnormal, it is responsible for collecting data subscription requirements and implementing reconnection and resubscription.

5. The data distribution system of the B / S architecture real-time data visualization system according to claim 1 is characterized in that: The backend is responsible for dynamically maintaining a subscription information list and a subscriber list. The subscription information list is used to record the number of subscriptions to data, and the subscriber list is used for data distribution.

6. The data distribution system of the B / S architecture real-time data visualization system according to claim 5, characterized in that: The subscription information list is used to record the number of subscriptions to the data. When subscription information is received, if there is subscription information for the data, the corresponding number of data subscriptions increases, otherwise a record is added to both the subscription information list and the subscriber list; if the websocket subscriber has not been recorded in the parameters in the subscriber list, the websocket is added to the relevant records; when unsubscribe information is received, the corresponding number of subscriptions decreases. If the number of subscriptions is reduced to zero, the corresponding records of the data in the subscription information list and the subscriber list are deleted at the same time.

7. The data distribution system of the B / S architecture real-time data visualization system according to claim 5, characterized in that: The subscriber list is used for data distribution, which means that when the real-time data is received and parsed, the table is immediately queried. If there is a subscriber record for the data, the websocket list of the record is traversed to distribute the data.

8. The data distribution system of the B / S architecture real-time data visualization system according to claim 1, characterized in that: The configuration strategy of the container cloud load balancing module is that the rack-mounted server provides normal and stable data distribution services for mobile terminals and scattered users, and the command and control center uses microservices running in the PC terminal container to distribute data.

9. The data distribution system of the B / S architecture real-time data visualization system according to claim 1, characterized in that: The front end includes visualization software, designer software and system management software. The visualization software is responsible for the final information presentation, the designer software is responsible for the development and production of visualization pages, and the system management software is responsible for basic information configuration and editing.

10. The data distribution system of the B / S architecture real-time data visualization system according to claim 1, characterized in that: The backend includes page management services, basic information services, data receiving and distribution services, and data storage services; the persistence layer directly accesses or operates the disk through a database.

Images