Visual large screen data increment real-time updating method with low bandwidth consumption

Abstract

The invention discloses a visual large screen data increment real-time updating method with low bandwidth consumption, system equipment and a storage medium, a client creates an initial data object, monitors the initial data object in real time and records an operation history, the client generates a data change record according to the operation history, and the data change record is used for updating the data change record according to the data change record. The method comprises the steps that a client receives a data change record, compares the data change record with a data change record which is pulled from a server and synchronized last time, screens incremental content, the client transmits the incremental content to the server, and the server asynchronously updates a database according to the incremental content and then returns an update result to the client. According to the method, the changed data object is obtained by adopting the modes of incremental real-time updating, real-time data object monitoring and operation history recording, and only the data change part, namely the incremental content, is transmitted and processed, so that the bandwidth consumption is remarkably reduced, the real-time performance of data synchronization is further improved, and the load of a server side is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for real-time incremental update of visualization large-screen data with low bandwidth consumption. Background Art

[0002] With the rapid development of intelligent industry, digital large screens have become important tools for real-time monitoring, data analysis, and information display in the manufacturing industry. With the help of visualization components and low-code blueprint dashboards, business personnel can efficiently and intuitively build business dashboards through a large-screen editor, thereby improving the efficiency of data display and decision-making. However, in actual applications, visualization large screens need to handle the collection, processing, and storage of massive amounts of data, which poses higher requirements for the technical architecture. Currently, in actual applications, visualization large screens need to handle the collection, processing, and storage of massive amounts of data, which poses higher requirements for the technical architecture. Most large-screen systems still use the full-update method for data synchronization. During the full-update process, the system usually first clears the existing data, then exports the entire data set from the source data table, and imports it into the target data table to overwrite the original data content.

[0003] However, a large screen often consists of dozens or even hundreds of components, involving massive data processing. Although full-data update is simple and straightforward to implement, as the data scale continues to grow, this method often brings huge data transmission and storage pressure, occupies a large amount of network resources, and thus affects the response speed and overall performance of the system. At the same time, the cost of moving the entire data set may become high and time-consuming. The limitations of full-update also make it difficult to achieve real-time multi-person collaboration during the digital large-screen editing process. Summary of the Invention

[0004] The present invention aims to overcome at least one defect of the above-mentioned prior art, and provides a method for real-time incremental update of visualization large-screen data with low bandwidth consumption, which is used to solve the huge technical problems brought by the full-update method of visualization large screens in the prior art.

[0005] The present invention provides a method for real-time incremental update of visualization large-screen data with low bandwidth consumption, including the following steps:

[0006] S01. The client creates an initial data object, listens to the initial data object in real time, and records the operation history;

[0007] S02. The client generates a data change record according to the operation history, compares the data change record with the data change record of the last synchronization pulled from the server side, and filters out the incremental content;

[0008] S03. The client transmits the incremental content to the server side;

[0009] S04. The server - side asynchronously updates the database according to the incremental content, and then returns the update result to the client - side.

[0010] Since the visual large - screen design involves the collection, processing, and storage of massive amounts of data, and most large - screen systems use the full - volume update method for synchronization, which will bring huge data transmission and storage pressure. Therefore, the visual large - screen of this application adopts incremental real - time update. By listening to data objects in real - time and recording operation history, the data objects with changes are obtained. Only the changed part of the data, that is, the incremental content, is transmitted and processed, significantly reducing bandwidth consumption, further improving the real - time performance of data synchronization, and optimizing the server - side load.

[0011] To ensure that data objects are not repeatedly created, the initial data objects in step S01 are created using the singleton pattern and the factory pattern. Using the singleton pattern can ensure that there is only one instance of the initial data object in the system, avoiding repeated instantiation, saving memory and system resources. The factory pattern, through unified object creation, centrally manages the object life cycle, improving the flexibility and encapsulation of data object creation.

[0012] The real - time listening operation uses the proxy pattern to control the reading and setting operations of the variables of the initial data object, realizes real - time monitoring of the property access of the data object and the modification of the data object, and uses the memento pattern to provide snapshots and restoration of the data object state, ensuring the reliability and consistency of real - time listening.

[0013] The operation of recording operation history is implemented using the publish - subscribe pattern, and synchronizes and restores the add, delete, and modify operations of the property of the operation record and its detailed information through forward and reverse operations.

[0014] The generation of the data change record in step S02 includes the following steps:

[0015] The client - side records the change operation record of the initial data object. The change operation record includes fields such as change operation type, change operation path, change value, and timestamp, providing a basis for generating accurate incremental data change records in the follow - up, ensuring that the client - side and the server - side can accurately compare data differences and avoiding ambiguity during incremental generation.

[0016] The client - side generates a version number according to the timestamp field. This version number is unique and is bound to the corresponding change operation record. It can not only provide a globally unique identifier for each change record, facilitating the quick screening and matching of incremental content during data synchronization, but also quickly judge the sequence or conflict of change records through the timestamp field in the version number, improving the comparison efficiency.

[0017] To achieve efficient data synchronization, the screening of incremental content in step S02 includes the following steps:

[0018] The client caches the first data change record locally, and the first data change record is generated by the client listening to the initial data object in real time and recording the operation history.

[0019] The client pulls the second data change record from the server side, and the second data change record is the data change record cached during the last server-side update.

[0020] The client compares the first data change record with the second data change record, and filters out the different operation contents in the data change record according to the version number field, which is recorded as the incremental content.

[0021] Preferably, the transmission of the incremental content in S03 uses the HTTP protocol and the TCP protocol with a custom message for transmission.

[0022] In S04, the server side asynchronously updates the database according to the incremental content, and then returns the update result to the client, including the following steps:

[0023] The server side receives the incremental content, detects whether the data object in the incremental content exists in the database. If it exists, it performs an asynchronous database update operation. The asynchronous update allows the server side to immediately respond to the client without waiting for the database operation to complete, and returns the corresponding HTTP status code according to the update result, so that the client can know that the request has been accepted.

[0024] During peak hours, there is a large amount of incremental content on the visualization large screen. Through asynchronous processing, the waiting time of the client can be significantly reduced. And the database MongoDB has the asynchronous operation characteristic. Therefore, the database is preferably the MongoDB database, which can ensure the fast reading and writing of data. By combining the HTTP request mechanism and the asynchronous operation characteristic of MongoDB, the data consistency can be ensured. At the same time, the reasonable HTTP status code and response mechanism enable the system to feedback the operation result in real time and effectively maintain the integrity of the data.

[0025] The present invention also provides a visualization large screen data incremental real-time update system with low bandwidth consumption, which is characterized by including the following modules:

[0026] Listening module: The client creates an initial data object, listens to the initial data object in real time and records the operation history.

[0027] Comparison module: The client generates a data change record according to the operation history, and compares the data change record with the data change record synchronized from the server side last time to filter out the incremental content.

[0028] Transmission module: The client transmits the incremental content to the server side.

[0029] Update module: The server side asynchronously updates the database according to the incremental content and then returns the update result to the client side.

[0030] The present invention also provides a device, which is characterized by including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a method for real-time incremental update of visualization large-screen data with low bandwidth consumption is implemented.

[0031] The present invention also provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium includes a stored computer program. When the computer program runs, the device where the computer-readable storage medium is located is controlled to execute a method for real-time incremental update of visualization large-screen data with low bandwidth consumption.

[0032] The present invention proposes a method, system, device, and storage medium for large-screen data synchronization based on incremental update. Compared with the existing method of full-scale update of large-screen data, it has more advantages in aspects such as data transmission efficiency, system performance optimization, and real-time guarantee. By overcoming the deficiencies of the traditional full-scale update method, it provides certain technical support for the wide application of visualization large screens in fields such as real-time monitoring, business analysis, and collaborative editing, and helps to promote the further development of smart industry and digital construction.

[0033] The present invention shows the following beneficial effects in multiple aspects:

[0034] (1) Improve data update efficiency

[0035] The traditional full-scale update method requires operations of clearing and rewriting the entire data set. As the data scale increases, the update efficiency gradually decreases. The present invention uses incremental update technology to synchronize only the changed data since the last update, avoiding repeated processing of unchanged data, thereby improving the data synchronization efficiency. In scenarios with high-frequency data updates, this technology can shorten the update cycle and ensure that the system completes the synchronization of large-scale data within seconds, fully meeting the real-time requirements.

[0036] (2) Reduce resource consumption

[0037] The present invention accurately captures the added, modified, and deleted changed data, reduces the data transmission volume, and reduces the dependence on network resources. At the same time, it reduces the consumption of storage and computing resources by the system during data transmission and processing, optimizes the overall resource utilization rate, and provides guarantee for the efficient operation of the system.

[0038] (3) Support high concurrency and high real-time requirements

[0039] The present invention combines a non-blocking transmission and an asynchronous processing mechanism, enabling the backend to efficiently process a large number of data requests while ensuring that the front-end can receive and display data in real time. In high-concurrency scenarios, it improves the system throughput, makes the operation more stable, and can meet the requirements of real-time data display on large screens. In addition, incremental data updates avoid the problems of data display lag or interruption that may occur in traditional full-volume update methods, further optimizing the user experience.

[0040] (4) Support the ability of multi-person collaborative editing

[0041] In the scenario of multi-person collaborative editing, the present invention effectively reduces the latency problem during collaborative editing by capturing the operation changes of each user in real time and synchronizing them to other user terminals immediately. Through the mechanism of recording and tracing the data change log, the present invention supports the revocation of incorrect operations and the restoration of states, improving the reliability and fault tolerance of the system. At the same time, the improved collaboration efficiency enables the present invention to perform well in the scenario of multi-person real-time editing and can meet complex collaborative requirements.

[0042] (5) Improve the flexibility and scalability of the system

[0043] The present invention combines the dynamic data model of MongoDB and the non-blocking characteristics of WebFlux, and can flexibly process complex data structures and dynamically changing data, showing a certain adaptability in various heterogeneous data sources and complex scenarios. In addition, this technology has good scalability, providing technical support for the subsequent function upgrade and multi-scenario expansion of the system. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a schematic diagram of the steps of the method for incrementally and real-time updating the visualization large-screen data with low bandwidth consumption.

[0046] Figure 2 It is a schematic diagram of the system for incrementally and real-time updating the visualization large-screen data with low bandwidth consumption. Detailed Embodiments

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the embodiments of the present invention in detail with reference to the drawings:

[0048] An embodiment of the present invention provides an embodiment of a method, system, device, and storage medium for real-time incremental update of visualization large-screen data with low bandwidth consumption. Although the logical order is shown in the flowchart, under certain data, the steps shown or described can be completed in a different order than here.

[0049] Embodiment 1: Refer to Figure 1

[0050] As Figure 1 shown in the step schematic diagram of a method for real-time incremental update of visualization large-screen data with low bandwidth consumption, the present invention provides a method for real-time incremental update of visualization large-screen data with low bandwidth consumption, including the following steps:

[0051] S01. The client creates an initial data object, listens to the initial data object in real time, and records the operation history;

[0052] S02. The client generates a data change record according to the operation history, compares the data change record with the data change record of the previous synchronization pulled from the server side, and filters out the incremental content;

[0053] S03. The client transmits the incremental content to the server side;

[0054] S04. The server side asynchronously updates the database according to the incremental content, and then returns the update result to the client.

[0055] Since the visualization large screen is designed for the collection, processing, and storage of massive data, and most large-screen systems use the full-volume update method for synchronization, which will bring huge data transmission and storage pressure. Therefore, the visualization large screen of the present application adopts incremental real-time update. By listening to data objects in real time and recording the operation history, the data objects that have changed are obtained, and only the changed part of the data, that is, the incremental content, is transmitted and processed, significantly reducing the bandwidth consumption, further improving the real-time performance of data synchronization, and optimizing the load of the server side.

[0056] To ensure that the data object will not be created repeatedly, the initial data object in step S01 is created using the singleton pattern and the factory pattern. Using the singleton pattern can ensure that there is only one instance of the initial data object in the system, avoiding repeated instantiation, saving memory and system resources, while the factory pattern improves the flexibility and encapsulation of data object creation through unified object creation and centralized management of object life cycles.

[0057] The real-time monitoring operation uses the proxy pattern to control the read and set operations of the variables of the initial data object, realizes the real-time monitoring of the property access of the data object and the modification of the data object, and uses the memento pattern to provide snapshots and restoration of the data object state to ensure the reliability and consistency of real-time monitoring.

[0058] The operation history recording operation is implemented using the publish-subscribe pattern. Through forward and reverse operation synchronization and restoration of the property addition, deletion, and modification operations of the operation record and their detailed information, all changes will be captured and converted into structured operation history record entries. The strategy pattern (Strategy Pattern) is used to support different recording strategies, such as local storage, remote logging services, etc. The records can be output in real-time or saved periodically to persistent storage.

[0059] Generating the data change record in step S02 includes the following steps:

[0060] The client records the change operation record of the initial data object. The change operation record includes fields such as the change operation type, change operation path, change value, and timestamp, providing a basis for generating accurate incremental data change records in the future, ensuring that the client and the server can accurately compare data differences and avoiding ambiguity during incremental generation.

[0061] The data changes include:

[0062] When adding an attribute, return the latest valid value and record the operation history. Use the memento pattern (MementoPattern) to save and restore the state of the object;

[0063] When modifying an attribute, first verify the legality of the new value, then create a new immutable data version, and record the modification operation history;

[0064] When deleting an attribute, first create a new immutable data version and record the modification operation history. Finally, perform the deletion operation and update the overall state.

[0065] The client generates a version number based on the timestamp field. This version number is unique and is bound to the corresponding change operation record. It can not only provide a globally unique identifier for each change record, facilitating quick screening and matching of incremental content during data synchronization, but also quickly determine the sequence or conflict of change records through the timestamp field in the version number, improving the comparison efficiency.

[0066] The client can also hijack the read and write operations of data objects through the proxy mode Proxy, record the paths and values of all change operations (addition, deletion, and modification), generate a structured operation log, such as {op: 'update', path: 'componentA.value', value: 100}. The operation log generates a unique identifier through a hash algorithm to avoid duplicate submissions or redundant transmissions.

[0067] To achieve efficient data synchronization, the steps for screening incremental content in step S02 include the following steps:

[0068] The client caches the first data change record locally. The first data change record is generated by the client listening to the initial data object in real time and recording the operation history.

[0069] The client pulls the second data change record from the server side. The second data change record is the data change record cached during the last server-side update.

[0070] The client compares the first data change record with the second data change record, and screens out the operation content with differences in the data change record according to the version number field, which is recorded as incremental content.

[0071] Preferably, the transmission of the incremental content in S03 uses the HTTP protocol and the TCP protocol with a custom message for transmission.

[0072] In step S04, the server side asynchronously updates the database according to the incremental content, and then returns the update result to the client, including the following steps:

[0073] The server side receives the incremental content, detects whether the data object in the incremental content exists in the database. If it exists, it performs an asynchronous database update operation. Asynchronous updates allow the server side to immediately respond to the client without waiting for the database operation to complete, and return the corresponding HTTP status code according to the update result, so that the client can know that the request has been accepted.

[0074] During peak hours, there is more incremental content on the visualization dashboard. Asynchronous processing can significantly reduce the client waiting time. Since the database MongoDB has asynchronous operation characteristics, therefore, the database is preferably a MongoDB database, which can ensure fast data reading and writing. By combining the HTTP request mechanism and the asynchronous operation characteristics of MongoDB, data consistency can be ensured. At the same time, a reasonable HTTP status code and response mechanism enable the system to provide real-time feedback on operation results and effectively maintain data integrity.

[0075] The client caches frequently accessed data to improve the response speed and uses the FlyweightPattern to reduce memory occupancy.

[0076] To optimize the processing performance of incremental content, non-critical tasks are processed asynchronously to avoid blocking the main thread, and asynchronous programming models such as Promise and async / await are used to implement non-blocking IO operations.

[0077] The client also provides an interface that allows users to roll back to any historical state based on history. The command pattern is used to implement the undo and redo functions, and the snapshot function is supported to facilitate multi-point comparison and analysis.

[0078] Example 2: Refer to Figure 2

[0079] As Figure 2 shown, the present invention also provides a visual large-screen data incremental real-time update system with low bandwidth consumption, which is characterized by including the following modules:

[0080] Monitoring module: The client creates an initial data object, monitors the initial data object in real time, and records the operation history;

[0081] Comparison module: The client generates a data change record according to the operation history, and compares the data change record with the data change record of the last synchronization pulled from the server side to filter out the incremental content;

[0082] Transmission module: The client transmits the incremental content to the server side;

[0083] Update module: The server side asynchronously updates the database according to the incremental content, and then returns the update result to the client.

[0084] To achieve the design principle of the smallest granularity and the smallest network transmission volume, when performing each TCP transmission, the parameters carried should be as small as possible and may meet the positioning and update requirements of the backend. Therefore, a set of operation definitions and data transmission carrier parameters for the cooperation between the front end and the back end need to be defined as follows:

[0085]

[0086] Among them,

[0087] O is the operation definition, a / u / d are add, update, delete respectively,

[0088] D is the operation path, key1->key2-key3, v is the specific update value.

[0089] S is the operation session, which is used to maintain the Session.

[0090] Therefore, each TCP data volume is at the B level, resulting in lower network bandwidth consumption.

[0091] The overall network transmission interaction adopts a completely non-blocking design to improve the system's response speed and concurrent processing ability. When the backend server receives a request, the system will immediately execute a broadcast operation to ensure that each relevant module can respond to user requests in a timely manner. Database operations are designed as post-processing to optimize the front-end user experience and avoid unnecessary delays.

[0092] HTTP status code design:

[0093] To achieve real-time feedback of operation results, the system accurately transmits the backend processing results through semantic HTTP status codes and structured response bodies:

[0094] - 200 OK: The data update is successful, and the response body contains the complete updated data;

[0095] - 201 Created: The new data is successfully added, and the unique identifier of the new data is returned;

[0096] - 404 Not Found: The target data does not exist, prompting the front end to terminate the invalid operation;

[0097] - 409 Conflict: Version conflict, triggering the front end to re-pull the latest data;

[0098] - 500 Internal Server Error: Server internal error, prompting the front end to retry.

[0099] For example, when the user updates the data of the large screen component, the backend realizes the feedback through the following process:

[0100] 1. The front end sends an HTTP PUT request, carrying the component ID and the updated value;

[0101] 2. The backend verifies the existence of the data. If it does not exist, it returns 404;

[0102] 3. If it exists, perform an asynchronous update of MongoDB. After successful update, return 200 and the updated data;

[0103] 4. If a concurrent conflict is detected during the update process, such as inconsistent version numbers, return 409;

[0104] 5. If a database operation exception occurs, return 500 and record the error log.

[0105] After the backend receives the operation data from the frontend, it broadcasts the data, broadcasts the frontend data to all Sessions within the Channel, performs an update operation on the database, and judges the consistency through the HTTP status. If the update fails, a broadcast is required to notify all Sessions to perform a synchronization operation.

[0106] The pseudocode is as follows:

[0107] / / When the backend receives the frontend operation data

[0108] onReceiveFrontendData(data):

[0109] / / Broadcast the data to all sessions

[0110] broadcastToAllSessions(data)

[0111] / / Update the database

[0112] updateStatus = updateDatabase(data)

[0113] / / Judge whether the update is successful

[0114] if updateStatus == FAILURE:

[0115] / / Broadcast and notify all sessions to perform a synchronization operation

[0116] broadcastToAllSessions("Synchronization required")

[0117] The present invention also provides a device, which is characterized by including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a method for real-time incremental update of visualization large screen data with low bandwidth consumption is implemented.

[0118] The present invention also provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute a method for real-time incremental update of visualization large screen data with low bandwidth consumption.

Claims

1. A method for real-time incremental update of visualization big screen data with low bandwidth consumption, characterized in that, It includes the following steps: S01. The client creates an initial data object, listens to the initial data object in real time, and records the operation history; S02. The client generates a data change record according to the operation history, compares the data change record with the data change record of the previous synchronization pulled from the server side, and filters out the incremental content; S03. The client transmits the incremental content to the server side; S04. The server side asynchronously updates the database according to the incremental content, and then returns the update result to the client.

2. A method for real-time incremental update of visual large-screen data with low bandwidth consumption according to claim 1, characterized in that, The initial data object in S01 is created using the singleton pattern and the factory pattern; The real-time listening operation is implemented using the proxy pattern and the memo pattern; The operation of recording the operation history is implemented using the publish-subscribe pattern.

3. A method for real-time incremental update of visualization large-screen data with low bandwidth consumption according to claim 1, characterized in that, The generation of the data change record in S02 includes the following steps: The client records the change operation record of the initial data object, and the change operation record includes a change operation type, a change operation path, a change value, and a timestamp field; The client generates a version number according to the timestamp field and binds it to the corresponding change operation record.

4. A method for real-time incremental update of visualization big screen data with low bandwidth consumption according to claim 3, characterized in that The filtering of the incremental content in S02 includes the following steps: The client caches the first data change record locally, and the first data change record is generated by the client listening to the initial data object in real time and recording the operation history; The client pulls the second data change record from the server side, and the second data change record is the data change record cached during the previous server side update; The client compares the first data change record with the second data change record, and filters out the different operation content in the data change record according to the version number field, which is recorded as the incremental content.

5. A method for real-time incremental update of visualization big screen data with low bandwidth consumption according to claim 1, characterized in that, The transmission of the incremental content in S03 is transmitted using the HTTP protocol and the TCP protocol with a custom message.

6. A method for real-time incremental update of visualization large-screen data with low bandwidth consumption according to claim 1, characterized in that, The server side in S04 asynchronously updates the database according to the incremental content, and then returns the update result to the client, including the following steps: The server side receives the incremental content, detects whether the data object in the incremental content exists in the database. If it exists, it performs an asynchronous database update operation and returns the corresponding HTTP status code according to the update result.

7. A method for real-time incremental update of visual big screen data with low bandwidth consumption according to claim 6, characterized in that, The database is preferably a MongoDB database.

8. A visualization large screen data incremental real-time update system with low bandwidth consumption, characterized in that, It includes the following modules: Listening module: The client creates an initial data object, listens to the initial data object in real time, and records the operation history; Comparison module: The client generates a data change record according to the operation history, compares the data change record with the data change record of the previous synchronization pulled from the server side, and filters out the incremental content; Transmission module: The client transmits the incremental content to the server side; Update module: The server side asynchronously updates the database according to the incremental content, and then returns the update result to the client.

9. A device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a method for real-time incremental update of visualization big screen data with low bandwidth consumption as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a method for real-time incremental update of visualization large-screen data with low bandwidth consumption as described in any one of claims 1 to 7.

Images