Data dynamic caching method and system

Abstract

The application provides a data dynamic caching method and system, which comprises the following steps: determining the data display content in a visualization large screen corresponding to a target data source; in the case that the target data source is determined to have data changes, obtaining target changed data in the target data source according to data change state information corresponding to the target data source and a target data change strategy set bound to the target data source, wherein the target data change strategy set at least comprises a data change strategy, the data change state information is determined based on change conditions of data having data changes in the target data source, and the target changed data is data having data changes in the target data source and satisfying the data change strategy; and performing data caching processing on the target changed data, so as to refresh the data display content in the visualization large screen. The application improves the efficiency and flexibility of data refreshing of the visualization large screen.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for dynamic data caching. Background Technology

[0002] With the continuous advancement of information technology, visualization screen technology has penetrated into all walks of life, becoming an important tool for real-time display of key data and business indicators.

[0003] In daily operation, visualization dashboard systems need to process large amounts of data. To ensure the timeliness and accuracy of the data, servers typically perform data consolidation operations periodically, combining multiple data sources. This data consolidation operation is a space-for-time trade-off approach in the development field. By systematically and structurally processing large amounts of source data, a new data model is formed, which improves the efficiency of subsequent data analysis. Subsequently, the visualization dashboard periodically calls backend interfaces to refresh the displayed data content, ensuring real-time information updates.

[0004] However, while related technologies can ensure smooth refresh of data on large visual dashboards, their data caching cycle configuration is often fixed. This means the system cannot dynamically adjust the data caching cycle according to changes in business requirements, which may limit the system's flexibility and responsiveness to some extent. Therefore, there is an urgent need for a dynamic data caching method and system to solve the above problems. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method and system for dynamic data caching.

[0006] This invention provides a method for dynamic data caching, comprising:

[0007] Determine the target data source for the data displayed on the visualization dashboard;

[0008] When it is determined that the target data source has undergone data change, the target changed data in the current target data source is obtained according to the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source. The target data change strategy set includes at least a data change strategy. The data change status information is determined based on the change status of the data in the target data source that has undergone data change. The target changed data is the data in the target data source that has undergone data change and the data change status information satisfies the data change strategy.

[0009] The target change data is cached for use in refreshing the data displayed on the visualization screen.

[0010] According to a data dynamic caching method provided by the present invention, the target data change strategy set further includes a time change strategy, the time change strategy includes a fixed time change strategy and a fixed period change strategy, wherein the fixed time change strategy is used to trigger the corresponding strategy operation when a preset time is reached, and the fixed period change strategy is used to trigger the corresponding strategy operation at the beginning of each preset period.

[0011] The target data change strategy set includes at least two of the data change strategies, and one of the fixed-time change strategy or the fixed-period change strategy.

[0012] According to a data dynamic caching method provided by the present invention, determining the target data source corresponding to the data display content in the visualization dashboard includes:

[0013] Based on the data table information or field information, obtain the target data table or target field in the database corresponding to the data display content;

[0014] Use the target data table or the target field as the target data source.

[0015] According to a dynamic data caching method provided by the present invention, the steps for determining that the target data source has undergone data changes are as follows:

[0016] Real-time acquisition of binary log information;

[0017] If it is determined from the binary log information that the target data source has undergone data changes, each piece of data that has undergone data changes is converted into the corresponding target message body;

[0018] Based on the change time information of the message body, each of the target message bodies is added to the message cache queue in sequence;

[0019] Based on a preset data model format, the data corresponding to each target message body in the message cache queue is converted to obtain the pending change data that has undergone data change.

[0020] If the comparison result between the pending change data that has undergone data change and the pending change data before the data change in the target data source satisfies the preset data change conditions, it is determined that the target data source has undergone data change;

[0021] When it is determined that the target data source has undergone data change, the step of obtaining the target changed data in the current target data source based on the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source includes:

[0022] Based on the policy configuration relationship between the pending change data that has undergone data change and the data change policies in the target data change policy set, the target data change policy corresponding to the pending change data that has undergone data change is determined from the target data change policy set.

[0023] Obtain the data change status information corresponding to the pending change data that is currently undergoing data change, wherein the data change status information includes at least the data change quantity information, data change type information, and data change duration of the pending change data that is currently undergoing data change;

[0024] If the data change quantity information, data change type information, and data change duration of the pending change data that is currently undergoing data change meet the target data change strategy, the pending change data that is currently undergoing data change is determined as the target change data.

[0025] According to a data dynamic caching method provided by the present invention, if it is determined from the binary log information that the target data source is currently undergoing data changes, converting each piece of data currently undergoing data changes into a corresponding target message body includes:

[0026] Determine the data change type, source table, changed data, and change time information for each data item that has undergone a data change;

[0027] The target message body is constructed based on the data change type information, the source table information, the changed data information, and the change time information.

[0028] According to a data dynamic caching method provided by the present invention, before determining the target data change policy corresponding to the currently changing pending data from the target data change policy set based on the policy configuration relationship between the currently changing pending data and the data change policies in the target data change policy set, the method further includes:

[0029] Obtain multiple preset data change strategy sets consisting of different data change strategies;

[0030] Based on the effective time of each of the preset data change strategy sets, construct a timeline corresponding to all the preset data change strategy sets;

[0031] Based on the current position of the corresponding time period on the time axis, the target data change strategy set is determined from the multiple preset data change strategy sets.

[0032] According to a data dynamic caching method provided by the present invention, the step of caching the target change data for refreshing the data display content on the visualization screen includes:

[0033] After obtaining the target change data, a target data refresh request information is generated;

[0034] Determine whether the target data refresh request information is the first data refresh request information generated in the current period. If it is the first data refresh request information generated in the current period, then perform data caching on the target changed data to obtain dynamic cached data.

[0035] Based on the dynamically cached data, the data displayed on the visualization screen is refreshed;

[0036] If the data refresh request is not the first one generated within the current period, delete the target data refresh request.

[0037] According to a data dynamic caching method provided by the present invention, the method further includes:

[0038] When it is determined that the visualization screen is in the initialization and startup phase, an initial service data query is performed based on the displayed data to obtain initial query data.

[0039] Based on the initial query data, a data sedimentation operation is performed to obtain initial cache data;

[0040] Based on the initial cached data, the data displayed on the visualization screen during the initialization and startup phase is refreshed.

[0041] The present invention also provides a dynamic data caching system, comprising:

[0042] The data source determination module is used to determine the target data source corresponding to the data displayed on the visualization dashboard.

[0043] The strategy execution module is used to, when it is determined that the target data source has undergone data change, obtain the target changed data in the current target data source based on the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source. The target data change strategy set includes at least a data change strategy, the data change status information is determined based on the change status of the data that has undergone data change in the target data source, and the target changed data is the data in the target data source that has undergone data change and the data change status information satisfies the data change strategy.

[0044] The cache control module is used to cache the target change data and refresh the data displayed on the visualization screen.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data dynamic caching method as described above.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data dynamic caching method as described above.

[0047] The data dynamic caching method and system provided by this invention, when it is determined that the target data source corresponding to the data display content in the visualization screen has undergone data change within the current period, obtains the changed data in the target data source that has undergone data change and whose data change status satisfies the corresponding data change strategy in the target data change strategy set, based on the change status information of the target data source and the target data change strategy set. Then, based on these changed data, dynamic cache data is constructed for refreshing the data display content in the visualization screen, thereby improving the efficiency and flexibility of data refresh in the visualization screen. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the data dynamic caching method provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the dynamic data caching architecture for large-screen visualization provided by the present invention;

[0051] Figure 3 This invention provides an internal logic diagram of the data receiver.

[0052] Figure 4 A schematic diagram of the structure of the dynamic data caching system provided by the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] With the rapid development of information technology, large-screen visualizations have been widely used across various industries to display key data and business metrics in real time. Whether in financial monitoring, public safety, traffic management, or enterprise operations, large-screen visualizations, with their intuitive and dynamic data display capabilities, greatly improve decision-making efficiency and business insight.

[0056] In general, visualization dashboard systems need to handle massive amounts of data. Services often periodically combine data sources for data consolidation, and then the visualization dashboard system periodically calls backend interfaces to refresh the data. This approach ensures smooth data refresh on the dashboard, but because the data consolidation cycle is often fixed, it cannot be dynamically adjusted to keep up with changes in business requirements.

[0057] To address the problems existing in the aforementioned related technologies, this invention provides a dynamic data caching method for large-screen visualizations, with the core objective of enhancing the flexibility and responsiveness of data refresh on these visualizations. This method first establishes a detailed set of data refresh strategy rules. When information in the underlying data source changes, these changes are immediately captured and pushed to the server. Subsequently, the system intelligently assesses the degree of data change according to the predefined refresh strategy. Once the magnitude of the data change reaches a preset refresh threshold, the system automatically activates the data caching process to update the cached data. After the update is completed, the latest data is immediately pushed to the large-screen interface, ensuring the real-time nature and accuracy of the displayed content.

[0058] Figure 1 This is a flowchart illustrating the data dynamic caching method provided by the present invention, as shown below. Figure 1 As shown, the present invention provides a method for dynamic data caching, comprising:

[0059] Step 101: Determine the target data source corresponding to the data displayed on the visualization dashboard.

[0060] In this invention, the visualization dashboard is typically linked to multiple databases, which store rich business data. When determining the target data source, the most relevant and accurate data is selected from these data sources based on the specific content displayed on the dashboard. For example, if the dashboard is used to display sales data, relevant data tables or fields from a sales database containing sales orders, customer information, and product details might be chosen as the target data source. In one embodiment, if a visualization dashboard is built to monitor the operational status of a data center, it needs to display the CPU utilization of each server. In this case, the "Server Status Table" in the database can be selected as the data source, and fields such as "Server ID" and "CPU Utilization" in the table can be specified as the data to be extracted. Then, the system will establish corresponding data extraction and mapping rules, and display the extracted CPU utilization data in real time at the corresponding position on the dashboard.

[0061] Furthermore, when determining the data content to be displayed on the visualization dashboard within the current period, it is necessary to select the target data source based on this period. This means filtering from multiple data sources to identify the most relevant and accurate data. For example, if the visualization dashboard is currently used to display monthly sales data, the sales database, which includes sales orders, customer information, and product details, is selected as the target data source. In the next stage, if the type of content displayed changes (e.g., from displaying monthly sales data to displaying warehouse inventory data), the data source for the next period needs to be redefined. If the type of content remains unchanged, the subsequent dynamic data caching steps continue based on the currently determined target data source.

[0062] In one embodiment, the data retrieved from the data source can be cleaned and transformed (such as removing duplicate data, converting data format, etc.), and aggregated or calculated according to business needs to form data in a standard format for subsequent data comparison and analysis.

[0063] Step 102: If it is determined that the target data source has undergone data change, the target changed data in the current target data source is obtained according to the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source. The target data change strategy set includes at least a data change strategy. The data change status information is determined based on the change status of the data in the target data source that has undergone data change. The target changed data is the data in the target data source that has undergone data change and whose data change status information satisfies the data change strategy.

[0064] In this invention, continuous monitoring of changes in the target data source is required, which can be achieved through techniques such as database triggers, polling queries, or real-time data streaming. When a change is detected in the data of the target data source, the system records detailed information about these changes, including the time of the change, the type of change (e.g., addition, modification, deletion), and the specific content of the change. It should be noted that in this invention, the process of determining whether a data change has occurred in the target data source requires first analyzing the comparison between the current and previous data changes. Only when the data change reaches a certain level is it determined that a data change has occurred in the target data source. That is, by analyzing the degree of data change, this invention ensures that the visualization dashboard does not frequently refresh due to minor data changes during each data refresh, thereby guaranteeing the stability of the visualization dashboard while ensuring real-time updates of the displayed content.

[0065] Furthermore, the system will organize the monitored change information (such as the data table information and change type information corresponding to the changed data) into change status information (such as the number of data tables with changed data, the change type of the changed data being a new data type, etc.). This change status information records the change status of each data item in detail, and also corresponds to a certain strategy in the data change strategy set. When the change status information of the relevant data meets a certain corresponding strategy, the strategy is triggered, and then these relevant data are used as target changed data for subsequent dynamic cache construction.

[0066] In this invention, a set of target data change strategies needs to be preset. This set of strategies defines the conditions or rules that must be met when data changes occur. For example, change type filtering: only focusing on newly added or modified data, ignoring deleted data; data scope restriction: only considering data changes within a specific time period or a specific business scope; change frequency control: limiting the update frequency of data changes to avoid instability in the large screen display due to excessively frequent data updates. In this invention, the data change strategy set is bound to the corresponding data source (or dataset). When it is determined that data in the data source has changed and its change status meets the strategy of the data change strategy set, the strategy can be triggered to determine the target changed data.

[0067] When acquiring target change data, this invention further processes and formats this data to ensure it meets the requirements for large-screen display. This may include operations such as data aggregation, transformation, and sorting. Finally, the processed target change data will be pushed onto a visual large screen for users to view and analyze in real time.

[0068] Step 103: The target change data is cached for use in refreshing the data display content on the visualization screen.

[0069] In a visual dashboard system, data is constantly changing, and these changes need to be reflected on the dashboard in real time or periodically. When the system receives target change data, this data typically includes updates or additions to certain key information. This target change data may come from multiple data sources, such as databases, API interfaces, and user input.

[0070] Furthermore, upon receiving this target change data, preprocessing tasks such as data parsing and format conversion can be performed to ensure data consistency and accuracy. Then, the received data is validated to ensure it conforms to the expected format and range, avoiding errors and anomalies in subsequent processing.

[0071] Furthermore, the verified target change data is integrated into the existing cache. Before refreshing the dashboard, a snapshot set can be generated. If problems occur during the refresh process, the system can quickly restore the previous state, avoiding data loss or display anomalies. In this invention, based on the data mapping relationship, the data in the dynamic cache is rendered to the corresponding position on the visualization dashboard, involving various operations such as chart updates, text replacement, and color changes.

[0072] The data dynamic caching method provided by this invention, when it is determined that the target data source corresponding to the data display content in the visualization screen has undergone data change within the current period, obtains the changed data in the target data source that has undergone data change and whose data change status satisfies the corresponding data change strategy in the target data change strategy set, based on the change status information of the target data source and the target data change strategy set. Then, based on these changed data, it constructs dynamic cache data for refreshing the data display content in the visualization screen, thereby improving the efficiency and flexibility of data refresh in the visualization screen.

[0073] Based on the above embodiments, the target data change strategy set further includes a time change strategy, which includes a fixed time change strategy and a fixed period change strategy. The fixed time change strategy is used to trigger the corresponding strategy operation when a preset time is reached, and the fixed period change strategy is used to trigger the corresponding strategy operation at the beginning of each preset period.

[0074] Based on the above embodiments, the target data change strategy set includes at least two data change strategies, and one fixed-time change strategy or one fixed-period change strategy.

[0075] In this invention, the target data change strategy set is a comprehensive framework designed to manage and execute change operations for different data objects. Within this framework, time-based change strategies are triggered based on specific time conditions and can be further subdivided into two main categories: fixed-time change strategies and fixed-period change strategies. Each of these strategies has its own application scenarios and triggering mechanisms.

[0076] Specifically, a fixed-time change strategy is a data change mechanism based on absolute time points. This strategy pre-sets one or more specific time points, and when the system time reaches these preset time points, the strategy operation associated with that time point is automatically triggered. This strategy is particularly suitable for scenarios that require data processing or updates at specific times, such as setting it to execute automatically at midnight every day. The advantage of the fixed-time change strategy lies in its accuracy and predictability, ensuring that data change operations are executed accurately and without error at the expected time.

[0077] Unlike fixed-time change strategies, fixed-period change strategies are based on a time cycle. This strategy first sets an initial time point and a time period (such as daily, weekly, or monthly), and then triggers the strategy operation every preset period starting from this initial time point. This strategy is suitable for data change tasks that need to be executed periodically, such as synchronizing the latest customer data from an external system every day at 3 PM. Fixed-period change strategies ensure data consistency and system maintenance efficiency by periodically executing data change operations.

[0078] The target data change strategy set is a collection of data change strategies and time change strategies. It includes at least two data change strategies (which can be triggered based on data content, data status, or other non-time conditions, including a comparison strategy for judging the data change status and a data strategy for judging the target changed data) and at least one time change strategy (which can be a fixed time change strategy or a fixed period change strategy). This allows the visualization dashboard system to flexibly combine and apply different strategies according to complex and ever-changing business needs, so as to achieve efficient and accurate data management.

[0079] Based on the above embodiments, determining the data display content in the visualization screen within the current target data source includes:

[0080] Based on the data table information or field information, obtain the target data table or target field in the database corresponding to the data display content;

[0081] Use the target data table or the target field as the target data source.

[0082] In data visualization or data processing strategies, data is typically stored in relational databases (such as MySQL and PostgreSQL) or non-relational databases (such as MongoDB and Cassandra), and this data is distributed across different tables and fields.

[0083] A data table is the basic unit for storing data in a database. It typically contains multiple rows and columns of data, with each row representing a record and each column representing a field. Based on the data table information, you can locate the specific table containing the required data. A field is a column in a data table used to store specific types of data (such as integers, strings, dates, etc.). Based on the field information, you can further refine your search to a specific field in the data table that contains the required data.

[0084] In this invention, it is necessary to parse the data table information or field information in the configuration file corresponding to the displayed data. This information may be provided in string form, such as "Device Alarm Table" or "Storage Capacity Table - Capacity Field". The parsed information is used to execute queries in the database. For data table information, the entire table can be queried directly; for field information, the table name and field name need to be specified to query data for a specific field. Based on the query results, the target data table or target field is determined.

[0085] Once the target data table or field is identified, it can be considered the target data source corresponding to the content displayed on the visualization dashboard. In this invention, configuring the data source typically involves specifying database connection information (such as database type, address, port, username, password, etc.) and detailed information about the target data table or field (such as table name, field name, data type, etc.). After configuration, data can be extracted from the target data source. For example, suppose a visualization dashboard displays device alarm information and storage capacity. The data source corresponding to the alarm information is the "Device Alarm Table"; the data source corresponding to the storage capacity is the "Storage Capacity Table - Capacity Field". The system parses this information, queries the "Capacity" field of the "Storage Capacity Table" in the database, and uses this field as the target data source. By directly locating the target data table or field, the time spent blindly searching the database is reduced, improving data processing efficiency.

[0086] Based on the above embodiments, the steps for determining that the target data source has undergone data changes are as follows:

[0087] Real-time acquisition of binary log information;

[0088] If it is determined from the binary log information that the target data source has undergone data changes, each piece of data that has undergone data changes is converted into the corresponding target message body;

[0089] Based on the change time information of the message body, each of the target message bodies is added to the message cache queue in sequence;

[0090] Based on a preset data model format, the data corresponding to each target message body in the message cache queue is converted to obtain the pending change data that has undergone data change.

[0091] If the comparison result between the pending change data that has undergone data change and the pending change data before the data change in the target data source satisfies the preset data change conditions, it is determined that the target data source has undergone data change;

[0092] When it is determined that the target data source has undergone data change, the step of obtaining the target changed data in the current target data source based on the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source includes:

[0093] Based on the policy configuration relationship between the pending change data that has undergone data change and the data change policies in the target data change policy set, the target data change policy corresponding to the pending change data that has undergone data change is determined from the target data change policy set.

[0094] Obtain the data change status information corresponding to the pending change data that is currently undergoing data change, wherein the data change status information includes at least the data change quantity information, data change type information, and data change duration of the pending change data that is currently undergoing data change;

[0095] If the data change quantity information, data change type information, and data change duration of the pending change data that is currently undergoing data change meet the target data change strategy, the pending change data that is currently undergoing data change is determined as the target change data.

[0096] Figure 2 This is a schematic diagram of the dynamic data caching architecture for large-screen visualization provided by the present invention, which can be referred to. Figure 2 As shown, in this invention, based on the data refresh requirements of the visualization dashboard, a corresponding dynamic data caching architecture is constructed, including a configuration center, data receiver, policy library manager, policy set manager, policy listener, dataset manager, and dashboard integration service, wherein:

[0097] The configuration center is used to store relevant configuration information, such as policy configuration, policy set configuration and data source information. Among them, the configuration of policies can meet a variety of business scenarios.

[0098] The data receiver interfaces with the data source and is responsible for receiving data change information pushed by the data source. Then, based on the subscription status of the policy, it pushes the changed data information to the policy library manager.

[0099] The policy library manager is used to maintain information about each policy and to receive information about changes in policy subscriptions;

[0100] The policy set manager is used to maintain policy set information and receive information about changes in policy set subscriptions;

[0101] The policy listener is used to receive information about changes in policy types and to check whether the policy set meets the triggering conditions.

[0102] The dataset manager is used to maintain dataset information, receive policy set change information, and refresh the data cache.

[0103] The large screen integration service connects with the front-end large screen UI components to provide dataset query and push services.

[0104] Figure 3 The internal logic diagram of the data receiver provided by the present invention can be referred to. Figure 3 As shown, the data receiver internally includes three core components: a binary log listener, a message buffer queue, and a message processor.

[0105] The listener monitors the binary log, which records all database changes (such as data insertion, update, and deletion operations). These records are stored in binary format, which is crucial for tasks such as data recovery, replication, and auditing. After analyzing the binary log information and confirming changes in the source table data corresponding to the data displayed in the task configuration, the listener converts the changed data rows into the target message body and places them in the message buffer queue.

[0106] Based on the above embodiments, if it is determined from the binary log information that the target data source is currently experiencing data changes, converting each piece of data that is currently undergoing data changes into a corresponding target message body includes:

[0107] Determine the data change type, source table, changed data, and change time information for each data item that has undergone a data change;

[0108] The target message body is constructed based on the data change type information, the source table information, the changed data information, and the change time information.

[0109] In this invention, a complete message body format is as follows:

[0110] Data change type information: ADD, UPDATE, DELETE;

[0111] Source table information of attribution: Source table information to which the message belongs;

[0112] Changed data information: Original information of the data rows that have changed;

[0113] Change time information: Time point when the change occurs.

[0114] Further, sort according to the change time of the message body, and place the message body read in the listener in the data queue for processing. Then, through the message processor, convert the original data model corresponding to the message body into a standard data model (i.e., the preset data model format). And according to the subscription mechanism of the data table, push the relevant information into the policy set manager.

[0115] In the present invention, the policies in the data change policy set are divided into two types: time change policy and data change policy. The time change policy includes two methods: fixed cycle and fixed time. Among them, the fixed cycle means that after this policy is triggered, after an interval of time n, the next policy is triggered; the fixed time means a specific time point to trigger the policy.

[0116] The data change policy is a model constructed by the present invention based on the data change status of the data source. Among them, the data change policy is composed of three parts: data source, change type, and comparison method. Among them, the data source can select a data table or a specific field in the database as the data source. For example, the device alarm table, storage capacity table - capacity field, etc. can all be used as the data source of the policy. The change type is the same as the message change type sent by the database in the message processor, and is divided into four types: all (i.e., covering all change types), new, modified, and deleted.

[0117] The comparison method is essentially a judgment logic, similar to the ">" in "a>b" and the "contains" in "a contain b". The present invention judges whether the policy is triggered through the comparison method and the boolean value returned by the comparison value, that is, judges whether the degree of data change of the data source has reached the standard of data change.

[0118] Specifically, the comparison method adopted by the present invention changes dynamically according to the selected data source and change type, and compares the change situation between the changed data corresponding to the display content of the same data before and after the change. For example, if a field of numerical field type and a change type of modification are selected, the comparison method can select the change rate ≥ x%; if a field of string type is selected, then contain / equal / not equal to x can be selected; if a data table is selected and the change of the deletion type occurs, the data volume change rate > x% or < x% can be selected. When the comparison result meets the above comparison method, the policy can be triggered.

[0119] In this invention, for time-changing strategies, the corresponding internal thread is started by the strategy listener to listen directly after the strategy is created.

[0120] Furthermore, considering that a single data change strategy may not meet actual judgment needs in real-world business scenarios, a strategy group concept, or strategy set, was designed. A strategy group is a collection of strategies; when any strategy within the group is triggered, that strategy is activated. This invention flexibly combines time-change and data-change strategies, effectively addressing data refresh needs across multiple scenarios.

[0121] Specifically, in this invention, there is a corresponding configuration relationship between the pending change data and the data change strategies in the target data change strategy set. This configuration relationship determines which strategies will be applied to which data. Further, for the current pending change data, its data change status information is obtained. This information includes at least: the number of data changes (e.g., how many records were changed), the type of data change (e.g., addition, modification, deletion), and the duration of the data change (e.g., the time period in which the change occurred). Then, the obtained data change status information is compared with the target data change strategy to check whether the number, type, and duration of data changes meet the conditions set in the corresponding strategy. If the change status information of the pending change data meets the conditions of the target data change strategy, then this data is identified as target change data, and it will be further processed or refreshed according to the strategy requirements.

[0122] For example, a strategy might require refreshing the data whenever the number of alarms displayed in the data reaches 10 or more, with a minimum refresh frequency of 5 minutes. This would require creating a data strategy 'a', binding it to the alarm table, specifying all alarm types, and setting the alarm count to ≥10. Simultaneously, a periodic strategy 'b' would be created, with a refresh frequency of 5 minutes. Finally, data strategies 'a' and 'b' would be added together as a single strategy to the strategy set.

[0123] In this invention, after the data change policy is created, a subscription mechanism will be enabled. The data table and change type in the data source will be used to subscribe to the data receiver. When the corresponding data changes, the message will be pushed to the policy set manager. The policy listener will first judge the data change according to the comparison method in the policy set. If the comparison method returns true, it means that the policy is triggered. Then, when the corresponding change status information meets the policy, the dataset will trigger the corresponding interface and refresh the cached data.

[0124] The present invention provides a dynamic data caching method for visual dashboards, which adds a data change strategy to the fixed-period refresh. The essence of this strategy is the change in the data source table; the strategy is triggered when the degree of change reaches a certain set threshold. Furthermore, multiple data change strategies and time change strategies can be combined to form a strategy set, which is then bound to the dataset. Ultimately, this reduces resource waste while achieving dynamic and flexible refresh of the dashboard dataset.

[0125] Based on the above embodiments, before determining the target data change strategy corresponding to the currently changing pending data from the target data change strategy set according to the strategy configuration relationship between the currently changing pending data and the data change strategies in the target data change strategy set, the method further includes:

[0126] Obtain multiple preset data change strategy sets consisting of different data change strategies;

[0127] Based on the effective time of each of the preset data change strategy sets, construct a timeline corresponding to all the preset data change strategy sets;

[0128] Based on the current position of the corresponding time period on the time axis, the target data change strategy set is determined from the multiple preset data change strategy sets.

[0129] In this invention, multiple data change strategy sets are predefined and stored. Each strategy set consists of different data change strategies, which involve data filtering conditions, change operations, triggering conditions, etc. Furthermore, a timeline is constructed based on the effective time of each preset data change strategy set. The timeline is a linear representation used to show the effectiveness of each strategy set within different time periods. Each strategy set has one or more effective intervals on the timeline, indicating the time periods during which the strategy set is effective.

[0130] When a data change operation is required, the current time period is first determined. Then, the timeline is searched to find which preset data change strategy sets are valid within this time period. Finally, one of these valid strategy sets is selected as the target data change strategy set to guide the current data change operation. This invention switches strategy groups based on time to achieve adaptation to more scenarios, and actively pushes the dataset to the dashboard after it is refreshed, reducing invalid API calls to the dashboard.

[0131] Based on the above embodiments, the step of caching the target change data for refreshing the data display content on the visualization screen includes:

[0132] After obtaining the target change data, a target data refresh request information is generated;

[0133] Determine whether the target data refresh request information is the first data refresh request information generated in the current period. If it is the first data refresh request information generated in the current period, then perform data caching on the target changed data to obtain dynamic cached data.

[0134] Based on the dynamically cached data, the data displayed on the visualization screen is refreshed;

[0135] If the data refresh request is not the first one generated within the current period, delete the target data refresh request.

[0136] In this invention, the dataset is positioned identically to the dataset of the visualization dashboard. It is bound to one or more data blocks of the visualization dashboard in the northward direction, and the source data is processed by the corresponding interface in the southward direction to form a new data model (i.e., modified data in a preset data model format). Based on this, cached information is established within the dataset. Data actively queried by the visualization dashboard can be directly obtained by querying this cached data. Simultaneously, a dataset can be bound to multiple policy sets, and the effective time of each policy set can be set according to the timeline. When any effective policy set is triggered, the dataset will trigger the interface and refresh the cached data. Therefore, in this invention, if there are no effective policy sets within a certain timeline, it means that the dataset will not be refreshed during that period.

[0137] Furthermore, if a single dataset has already triggered a dataset refresh operation once at the same time, any subsequent refresh request during the current refresh period will be discarded. Specifically, when target changed data (i.e., data that needs to be updated in the cache) is detected, a target data refresh request is first generated. This request contains all the necessary information for the refresh, such as the dataset identifier, detailed information about the changed data, and the time the request was generated. After generating the target data refresh request, it is checked whether this request is the first data refresh request generated within the current period. This can be achieved by comparing the request generation time with the start time of the current period. If the request is the first one generated after the start of the current period, then it is the first request; otherwise, it is not the first request.

[0138] If the result indicates that this is the first data refresh request generated within the current period, then the target changed data will be cached. This involves extracting the necessary information from the target changed data and storing it in the cache for fast access later. During the caching process, a series of data processing steps can be performed, such as data validation and format conversion, to ensure the accuracy and consistency of the cached data.

[0139] If the result is not the first data refresh request generated within the current period, then this request is deleted. This is because only one data refresh operation needs to be processed at any given time. If the same request is received again later, it will be considered redundant and discarded by the system. This approach avoids the negative impact of excessive refresh operations on performance and ensures the timeliness and accuracy of cached data.

[0140] Finally, once the cached dataset is refreshed, a push mechanism will be triggered. The latest dataset information will be actively pushed to the visualization dashboard through the connected service, at which point the visualization dashboard will trigger a data refresh of the corresponding components.

[0141] Based on the above embodiments, the method further includes:

[0142] When it is determined that the visualization screen is in the initialization and startup phase, an initial service data query is performed based on the displayed data to obtain initial query data.

[0143] Based on the initial query data, a data sedimentation operation is performed to obtain initial cache data;

[0144] Based on the initial cached data, the data displayed on the visualization screen during the initialization and startup phase is refreshed.

[0145] In this invention, if this is the first time the visualization dashboard page is displayed, it is necessary to actively call the service to query data. Since there is no cached data in the dataset at this time, the corresponding dataset will trigger a data settling process to populate the cached data, and the dashboard data will begin to be displayed.

[0146] Furthermore, after the initialization and startup phase, the visualization dashboard will no longer actively initiate service requests except for manual page refreshes. Instead, the backend service will proactively push dataset change information, thereby effectively reducing invalid data calls.

[0147] In this invention, the policy manager reads configuration information, initializes relevant policies, and sends data change subscription information to the data receiver. After all policies are initialized, the policy set manager initializes the policy groups and binds them to snapshot sets according to configuration rules. After the data receiver starts, it begins receiving change information from the database and pushes the change information to the policy set manager according to the subscription configuration. Once the service is running stably, policies and policy sets can be dynamically adjusted based on operational status, thereby controlling the refresh frequency and refresh policy of the snapshot sets. During this process, the service can run normally without requiring any coding.

[0148] The dynamic data caching method for visualization dashboards provided by this invention adds a data strategy to the concept of a periodic strategy for visualization dashboards. This strategy is based on changes to the underlying database data. By receiving data messages and comparing them with its own configured rules, it determines the triggering timing of the strategy. Multiple strategy combinations and multiple strategy groups are bound to a dataset along with a timeline through strategy groups. When a strategy is triggered, a data refresh of the snapshot set is triggered, and finally, the latest data is fed back to the visualization dashboard through a data push mechanism within the snapshot set.

[0149] The data dynamic caching system provided by the present invention is described below. The data dynamic caching system described below can be referred to in correspondence with the data dynamic caching method described above.

[0150] Figure 4 A schematic diagram of the structure of the dynamic data caching system provided by the present invention is shown below. Figure 4 As shown, this invention provides a dynamic data caching system, including a data source determination module 401, a strategy execution module 402, and a cache control module 403. The data source determination module 401 determines the target data source corresponding to the data display content on the visualization screen. The strategy execution module 402, upon determining that the target data source has undergone data changes, obtains the target changed data in the target data source based on the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source. The target data change strategy set includes at least a data change strategy, and the data change status information is determined based on the changes in the data in the target data source. The target changed data is the data in the target data source that has undergone data changes and whose data change status information satisfies the data change strategy. The cache control module 403 performs data caching processing on the target changed data and refreshes the data display content on the visualization screen.

[0151] The data dynamic caching system provided by this invention, when it is determined that the target data source corresponding to the data display content in the visualization screen has undergone data change within the current period, obtains the changed data in the target data source that has undergone data change and whose data change status satisfies the corresponding data change strategy in the target data change strategy set, based on the change status information of the target data source and the target data change strategy set. Then, based on these changed data, it constructs dynamic cache data for refreshing the data display content in the visualization screen, thereby improving the efficiency and flexibility of data refresh in the visualization screen.

[0152] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0153] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504, wherein the processor 501, communications interface 502, and memory 503 communicate with each other via the communication bus 504. The processor 501 can call logical instructions in the memory 503 to execute a dynamic data caching method. This method includes: determining the target data source corresponding to the data display content on the visualization screen; when it is determined that the target data source has undergone data change, obtaining the target changed data in the current target data source based on the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source, wherein the target data change strategy set includes at least a data change strategy, the data change status information is determined based on the change status of the data in the target data source that has undergone data change, and the target changed data is the data in the target data source that has undergone data change and the data change status information satisfies the data change strategy; and caching the target changed data for refreshing the data display content on the visualization screen.

[0154] Furthermore, the logical instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the data dynamic caching method provided by the above methods, the method including: determining the target data source corresponding to the data display content in the visualization screen; when it is determined that the target data source has undergone data change, obtaining the target changed data in the current target data source according to the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source, wherein the target data change strategy set includes at least a data change strategy, the data change status information is determined based on the change of the data in the target data source that has undergone data change, and the target changed data is the data in the target data source that has undergone data change and the data change status information satisfies the data change strategy; and performing data caching processing on the target changed data for refreshing the data display content in the visualization screen.

[0156] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data dynamic caching method provided in the above embodiments. The method includes: determining the target data source corresponding to the data display content in the visualization screen; when it is determined that the target data source has undergone data change, obtaining the target changed data in the target data source according to the data change status information corresponding to the target data source and the target data change strategy set bound to the target data source, wherein the target data change strategy set includes at least a data change strategy, the data change status information is determined based on the change status of the data that has undergone data change in the target data source, and the target changed data is the data in the target data source that has undergone data change and the data change status information satisfies the data change strategy; and performing data caching processing on the target changed data for refreshing the data display content in the visualization screen.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically caching data, the method comprising: The method comprises the steps of: determining the data display content in the current corresponding target data source in the visualization large screen; in the case where it is determined that the target data source has data changes, acquiring target changed data in the current target data source according to data change state information corresponding to the target data source and a target data change strategy set bound to the target data source, wherein the target data change strategy set at least comprises a data change strategy, the data change state information is determined based on change of data in the target data source that has data changes, and the target changed data is data in the target data source that has data changes and whose data change state information satisfies the data change strategy; performing data cache processing on the target changed data for refreshing the data display content in the visualization large screen; the determination step that the target data source has data changes is as follows: real-time acquisition of binary log information; if it is determined according to the binary log information that the target data source currently has data changes, converting each data that currently has data changes into a corresponding target message body; based on change time information of the message body, sequentially adding each target message body to a message cache queue; based on a preset data model format, performing format conversion on data corresponding to each target message body in the message cache queue to obtain pending changed data that currently has data changes; if a comparison result between the pending changed data that currently has data changes and pending changed data before data changes corresponding to the pending changed data that currently has data changes in the target data source satisfies a preset data change condition, it is determined that the target data source has data changes; the step of acquiring target changed data in the current target data source according to data change state information corresponding to the target data source and a target data change strategy set bound to the target data source in the case where it is determined that the target data source has data changes comprises the steps of: determining a target data change strategy corresponding to the pending changed data that currently has data changes from the target data change strategy set according to a strategy configuration relationship between the pending changed data that currently has data changes and the data change strategy in the target data change strategy set; acquiring the data change state information corresponding to the pending changed data that currently has data changes, wherein the data change state information at least comprises data change quantity information, data change type information and data change duration of the pending changed data that currently has data changes; if the data change quantity information, the data change type information and the data change duration of the pending changed data that currently has data changes satisfy the target data change strategy, the pending changed data that currently has data changes is determined as the target changed data.

2. The data dynamic caching method of claim 1, wherein, The target data change strategy set further includes a time change strategy, the time change strategy includes a fixed time change strategy and a fixed cycle change strategy, wherein the fixed time change strategy is used to trigger a corresponding strategy operation when a preset time is reached, and the fixed cycle change strategy is used to trigger a corresponding strategy operation at the beginning of each preset cycle; the target data change strategy set includes at least two data change strategies and one fixed time change strategy or fixed cycle change strategy.

3. The data dynamic caching method according to any of claims 1 to 2, characterized in that, The method further includes: According to the data table information or the field information, the target data table or the target field corresponding to the data display content in the database is obtained; The target data table or the target field is taken as the target data source.

4. The data dynamic caching method of claim 3, wherein, Before determining the target data change strategy corresponding to the pending change data from the target data change strategy set according to the strategy configuration relationship between the pending change data and the data change strategy in the target data change strategy set, the method further includes: Obtain a plurality of preset data change strategy sets composed of different data change strategies; Based on the effective time of each preset data change strategy set, a time axis corresponding to all preset data change strategy sets is constructed; According to the position of the current corresponding time period on the time axis, the target data change strategy set is determined from the plurality of preset data change strategy sets.

5. The data dynamic caching method of claim 2, wherein, The data caching processing of the target change data is used to refresh the data display content in the visual large screen, including: After obtaining the target change data, a target data refresh request information is generated; Determine whether the target data refresh request information is the first data refresh request information generated in the current cycle, if it is the first data refresh request information generated in the current cycle, the target change data is subjected to data caching processing to obtain dynamic caching data; Based on the dynamic caching data, the data display content in the visual large screen is refreshed; If it is not the first data refresh request information generated in the current cycle, the target data refresh request information is deleted.

6. The data dynamic caching method of claim 1, wherein, The method further includes: When it is determined that the visual large screen is in the initialization startup stage, initial service data query is performed according to the data display content to obtain initial query data; Based on the initial query data, a data sedimentation operation is performed to obtain initial caching data; According to the initial caching data, the data display content in the visual large screen in the initialization startup stage is refreshed.

7. A data dynamic caching system, characterized by, The system is used to implement the data dynamic caching method according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the data dynamic caching method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the data dynamic caching method according to any one of claims 1 to 6.

Images