Data query method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a data query method and device, electronic equipment and a nonvolatile storage medium, and is applied to the technical field of storage. The method comprises the steps of generating a data synchronization request whenever it is detected that distributed cluster data changes; according to the data synchronization request, reading changed target distributed cluster data from the distributed cluster, and synchronizing the target distributed cluster data into a data cache of a target node when a data updating condition is met; wherein the data updating condition is that data flows from the distributed cluster to a target node, and the target node is a computer node receiving a client data query request in the distributed cluster; and when a data query request is received, querying corresponding target data from the data cache. According to the invention, the problems of high query response delay and large system resource consumption in related technologies can be solved, the query efficiency of distributed cluster data is effectively improved, and the resource consumption of a distributed storage system is reduced.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data query method, device, electronic equipment and non-volatile storage medium. Background Art

[0002] With the development of storage technology, distributed storage technology has been widely used. A distributed cluster consists of multiple computer nodes. The computer nodes communicate and collaborate with each other through the network to jointly complete a task or provide a service.

[0003] Related technologies query distributed cluster data in an event-driven manner. The query request needs to go through multiple intermediate links such as event triggering, status checking and data retrieval. This not only leads to a long data query time, slow response and poor real-time performance, but also each query needs to go through the event management module and cluster state machine, resulting in a large resource consumption of the distributed storage system.

[0004] In view of this, effectively improving the query efficiency of distributed cluster data and reducing the resource consumption of distributed storage systems are technical problems that technical personnel in this field need to solve.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present invention provides a data query method, device, electronic device and non-volatile storage medium, which can effectively improve the query efficiency of distributed cluster data and reduce the resource consumption of the distributed storage system.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In one aspect, the present invention provides a data query method, comprising:

[0009] Whenever a distributed cluster data change is detected, a data synchronization request is generated;

[0010] According to the data synchronization request, the changed target distributed cluster data is read from the distributed cluster, and if the data update condition is met, the target distributed cluster data is synchronized to the data cache of the target node; wherein the data update condition is that the data flows from the distributed cluster to the target node, and the target node is a computer node in the distributed cluster that receives the client data query request;

[0011] When a data query request is received, the corresponding target data is queried from the data cache.

[0012] In a first exemplary implementation, whenever a distributed cluster data change is detected, the method includes:

[0013] Pre-build a monitoring command library, the monitoring command library includes multiple monitoring commands, and the monitoring command uniquely corresponds to one data operation mode;

[0014] Based on the monitoring command library, a matching monitoring command is used to monitor data operations performed by a cluster state machine of the distributed cluster.

[0015] In a second exemplary implementation, the generating a data synchronization request includes:

[0016] Assign unique indexes to each data item of distributed cluster data in advance;

[0017] A data synchronization request is generated according to a target index of the changed target distributed cluster data, so as to read the target distributed cluster data from a corresponding computer node of the distributed cluster through the target index.

[0018] In a third exemplary implementation, after synchronizing the target distributed cluster data to the data cache of the target node, the method further includes:

[0019] Pre-adding a user identifier and a timestamp to the stored data in the data cache as a cache key;

[0020] Obtaining the access frequency, access time and user location of each stored data of the data cache at the current moment to obtain data access characteristics and distributed cluster system load characteristics;

[0021] Determine expired data according to the cache key and access time of each stored data in the data cache, and delete the expired data from the data cache;

[0022] The stored data in the data cache are dynamically adjusted according to the data access characteristics and the distributed cluster system load characteristics.

[0023] In a fourth exemplary implementation, the step of reading the changed target distributed cluster data from the distributed cluster includes:

[0024] Obtain current business demand characteristics, distributed cluster data volume, and data change frequency;

[0025] Matching a data synchronization method according to the current business demand characteristics, the amount of distributed cluster data, and the frequency of data changes;

[0026] According to the data synchronization method, the target distributed cluster data is read from the distributed cluster.

[0027] In a fifth exemplary implementation, whenever a distributed cluster data change is detected, the method includes:

[0028] Decompose the current distributed cluster data into multiple time slices according to the time series;

[0029] Based on different topological parameters, a topological structure is constructed for describing the connection relationship between data points for each time slice;

[0030] Determine the topological feature information of the current distributed cluster data according to the topological features of each time slice and the corresponding topological parameters;

[0031] A change point in a time series corresponding to the current distributed cluster data is determined based on the topological feature information.

[0032] In a sixth exemplary implementation, determining a change point in a time series corresponding to the current distributed cluster data based on the topological feature information includes:

[0033] Removing outliers in the data sequence corresponding to the topological feature information and filling missing values ​​in the data sequence to obtain an initial data sequence;

[0034] Performing standardization processing on the initial data sequence to obtain a target data sequence;

[0035] Get the preset sliding window length;

[0036] According to the sliding window length, a window is slid on the target data sequence, and the mean and variance of the data in each sliding window are calculated as the data statistical features of each sliding window;

[0037] By comparing the statistical features of the data corresponding to the adjacent front and rear sliding windows, the suspected change point position is determined as the candidate change point position;

[0038] The likelihood ratio test method is used to perform statistical test processing on each candidate position of the change point to determine the false change point;

[0039] Remove false change points from each change point candidate position to obtain change point position estimation data;

[0040] Density analysis is performed on the change point position estimation data using kernel density estimation to determine the target change point position.

[0041] Another aspect of the present invention provides a data query device, comprising:

[0042] The data monitoring module is used to generate a data synchronization request whenever a distributed cluster data change is detected;

[0043] A data synchronization module, used to read the changed target distributed cluster data from the distributed cluster according to the data synchronization request, and synchronize the target distributed cluster data to the data cache of the target node when the data update condition is met; wherein the data update condition is that the data flows from the distributed cluster to the target node, and the target node is a computer node in the distributed cluster that receives the client data query request;

[0044] The data query module is used to query the corresponding target data from the data cache when a data query request is received.

[0045] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the data query method as described in any of the preceding items when executing a computer program stored in a memory.

[0046] Finally, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data query method as described in any of the preceding items are implemented.

[0047] The advantage of the technical solution provided by the present invention is that by setting cache data on the target node responsible for responding to the client query request, the latest data of the distributed cluster data is synchronized to the target node in real time, the query operation is completed directly on the target node, the data transmission path is optimized, and the intermediate processing links for responding to the query request are reduced. This not only avoids the interaction of multiple modules during the query process, reduces the query response time, and realizes the fast reading and writing of distributed cluster data, but also reduces the burden of components such as the cluster state machine, reduces system resource consumption, ensures that the data queried by the client is the latest data, and improves the accuracy and efficiency of data query. Furthermore, caching the distributed cluster data to the target node can also ensure the high reliability of the data synchronization process, avoid data loss from causing losses to the business of the distributed storage system, and improve the security and stability of the distributed storage system.

[0048] In addition, the present invention also provides a corresponding implementation device, electronic device and non-volatile storage medium for the data query method, which further makes the method more practical, and the device, electronic device and non-volatile storage medium have corresponding advantages.

[0049] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a data query method of the related technology;

[0052] Figure 2 A flow chart of a data query method provided by the present invention;

[0053] Figure 3 A schematic diagram of the structural framework of the data query system provided by the present invention;

[0054] Figure 4 A schematic diagram of a flow chart of another data query method provided by the present invention;

[0055] Figure 5 A structural diagram of a specific implementation of the data query device provided by the present invention;

[0056] Figure 6 A structural diagram of a specific implementation of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first" and "second" in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0058] In related technologies, cluster data query methods are as follows: Figure 1 As shown in the figure, the client sends a query request to the management node. After receiving the query request, the management node sends a query event request to the event management module. The event management module receives the event query request and distributes the event to the distributed cluster. After receiving the event, the cluster state machine queries the cluster data and sends the query result to the management node. After receiving the feedback data, the management node returns the query result to the client. It can be seen that the cluster data query process involves the interaction of multiple modules, which requires multiple intermediate links, including event triggering, status checking and data retrieval. These links increase the query delay, slow query speed, and long query response time. In scenarios where data is updated frequently, the data queried by the client may be delayed and the real-time performance is poor. Each query needs to go through multiple components such as the event management module and the cluster state machine, which consumes a lot of system resources. Especially in high-concurrency scenarios, resource consumption is particularly prominent, which increases the burden on the distributed storage system.

[0059] In view of this, the present invention performs data caching in the management node that responds to the query request of the client, and performs data synchronization update by monitoring the changes of the distributed cluster data and based on the access control method, so as to ensure that the cluster data cached by the management node is updated in real time, and realize fast, efficient and low resource consumption cluster data query. It is not only conducive to improving the overall performance of the distributed storage system and providing users with a smoother and real-time data service experience, but also takes into account the scalability and maintainability of the distributed storage system to ensure its long-term operation stability and reliability. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0060] First see Figure 2 , Figure 2 A flowchart of a data query method provided in this embodiment may include the following contents:

[0061] S201: Whenever a distributed cluster data change is detected, a data synchronization request is generated.

[0062] In this embodiment, the distributed cluster is managed by a distributed storage system, which includes multiple computer nodes, a transaction manager, and a cluster state machine. Among the multiple computer nodes, there is a computer node that responds to the client querying cluster data, which is defined as the target node in this embodiment. The data query system composed of the client and the distributed cluster is as follows: Figure 3As shown, the present invention is applicable to a client querying data of a distributed cluster.

[0063] S202: Reading changed target distributed cluster data from the distributed cluster according to the data synchronization request, and synchronizing the target distributed cluster data to the data cache of the target node if the data update condition is met.

[0064] Once the previous step detects a change in the distributed cluster data, such as data creation, update and deletion operations, a data synchronization request indicating the data change will be generated. Based on the data synchronization request, this step reads data from the corresponding location according to the corresponding data synchronization strategy, and defines the data that needs to be synchronized as the target distributed cluster data. After reading the target distributed cluster data and synchronizing the target distributed cluster data to the target node, the target node will first check whether the data update condition is met. The data update condition is that the data flows from the distributed cluster to the target node, that is, if it is cluster data sent by the client or a computing node, it is not allowed to be synchronized to the data cache. The data cache is the storage space of the computer node that receives the client data query request in the distributed cluster. A storage space can be applied in advance in the memory or external storage of the target node to store the cached distributed cluster data. Furthermore, the node responsible for executing the data synchronization request can remain silent when there is no data change notification, that is, when no data synchronization request is received, and does not consume system resources. Only after receiving the data synchronization request, it is activated and ready to synchronize data, further reducing the consumption of system resources. Furthermore, during the data synchronization process, corresponding fault handling mechanisms can be designed for network failures, node failures, etc. to ensure the stability of data synchronization. In order to ensure data security, during the data synchronization process, data can also be encrypted to prevent data leakage.

[0065] S203: When a data query request is received, corresponding target data is searched from the data cache.

[0066] When the client sends a query request to the distributed cluster, the data query request carries the relevant information of the data to be queried, the target node obtains the data query request, determines the data to be queried by parsing the data query request, and then retrieves the data from its own data cache. The data cache stores all and the latest queryable data items in the distributed cluster, reducing the time complexity of data retrieval, and can quickly retrieve the data to respond to the query request, that is, the target data defined in this step is fed back to the client as the query response result. It can be seen from this that when the client queries the cluster data, the present invention only needs to query the data cache to complete the link, and can retrieve the latest data as quickly as possible and feed it back to the client.

[0067] In the technical solution provided in this embodiment, by setting cache data on the target node responsible for responding to the client's query request, the latest data of the distributed cluster data is synchronized to the target node in real time, and the query operation is completed directly on the target node, optimizing the data transmission path and reducing the intermediate processing links for responding to the query request. This not only avoids the interaction of multiple modules during the query process, reduces the query response time, and realizes the fast reading and writing of distributed cluster data, but also reduces the burden on components such as the cluster state machine, reduces system resource consumption, ensures that the data queried by the client is the latest data, and improves the accuracy and efficiency of data query. Furthermore, caching the distributed cluster data to the target node can also ensure the high reliability of the data synchronization process, avoid data loss from causing losses to the business of the distributed storage system, and improve the security and stability of the distributed storage system.

[0068] Based on the above embodiment, in order to avoid affecting the normal business process of the distributed storage system, based on the above embodiment, the present invention also provides an exemplary monitoring method, which may include the following contents:

[0069] A monitoring command library is pre-built, the monitoring command library includes multiple monitoring commands, and the monitoring command uniquely corresponds to a data operation mode; based on the monitoring command library, a matching monitoring command is used to monitor the data operation performed by the cluster state machine of the distributed cluster.

[0070] In this embodiment, each listening command in the listening command library can be set according to the actual situation, and the present invention does not make any limitation on this. Data operations such as querying volumes, querying hosts, uploading data, deleting data, etc. use different listening commands to monitor different data operation commands, thereby isolating each command, so that the interference with the normal business process of the distributed storage system will be reduced to interference, ensuring that the monitoring operation of the distributed cluster data will not affect the stable operation of the cluster. Further, through isolated command monitoring, when the scale of the distributed storage system becomes larger, in order to ensure the monitoring effect, only the corresponding monitoring nodes need to be added, which can adapt to the ever-changing business needs and is conducive to the horizontal expansion of the distributed cluster.

[0071] Based on the above embodiments, in order to further improve the data query efficiency, the present invention also provides the following implementation method:

[0072] A unique index is pre-assigned to each data item of the distributed cluster data; and a data synchronization request is generated according to the target index of the changed target distributed cluster data to read the target distributed cluster data from the corresponding computer node of the distributed cluster through the target index.

[0073] In this embodiment, the target index is an index of the target distributed cluster data. If there are multiple target distributed cluster data, there are multiple corresponding target indexes.

[0074] After receiving the modification event from the business module, the transaction manager will distribute the modification event to the cluster. After receiving the event, the cluster state machine will modify the cluster data. In order to quickly locate data changes, a unique index is assigned to each data item. After the application monitors the cluster data changes, it initiates a data synchronization request to the data synchronization module. The data synchronization request contains the index of the changed data so that it can retrieve the corresponding data from the distributed cluster data as soon as possible, which is convenient for quickly responding to the data synchronization request.

[0075] As can be seen from the above, this embodiment can quickly define changed data, improve data query efficiency, and further reduce query request response delay by assigning indexes to data items.

[0076] In order to further improve the data query efficiency, based on the above embodiments, the present invention also provides the following embodiments:

[0077] Pre-add a user identifier and a timestamp to the stored data in the data cache as a cache key; obtain the access frequency, access time and user location of each stored data in the data cache at the current moment to obtain data access characteristics and distributed cluster system load characteristics; determine expired data based on the cache key and access time of each stored data in the data cache, and delete the expired data from the data cache; dynamically adjust each stored data in the data cache based on the data access characteristics and the distributed cluster system load characteristics.

[0078] In this embodiment, in order to avoid cache penetration and cache avalanche, a unique cache key can also be set for each stored data in the data cache data, and the cache key can be generated based on at least parameters such as user identification and timestamp. Data access patterns and system load conditions are identified by data access frequency, access time, user location, etc. The freshness and effectiveness of the cache content are ensured by clearing the cache automatic expiration time and cache data that is about to expire. Furthermore, a dynamic cache adjustment strategy can be set, such as dynamically adjusting the cache life span according to the content update frequency, setting a longer life span for content with a low update frequency, and setting a shorter life span for content with high real-time requirements. In other words, before cleaning the cache data, it is necessary to consider the application scenario of the distributed storage system, so as to balance cache efficiency and query accuracy.

[0079] It can be seen from the above that this embodiment can effectively adjust the cache content dynamically according to the data access mode and system load, taking into account both the data query efficiency and the validity of the cached data, and further saving system storage resources and computing resources.

[0080] In order to further improve the flexibility of data query, based on the above embodiment, the present invention further provides the following embodiment, which may include the following contents:

[0081] Acquire current business demand characteristics, distributed cluster data volume and data change frequency; match the data synchronization method according to the current business demand characteristics, the distributed cluster data volume and the data change frequency; read the target distributed cluster data from the distributed cluster according to the data synchronization method.

[0082] In this embodiment, the business demand characteristics are to reflect the changes in the stored data caused by the current business. If the business is frequent in writing data or frequent in reading data, the update frequency and update amount of the distributed storage cluster data can be determined according to different business demand characteristics. Data synchronization methods may include incremental synchronization, full synchronization and hybrid synchronization. Incremental updates only process changed data, reduce resource consumption, improve efficiency, and are suitable for scenarios with large data volumes and frequent updates. Full updates are used when major changes occur in the data model to ensure data consistency, but they consume large resources and are suitable for scenarios with small data volumes or low update frequencies. Hybrid updates combine the advantages of incremental updates and full updates. For example, incremental updates are used daily to maintain the timeliness of data, and full updates are performed regularly to ensure data integrity and consistency.

[0083] As can be seen from the above, this embodiment matches the most appropriate data synchronization method based on the business and the data itself, has better flexibility, and can balance the real-time performance of data update and system resource consumption.

[0084] Furthermore, in order to better balance the real-time performance of data updates and system resource consumption, on the basis of matching the most appropriate data synchronization method and cache update method, the present invention can further optimize the data structure of each stored data in the data cache, which may include the following contents:

[0085] A hash function can be selected from related technologies or designed according to actual scenarios. The function can map any key to an index position in the hash table on the basis of being able to evenly distribute keys to reduce conflicts. Then create an array of a certain size to store the hash table, and a linked list or other data structure to resolve hash conflicts to complete the initialization of the hash table. Whenever the target distributed cluster data enters the data cache, the hash function is used to calculate the hash value of the key corresponding to the target distributed cluster data to determine its position in the hash table. If the position is already occupied, that is, a conflict occurs, and a conflict resolution strategy such as the chain address method, open addressing method, etc. is used to handle the conflict. If it is not occupied, the key-value pair corresponding to the target distributed cluster data is inserted into the calculated position. When the target node searches for data from the data cache, for the sake of ease of description, it can be defined as the data to be queried, and the hash function is used to calculate the hash value of the key corresponding to the data to be queried, and the corresponding position in the hash table is accessed. If the position is empty, the key does not exist; if it is not empty, check whether the key at the position matches the key to be found. If a conflict occurs, traverse the conflict resolution linked list until a matching key is found or the linked list ends. Taking into account the expansion of the scale of the distributed cluster and ensuring data query efficiency, this embodiment can also expand the hash table. When the load factor of the hash table exceeds a preset threshold, the load factor is the ratio of the number of stored elements to the size of the hash table to expand the hash table, such as creating a new, larger hash table and re-hashing all elements into the new table.

[0086] It can be seen from the above that this embodiment not only improves data query efficiency by optimizing the data structure, but also ensures high data query efficiency when the scale of the distributed cluster increases.

[0087] In order to further improve the efficiency of changed data detection, based on the above embodiments, the present invention further provides the following embodiments, which may include the following contents:

[0088] Decompose the current distributed cluster data into multiple time slices according to the time series; construct a topological structure for describing the connection relationship between data points for each time slice based on different topological parameters; determine the topological feature information of the current distributed cluster data according to the topological features of each time slice and the corresponding topological parameters; determine the change point in the time series corresponding to the current distributed cluster data based on the topological feature information.

[0089] In this embodiment, the time series is decomposed into a series of time slices, so that the continuous time series data is divided into discrete slices that can be analyzed at each time point. On each time slice, in order to further improve the connection relationship between data points and get closer to the topological structure of the underlying space, a Vietoris-Rips (topological structure name) complex with different diameters ε, i.e., topological parameters, can be used to construct a topological structure. Betti numbers and persistence values ​​can be calculated on each time slice as topological features. Among them, the Betti number is used to represent the invariant used to describe the connectivity of the topological space in the topology, which can represent the number of connected branches, and the persistence value is used to describe the persistence of topological features at different scales, which can be the duration or length of each homology class. The topological features can be visualized, based on the Betti number as the number of horizontal lines intersecting the vertical line, and the persistence as the persistence length of each homology class [𝑧], the topological features are represented in the form of barcodes. According to the Betti number and persistence value of all time slices, the corresponding Betti number matrix and persistence matrix are generated. These two matrices contain the topological feature information of all time slices, providing a data basis for subsequent change point detection, and any change point detection method can be used to obtain change point estimation. In other words, using the previously obtained Betti number matrix and persistence matrix, the change point detection algorithm is used to identify the change points in the time series. The change point detection method is to identify the state changes of cluster motion according to the time series, and can identify the generation parameter mutations of cluster sequence data.

[0090] As a simpler change point detection method, this embodiment also provides an exemplary change point detection method, which may include the following content:

[0091] The outliers in the data sequence corresponding to the topological feature information are removed, and the missing values ​​in the data sequence are filled to obtain the initial data sequence, and the initial data sequence is standardized to obtain the target data sequence. The data is preprocessed through these two steps to improve the accuracy of change point detection. The preset sliding window length is obtained, and the sliding window length is determined based on the sensitivity and computational complexity of the detection. According to the sliding window length, the window is slid on the target data sequence, and the mean and variance of the data in each sliding window are calculated as the data statistical features of each sliding window; by comparing the data statistical features corresponding to the adjacent front and rear sliding windows, the suspected change point position is determined as the change point candidate position; the statistical test processing of each change point candidate position is performed by the likelihood ratio test method to determine whether there is a change point, so as to determine the false change point. False change points are removed from each change point candidate position to obtain change point position estimation data; of course, when performing statistical tests on the change point candidate positions, methods such as t test and F test can also be used. Finally, kernel density estimation is used to perform density analysis on the change point position estimation data to determine the target change point position. The target change point position is the accurate change point position that is finally determined. When refining the change point position, methods such as PELT (Pruned Exact Linear Time, change point detection method), difference method, virtual variable method, and regional transformation model can also be used to determine it, which does not affect the implementation of the present invention.

[0092] As can be seen from the above, this embodiment can ensure that the change data of large-scale distributed cluster data is captured in real time through the above-mentioned efficient data change detection method, thereby further improving the data query efficiency.

[0093] In order to make the cluster data query process of the present invention more clear to those skilled in the art, the present invention also combines Figure 4 An exemplary data query process is provided, which may include the following contents:

[0094] Before the client sends a data query request, the distributed storage system performs the following steps:

[0095] a. After receiving the modification event from the business module, the transaction manager distributes the modification event to the cluster.

[0096] b. After receiving the event, the cluster state machine modifies the cluster data.

[0097] c. The data monitoring module, after monitoring the cluster data changes, initiates a data synchronization request with an index to the data synchronization module.

[0098] d. After receiving the data synchronization request, the data synchronization module reads the target distributed cluster data based on the index and synchronizes the distributed cluster data to the data cache module through access control.

[0099] The query response process of cluster data may include the following steps:

[0100] (1) Query request initiation: The client sends a data query request to the target node, and the request contains the identification information of the required data.

[0101] (2) Query processing: After receiving the query request, the target node directly accesses the data cache module. Since the data cache module has stored the latest cluster data, it can quickly return the query results, greatly reducing the response time. In actual tests, the response time of data queries is more than 50% lower than that of related technologies, significantly improving the user experience. By optimizing the data synchronization and caching mechanism, the stability of the system has been greatly improved and the failure rate has been reduced by 30%. By reducing unnecessary resource occupation, the system resource utilization rate has been increased by 20%, reducing the operation and maintenance costs.

[0102] As can be seen from the above, the query operation of this embodiment is completed directly on the management node, which can not only ensure that the client obtains the latest query data as quickly as possible, but also reduce the burden on components such as the cluster state machine and reduce the resource consumption of the distributed storage system.

[0103] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1 , Figure 2 and Figure 4 This is just a schematic and does not mean that this is the only execution order.

[0104] The present invention also provides a corresponding device for the data query method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The data query device provided by the present invention is introduced below. The device is used to implement the data query method provided by the present invention. In this embodiment, the data query device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the data query method disclosed in Example 1. The program module referred to in this embodiment refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of the data query device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The data query device described below and the data query method described above can be referenced to each other.

[0105] From the perspective of functional modules, see Figure 5 , Figure 5The structure diagram of the data query device provided in this embodiment under a specific implementation mode, the device may include:

[0106] The data monitoring module 501 is used to generate a data synchronization request whenever a distributed cluster data change is detected.

[0107] The data synchronization module 502 is used to read the changed target distributed cluster data from the distributed cluster according to the data synchronization request, and synchronize the target distributed cluster data to the data cache of the target node when the data update condition is met; wherein the data update condition is that the data flows from the distributed cluster to the target node, and the target node is a computer node in the distributed cluster that receives the client data query request.

[0108] The data query module 503 is used to query the corresponding target data from the data cache when a data query request is received.

[0109] For example, in some implementations of this embodiment, the data monitoring module 501 may also be used to:

[0110] Pre-build a monitoring command library, the monitoring command library includes multiple monitoring commands, and the monitoring command uniquely corresponds to one data operation mode;

[0111] Based on the monitoring command library, a matching monitoring command is used to monitor data operations performed by a cluster state machine of the distributed cluster.

[0112] For example, in some other implementations of this embodiment, the data monitoring module 501 may be further used for:

[0113] Assign unique indexes to each data item of distributed cluster data in advance;

[0114] A data synchronization request is generated according to a target index of the changed target distributed cluster data, so as to read the target distributed cluster data from a corresponding computer node of the distributed cluster through the target index.

[0115] Exemplarily, in some other implementations of this embodiment, the above device may further include a cache update module, which may be further used to:

[0116] Pre-adding a user identifier and a timestamp to the stored data in the data cache as a cache key;

[0117] Obtaining the access frequency, access time and user location of each stored data of the data cache at the current moment to obtain data access characteristics and distributed cluster system load characteristics;

[0118] Determine expired data according to the cache key and access time of each stored data in the data cache, and delete the expired data from the data cache;

[0119] The stored data in the data cache are dynamically adjusted according to the data access characteristics and the distributed cluster system load characteristics.

[0120] Illustratively, in some other implementations of this embodiment, the data query module 503 may be further used to:

[0121] Obtain current business demand characteristics, distributed cluster data volume, and data change frequency;

[0122] Matching a data synchronization method according to the current business demand characteristics, the amount of distributed cluster data, and the frequency of data changes;

[0123] According to the data synchronization method, the target distributed cluster data is read from the distributed cluster.

[0124] For example, in some other implementations of this embodiment, the data monitoring module 501 may be further used for:

[0125] Decompose the current distributed cluster data into multiple time slices according to the time series;

[0126] Based on different topological parameters, a topological structure is constructed for describing the connection relationship between data points for each time slice;

[0127] Determine the topological feature information of the current distributed cluster data according to the topological features of each time slice and the corresponding topological parameters;

[0128] A change point in a time series corresponding to the current distributed cluster data is determined based on the topological feature information.

[0129] As an exemplary implementation of the above embodiment, the data monitoring module 501 may be further used for:

[0130] Removing outliers in the data sequence corresponding to the topological feature information and filling missing values ​​in the data sequence to obtain an initial data sequence;

[0131] Performing standardization processing on the initial data sequence to obtain a target data sequence;

[0132] Get the preset sliding window length;

[0133] According to the sliding window length, a window is slid on the target data sequence, and the mean and variance of the data in each sliding window are calculated as the data statistical features of each sliding window;

[0134] By comparing the statistical features of the data corresponding to the adjacent front and rear sliding windows, the suspected change point position is determined as the candidate change point position;

[0135] The likelihood ratio test method is used to perform statistical test processing on each candidate position of the change point to determine the false change point;

[0136] Remove false change points from each change point candidate position to obtain change point position estimation data;

[0137] Density analysis is performed on the change point position estimation data using kernel density estimation to determine the target change point position.

[0138] The data query device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 6 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Figure 6 As shown, the electronic device includes a memory 60 for storing a computer program; and a processor 61 for implementing the steps of the data query method mentioned in any of the above embodiments when executing the computer program.

[0139] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 61 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 61 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0140] The memory 60 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 60 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 60 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 60 may also include both an internal storage unit of an electronic device and an external storage device. The memory 60 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the data query method, etc., but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, the relevant steps of the data query method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include but is not limited to data corresponding to the data query result.

[0141] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64 or a network interface, a power supply 65 and a communication bus 66. Among them, the display screen 62 and the input / output interface 63, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 64 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 66 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 67 to implement various functions.

[0143] It is understandable that if the data query method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data query method recorded in any of the above embodiments are performed.

[0144] It is understandable that if the data query method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, it performs the steps of the data query method described in any of the above embodiments.

[0145] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices, electronic devices, and non-volatile storage media, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0146] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] The above is a detailed introduction to a data query method, device, electronic device and non-volatile storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data query method, characterized in that: include: Whenever a distributed cluster data change is detected, a data synchronization request is generated; According to the data synchronization request, the changed target distributed cluster data is read from the distributed cluster, and if the data update condition is met, the target distributed cluster data is synchronized to the data cache of the target node; wherein the data update condition is that the data flows from the distributed cluster to the target node, and the target node is a computer node in the distributed cluster that receives the client data query request; When a data query request is received, the corresponding target data is queried from the data cache.

2. The data query method according to claim 1, characterized in that: Whenever a distributed cluster data change is detected, it includes: Pre-build a monitoring command library, the monitoring command library includes multiple monitoring commands, and the monitoring command uniquely corresponds to one data operation mode; Based on the monitoring command library, a matching monitoring command is used to monitor data operations performed by a cluster state machine of the distributed cluster.

3. The data query method according to claim 1, characterized in that: The generating of the data synchronization request comprises: Assign unique indexes to each data item of distributed cluster data in advance; A data synchronization request is generated according to a target index of the changed target distributed cluster data, so as to read the target distributed cluster data from a corresponding computer node of the distributed cluster through the target index.

4. The data query method according to claim 1, characterized in that: After synchronizing the target distributed cluster data to the data cache of the target node, the method further includes: Pre-adding a user identifier and a timestamp to the stored data in the data cache as a cache key; Obtaining the access frequency, access time and user location of each stored data of the data cache at the current moment to obtain data access characteristics and distributed cluster system load characteristics; Determine expired data according to the cache key and access time of each stored data in the data cache, and delete the expired data from the data cache; The stored data in the data cache are dynamically adjusted according to the data access characteristics and the distributed cluster system load characteristics.

5. The data query method according to claim 1, characterized in that: The step of reading the changed target distributed cluster data from the distributed cluster includes: Obtain current business demand characteristics, distributed cluster data volume, and data change frequency; Matching a data synchronization method according to the current business demand characteristics, the amount of distributed cluster data, and the frequency of data changes; According to the data synchronization method, the target distributed cluster data is read from the distributed cluster.

6. The data query method according to any one of claims 1 to 5, characterized in that: Whenever a distributed cluster data change is detected, it includes: Decompose the current distributed cluster data into multiple time slices according to the time series; Based on different topological parameters, a topological structure is constructed for describing the connection relationship between data points for each time slice; Determine the topological feature information of the current distributed cluster data according to the topological features of each time slice and the corresponding topological parameters; A change point in a time series corresponding to the current distributed cluster data is determined based on the topological feature information.

7. The data query method according to claim 6, characterized in that: The determining, based on the topological feature information, a change point in a time series corresponding to the current distributed cluster data includes: Removing outliers in the data sequence corresponding to the topological feature information and filling missing values ​​in the data sequence to obtain an initial data sequence; Performing standardization processing on the initial data sequence to obtain a target data sequence; Get the preset sliding window length; According to the sliding window length, a window is slid on the target data sequence, and the mean and variance of the data in each sliding window are calculated as the data statistical features of each sliding window; By comparing the statistical features of the data corresponding to the adjacent front and rear sliding windows, the suspected change point position is determined as the candidate change point position; The likelihood ratio test method is used to perform statistical test processing on each candidate position of the change point to determine the false change point; Remove false change points from each change point candidate position to obtain change point position estimation data; Density analysis is performed on the change point position estimation data using kernel density estimation to determine the target change point position.

8. A data query device, characterized in that: include: The data monitoring module is used to generate a data synchronization request whenever a distributed cluster data change is detected; A data synchronization module, used to read the changed target distributed cluster data from the distributed cluster according to the data synchronization request, and synchronize the target distributed cluster data to the data cache of the target node when the data update condition is met; wherein the data update condition is that the data flows from the distributed cluster to the target node, and the target node is a computer node in the distributed cluster that receives the client data query request; The data query module is used to query the corresponding target data from the data cache when a data query request is received.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the data query method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data query method according to any one of claims 1 to 7 are implemented.

Images