Cache data statistical analysis method

Abstract

The invention discloses a cache data statistical analysis method. The method comprises the following steps that a user sends a POST request; judging a statistical type of request data sent by a user, and connecting a cache instance to obtain all cluster nodes; reading data information in stages, and storing the data information in a map; merging the data information in the map, and classifying the data information according to library and data types; and storing an aggregation result to a database and sending the aggregation result to a user. According to the method, the high-concurrency characteristic is utilized, multiple nodes are rapidly and simultaneously subjected to statistical analysis, and the effect can be visually checked. According to the method, all nodes in the cluster can be counted through one request, full automation is achieved, and manual intervention is not needed; a go coroutine is adopted, and data statistics and result aggregation are carried out in a high-concurrency mode; and an interface is provided, so that trigger / result feedback is more elegant, and integrated display with a front end is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of software operation and maintenance, and in particular to a cache data statistical analysis method. Background Art

[0002] Redis (Remote Dictionary Server) is an open source, high-performance key-value storage system. It can not only be used as a database, cache, and message middleware, but also because of its characteristics, it is particularly suitable for caching in distributed environments.

[0003] Redis officially provides a single-node cache large key and hot key statistics tool, but there are the following problems: a. Only one node can be counted at a time. For clusters, each master node must be executed once, which is inefficient. Especially in container scenarios, you need to enter the container first and then execute, which is manual and prone to errors.

[0004] b. The statistical results are simple. The displayed results do not indicate which database the key is in, and only show the largest keys in text format, which is not easy to read.

[0005] c. No interface is provided, it is executed by local binary files, and it is not easy to integrate with the front-end for graphical display. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a cache data statistical analysis method, comprising the following steps: S1. The user sends a POST request; S2. Determine the statistical type of the request data sent by the user, connect to the cache instance, and obtain all cluster nodes; S3. Start the go coroutine, perform statistical operations on each node, and calculate the key size; S4. Start the go coroutine, perform statistical operations on each library, and obtain the number of key accesses; S5. Read data information in stages and store them in a map; S6. Merge the data information in the map and classify them by library and data type; S7. Save the aggregation results to the database and send them to the user.

[0007] Furthermore, the step S2 includes the following sub-steps: when the statistical type of the request data sent by the user is a large key, executing step S3; when the statistical type of the request data sent by the user is a hot key, executing step S4.

[0008] Furthermore, in the step S2, when the statistical type of the request data sent by the user is slow query data, the cache interface is called and the log field is parsed to obtain data information, and step S5 is executed.

[0009] Furthermore, the S3 step specifically includes the following sub-steps: S31. Arrange the keys in reverse order according to the byte size; S32. Take the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, the library, the type, the key name, and the size of the occupied bytes.

[0010] Furthermore, the step S4 specifically includes the following sub-steps: S41. Arrange the keys in reverse order according to the access frequency; S42. Take the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, the library, the type, the key name, and the number of accesses.

[0011] Furthermore, when the request data statistics type sent by the user is slow query data, the method specifically includes the following sub-steps: arranging the keys in reverse order according to the trigger time, and taking the first 20 data information for statistics; wherein the data information specifically includes: the node, time, execution duration, IP port, and execution command.

[0012] Furthermore, the data information read in step S5 specifically includes: key name, data type, data occupied size, and access frequency.

[0013] The present invention provides a cache data statistical analysis method, which has the following beneficial effects: The present invention uses the characteristics of high concurrency to quickly perform statistical analysis on multiple nodes at the same time, and the effect can be viewed intuitively. In the present invention, statistics can be performed on all nodes in the cluster through one request, which is fully automatic and does not require manual intervention; go coroutines are used to perform data statistics and result aggregation with high concurrency; an interface is provided to make triggering / result feedback more elegant and convenient for integration and display with the front end. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0015] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0016] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0017] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.

[0018] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0019] like Figure 1 As shown, the present invention provides a cache data statistical analysis method, comprising the following steps: S1. The user sends a POST request; S2. Determine the statistical type of the request data sent by the user, connect to the cache instance, and obtain all cluster nodes; S3. Start the go coroutine, perform statistical operations on each node, and calculate the key size; S4. Start the go coroutine, perform statistical operations on each library, and obtain the number of key accesses; S5. Read data information in stages and store them in a map; S6. Merge the data information in the map and classify them by library and data type; S7. Save the aggregation results to the database and send them to the user.

[0020] Among them, step S2 contains the following sub-steps: when the statistical type of the request data sent by the user is a large key, execute step S3; when the statistical type of the request data sent by the user is a hot key, execute step S4. In step S2, when the statistical type of the request data sent by the user is slow query data, call the cache interface and parse the log field to obtain data information, and execute step S5.

[0021] For a single node, a go coroutine is started for each library, and all data in this library is concurrently counted and saved in the map to improve the execution speed. For each data structure type, such as string, list, hash, zset, set, a go coroutine is started, and all data of this type is concurrently counted and saved in the map.

[0022] Step S3 specifically includes the following sub-steps: S31. Arrange the keys in reverse order according to the byte size; S32. Take the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, the library, the type, the key name, and the size of the occupied bytes.

[0023] Step S4 specifically includes the following sub-steps: S41. Arrange the keys in reverse order according to the access frequency; S42. Take the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, the library, the type, the key name, and the number of accesses.

[0024] When the request data statistics type sent by the user is slow query data, the following sub-steps are specifically included: the keys are arranged in reverse order according to the trigger time, and the first 20 data information are taken for statistics; the data information specifically includes: the node, time, execution duration, IP port, and execution command.

[0025] The data information read in step S5 specifically includes: key name, data type, data size, and access frequency.

[0026] The present invention uses the characteristics of high concurrency to quickly perform statistical analysis on multiple nodes at the same time, and the effect can be viewed intuitively. In the present invention, statistics can be performed on all nodes in the cluster through one request, which is fully automatic and does not require manual intervention; go coroutines are used to perform data statistics and result aggregation with high concurrency; an interface is provided to make triggering / result feedback more elegant and convenient for integration and display with the front end.

[0027] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A cache data statistical analysis method, It is characterized in that The following steps are involved: S1. User sends a POST request; S2. Determine the statistical type of the request data sent by the user, connect to the cache instance, and obtain all cluster nodes; S3. Start the go coroutine, perform statistical operations on each node, and calculate the key size; S4. Start the go coroutine, perform statistical operations on each library, and obtain the number of key accesses; S5. Read data information in stages and store it in the map; S6. Merge the data information in the map and classify them by library and data type; S7. Save the aggregation results to the database and send them to the user.

2. The cache data statistical analysis method according to claim 1, It is characterized in that The step S2 includes the following sub-steps: when the statistical type of the request data sent by the user is a large key, executing step S3; when the statistical type of the request data sent by the user is a hot key, executing step S4.

3. According to the cache data statistical analysis method of claim 2, It is characterized in that In the step S2, when the statistical type of the request data sent by the user is slow query data, the cache interface is called and the log field is parsed to obtain data information, and step S5 is executed.

4. According to the cache data statistical analysis method of claim 2, It is characterized in that The S3 step specifically includes the following sub-steps: S31. Arrange the keys in reverse order according to the byte size; S32. Take the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, the library, the type, the key name, and the size of the occupied bytes.

5. According to the cache data statistical analysis method of claim 2, It is characterized in that The step S4 specifically includes the following sub-steps: S41. Arrange the keys in reverse order according to the access frequency; S42. Take the first 20 pieces of data information for statistics; the data information specifically includes: the node, the library, the type, the key name, and the number of accesses.

6. According to the cache data statistical analysis method of claim 3, It is characterized in that When the statistical type of the request data sent by the user is slow query data, the method specifically includes the following sub-steps: arranging the keys in reverse order according to the trigger time, and taking the first 20 pieces of data information for statistics; wherein the data information specifically includes: the node, time, execution duration, IP port, and execution command.

7. The cache data statistical analysis method according to claim 1, It is characterized in that The data information read in step S5 specifically includes: key name, data type, data occupied size, and access frequency.

Images