Storage data statistical method and device and storage medium

Abstract

The invention discloses a storage data statistical method and device and a storage medium. The method comprises the steps of obtaining real-time statistical data and offline statistical data of each statistical point; wherein the statistical points are statistical entities representing statistics of set storage data stored in a set time period, and the offline statistical data are obtained by performing full scanning on the storage data currently belonging to the statistical points in an offline statistical task execution time period; and for each statistical point, selecting one of the real-time statistical data and the offline statistical data of the statistical point as target statistical data of the statistical point based on a time relationship between an offline statistical task execution time period and a set time period corresponding to the statistical point. In this way, the statistical information of the stored data can be accurately and timely obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and storage medium for storing statistical data. Background Technology

[0002] Databases have been widely used in many fields. Taking graph databases as an example, statistical analysis of graph data is crucial in application scenarios such as social network analysis, knowledge graph construction, recommendation systems, and intelligent transportation. Accurate and timely acquisition of the data volume corresponding to various points or edges plays a key role in understanding the data scale, optimizing query strategies, and supporting decision-making.

[0003] Therefore, it is of great significance to obtain data accurately and in a timely manner. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a method, device, and storage medium for statistical analysis of stored data, which can accurately and timely obtain statistical information on stored data.

[0005] To address the aforementioned technical problems, this application provides a method for statistical analysis of stored data. This method includes: acquiring real-time statistical data and offline statistical data for each statistical point; wherein, a statistical point is a statistical entity representing statistical analysis of a set of stored data stored within a set time period, and the offline statistical data is obtained by performing a full scan of the stored data belonging to the statistical point during the execution of an offline statistical task; for each statistical point, based on the time relationship between the execution time of the offline statistical task and the set time period corresponding to the statistical point, one of the real-time statistical data and the offline statistical data is selected as the target statistical data for the statistical point.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed to implement the above-mentioned method.

[0008] The above scheme selects either real-time or offline statistical data for a statistical point as its target statistical data based on the temporal relationship between the offline statistical task execution period and the set time period corresponding to the statistical point. Compared to simply merging offline statistical data before and online statistical data after a fixed time point to obtain the final statistical result, this application's method of selecting target statistical data based on temporal relationship fully combines the timeliness of real-time statistical data, the comprehensive accuracy of offline statistical data, and the coverage of the set time period for the statistical point and the offline statistical task execution period. This allows for a more accurate match to the real-time statistical needs of the statistical points, ultimately obtaining the target statistical data for each statistical point efficiently and accurately. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an embodiment of the data storage statistics method provided in this application; Figure 2 yes Figure 1 The flowchart of step S11 shown is a schematic diagram of one embodiment. Figure 3 This is a schematic diagram of an embodiment of the statistical graph model provided in this application; Figure 4 This is a schematic diagram illustrating the effect of an embodiment of real-time statistical data provided in this application; Figure 5 This is a schematic diagram illustrating the effect of an embodiment of clearing expired statistical points provided in this application; Figure 6 This is a schematic diagram illustrating the effect of an embodiment of offline statistical data calibration statistical chart and real-time statistical data provided in this application; Figure 7 This is a schematic diagram of the framework of an embodiment of the electronic device provided in this application; Figure 8 This is a schematic diagram of the framework of the computer-readable storage medium provided in this application. Detailed Implementation

[0010] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0011] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0012] It's worth noting that, taking graph databases as an example, graph databases are widely used in numerous fields, such as social network analysis, knowledge graph construction, recommendation systems, and intelligent transportation. In these application scenarios, statistical analysis of graph data is crucial. Accurately and promptly obtaining the data volume corresponding to various vertex and edge labels plays a key role in understanding the data scale, optimizing query strategies, and supporting decision-making.

[0013] Traditional graph databases have several shortcomings in data statistics. Firstly, many graph databases employ periodic full-scale statistics, updating the entire database at pre-defined time intervals. This approach has a low update frequency, making it difficult to meet the demands of real-time applications. For example, in social network monitoring, user relationships and interaction data change rapidly. Long statistical periods may prevent decision-makers from promptly grasping the dynamic scale and structural characteristics of the network, leading to incorrect decisions. Secondly, existing statistical methods often face performance bottlenecks when processing large-scale graph data. When graph databases contain massive amounts of nodes and edges, traditional statistical methods can result in excessively long query times, or even system lag or crashes. For instance, for a knowledge graph containing tens of millions of nodes and billions of edges, obtaining the number of edge labels for a specific node using conventional traversal methods could take several minutes or even longer (depending on the data size), which is completely unacceptable in online service scenarios with high real-time requirements.

[0014] Furthermore, graph databases typically possess indexing engines, supporting graph data queries using graph languages. For example, the Janusgraph graph database, while offering a count operator for graph data statistics, relies either on an index for the label or on traversing the label's data within the storage engine to calculate the data volume for each vertex, edge, and label. While an external index can quickly return the corresponding data volume for vertex, edge, and label data, in practical use cases, many label types are only used for direct data storage and display, without attribute-based queries. Therefore, indexing is not necessary for all label types. Moreover, indexing all labels would drastically increase index storage space, leading to higher costs. On the other hand, for unindexed labels, the count operator directly triggers a full graph scan of the storage engine, with time consumption directly proportional to data size, failing to meet real-time statistical requirements.

[0015] Not only graph databases, but other databases are also unable to meet the needs of real-time statistics when data storage volume increases dramatically.

[0016] To address the aforementioned issues, this application proposes a data storage statistics method. This method combines real-time and offline statistical data for each statistical point to obtain the data volume of the set storage data corresponding to each statistical point.

[0017] It should be noted that the data storage statistics method provided in this application is not only applicable to the statistics of stored data in graph databases (such as JanusGraph), but also applicable to the statistics of stored data in other types of databases.

[0018] To aid understanding, let's first briefly explain the three concepts: statistical points, real-time statistics, and offline statistics: A statistical point is a statistical entity that represents the statistical analysis of stored data within a specified time period.

[0019] In the field of graph databases, the stored data is graph-structured (graph data composed of nodes and / or edges). Graph data can be queried using graph languages. To facilitate the query of statistical data corresponding to each statistical point, the statistical point can be constructed as an entity point in the graph structure. The entity point is associated with several attribute information, that is, the statistical point is an entity point associated with several attribute information. For example, the several attribute information of the statistical point includes at least the set time period and the description information of the stored data.

[0020] This application is used to perform statistical analysis on stored data at various statistical points, wherein a statistical point is a statistical entity associated with a set time period and descriptive information of the stored data.

[0021] The time period is set as the time range for statistics, which is used to clarify the time boundary of the statistical points. The specific statistical granularity can be flexibly set according to actual needs; for example, when using days as the unit, the corresponding time period is from 00:00 to 24:00 every day; when using 4 hours as the unit, the corresponding time period is every 4 hours.

[0022] For example, if the time period for the statistical point is set to 00:00-24:00 on May 20, 2024 (or 20240520), it means that the statistical data for that statistical point is the total amount of data stored in the database from 00:00 to 24:00 on May 20, 2024.

[0023] If you want to collect data stored in the database from 00:00 to 24:00 on May 20, 2024, then the time period for the statistical point should be set to 00:00 to 24:00 on May 20, 2024.

[0024] Setting the stored data as the object of statistics clarifies what type of data is being statistically analyzed. Taking the objects of statistics as nodes or edges of graph data as an example, the description information for setting the stored data of the statistical points can include: graph name, label name of the point or edge, whether it is an edge, etc.

[0025] The graph name is used to represent the business graph to which the data belongs (such as a user social relationship graph or a device operation status graph); the label name is used to represent the label of the corresponding point or edge (such as the label "user" for a point and the label "follow" for an edge); and "whether it is an edge" is used to mark whether the data is point data or edge data (such as "yes" or "no", "yes" indicates an edge, and "no" indicates a point).

[0026] In some implementation scenarios, to facilitate the rapid acquisition of the current statistical results (i.e., the current statistic) of a statistical point, the current statistical result can also be used as the attribute information of the statistical point. For details on the current statistic, please refer to the description below.

[0027] Real-time statistics and offline statistics are both data statistics methods. Real-time statistics are a data statistics method in which statistics are performed in real time while the node is storing data. For example, every time a node stores a piece of data, the current amount of stored data at the corresponding statistical point is incremented by 1. Offline statistics are a post-event statistical method. Specifically, after the data has been stored in the database, a full scan of the stored data belonging to the statistical point in the database is performed at a certain time period to obtain the corresponding statistical data.

[0028] Real-time statistics are performed in real time, so the results can be obtained quickly. Offline statistics are based on all the data stored in the database, which can ensure the accuracy of the data statistics, but because it takes a long time to perform statistics based on all the data, the results cannot be obtained in real time.

[0029] In view of this, this application cleverly integrates the advantages of real-time statistics and offline statistics, and performs statistics on stored data by combining the advantages and characteristics of real-time statistics and offline statistics, so as to quickly and accurately determine the statistical results of stored data.

[0030] Furthermore, the real-time statistics and data storage of statistical points are not affected by offline statistical tasks. During the execution period of offline tasks, there may be cases where data from some statistical points continues to enter the database after offline scanning. Therefore, the accuracy of offline statistical results cannot be guaranteed during the execution period of offline statistical tasks.

[0031] For example, if the offline statistics task is executed on September 13th, and the set time period for statistical point A is September 12th, and the set time period for statistical point B is September 13th; for statistical point A, since the data corresponding to statistical point A was already stored in the database on September 12th, when the offline statistics task is executed on September 13th, all the data for statistical point A has been stored in the database, so the offline statistics for statistical point A are accurate; however, for statistical point B, if the scanning of the data corresponding to statistical point B starts at 12:00 noon on September 13th, the database only stores the data for statistical point B from the early morning of September 13th to 12:00 noon. If the data for statistical point B continues to be stored after 12:00 noon, then the offline scan can only scan the data stored before 12:00 noon, while the data stored in the database after 12:00 noon is not scanned. Therefore, the statistical result for statistical point B is inaccurate. That is, the offline statistical result for statistical point B (set time period is September 13th) obtained during the offline statistics task execution period (September 13th) is inaccurate.

[0032] It is evident that if the execution time of the offline statistics task overlaps with the set time period of the statistics point, the accuracy of the offline statistical data of the corresponding statistics point cannot be guaranteed. Therefore, in cases where the execution time of the offline statistics task overlaps with the set time period of the statistics point, the real-time statistical data of the statistics point should be taken as the standard.

[0033] In view of this, in order to ensure that the statistical results of each statistical point can be obtained in a timely and accurate manner, this application creatively proposes to select one of the real-time statistical data and offline statistical data of the statistical point as the target statistical data of the statistical point based on the time relationship between the execution period of the offline statistical task and the set time period corresponding to the statistical point.

[0034] The general logic is as follows: if the set time period of the statistical point is before the execution time period of the offline statistical task, the accuracy of the offline statistical data of the statistical point can be guaranteed. Therefore, under this condition, the offline statistical data of the statistical point is selected as the corresponding target statistical result. However, if the set time period of the statistical point is within or after the execution time period of the offline statistical task, either the accuracy of the offline statistical data of the statistical point cannot be guaranteed, or there is no offline statistical data of the statistical point at present. Therefore, under this time condition, the real-time statistical data of the statistical point is selected as the corresponding target statistical data.

[0035] Specifically, please refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data statistics method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes: S11: Obtain real-time statistical data and offline statistical data for each statistical point; where a statistical point is a statistical entity that represents the statistical analysis of a set of stored data stored within a set time period, and offline statistical data is obtained by performing a full scan of the stored data belonging to the statistical point during the execution period of the offline statistical task.

[0036] This embodiment combines the timeliness of real-time statistical data, the full accuracy of offline statistical data, the boundary attributes of the time period set for statistical points, and the coverage of the offline statistical task execution period to efficiently and accurately determine the target statistical data (statistical results) for each statistical point. For an introduction to statistical points, real-time statistics, and offline statistics, please refer to the preceding description.

[0037] Among them, real-time statistical data is obtained by performing real-time statistics using data processed by nodes, while offline statistical data is obtained by performing a full scan of the stored data in the database that currently belongs to the statistical points.

[0038] In some implementation scenarios, the stored data is stored in a distributed database, which includes several nodes and a data storage engine. For example, the distributed database is a graph database called JanusGraph, which comprises multiple JanusGraph nodes. When data is accessed in parallel, each JanusGraph node receives a corresponding data write request. These nodes quickly and accurately save the statistical mapping information corresponding to the data labels into memory to obtain the amount of stored data for their own statistical points. Subsequently, each node can update the real-time statistical data according to a preset update frequency, using the data storage volume of its own statistical points. The preset update frequency can be once every 10 seconds to ensure that the real-time statistical data reflects the latest statistical dynamics of the statistical points in a timely and accurate manner.

[0039] The data storage engine is used for persistent data storage. After receiving data, several nodes will not allow the data to remain in their local memory indefinitely (otherwise, data would be lost if a node crashes). Instead, they will asynchronously or synchronously write the locally cached data into the data storage engine. Offline statistical data is obtained by performing a full scan of the stored data at the statistical points written to the data storage engine.

[0040] As can be seen above, in a distributed database scenario, each node retains its own statistical data (data volume) for the statistical points it has collected in its memory. In this scenario, the real-time statistical data for each statistical point is updated based on the data volume of each statistical point collected by each node. It should be noted that what each node retains in its memory is the statistical data (data volume) for the statistical points collected in real time by that node, not the stored data itself. The stored data is stored in a dedicated storage engine.

[0041] Furthermore, in order to reflect the latest statistical dynamics (real-time statistical data) of the statistical points in a timely and accurate manner, the real-time statistical data of each statistical point can be updated according to a preset update frequency. The specific update frequency can be determined based on actual experience and needs. For example, a preset update frequency of once every 10 seconds can be set to ensure timely acquisition of statistical information without affecting data storage efficiency.

[0042] Specifically, please refer to Figure 2 , Figure 2 yes Figure 1 The flowchart shown is a schematic diagram of an embodiment of step S11. In this embodiment, step S11 obtains real-time statistical data for each statistical point, including: S21: Select the target node for this real-time statistics update from multiple nodes.

[0043] In this embodiment, to avoid the situation where multiple nodes participate in the real-time statistical data update at the same time, which may lead to erroneous updates of statistical data, a target node can be selected from multiple nodes for each update. Only the selected target node can participate in the update of the real-time statistical data, while other unselected nodes cannot participate in the update of the real-time statistical data.

[0044] It should be noted that nodes that are not selected simply do not participate in the update of this real-time statistics, but they will still count the amount of stored data at each statistical point in real time.

[0045] S22: For each statistical point, obtain the incremental storage data of the statistical point of the target node within the target time period; the target time period is the period between the current time the target node is the target node and the previous time it was the target node.

[0046] S23: The sum of the stored data increment and the real-time statistical data updated in the previous statistical point is used as the real-time statistical data updated in this statistical point.

[0047] S24: According to the preset update frequency, repeatedly execute the process of selecting the target node for this real-time statistical data update from multiple nodes and the subsequent steps.

[0048] It should be noted that since real-time statistics are constantly updated, and each update is based on the previous real-time statistics rather than starting from 0, each time real-time statistics are updated, it is necessary to know the incremental storage data of each statistical point during the time period (target time period) between the current selection as the target node and the previous selection as the target node. Then, the sum of the incremental storage data of each statistical point and the previously updated real-time statistics is used as the real-time statistics of each statistical point in this update.

[0049] To facilitate the calculation of the increase in stored data for each node during the target time period between the current and previous selections as the target node, without affecting the node's ability to continue calculating the stored data for each statistical point, two storage areas (or containers) can be pre-set in the memory of each node. These are the first storage area and the second storage area. The first storage area is used to store the stored data for each statistical point calculated by the corresponding node. The stored data is calculated when the corresponding node stores the data corresponding to the statistical point (for example, when caching the corresponding data into memory, or when persistently storing the cached data in memory into the storage engine). The second storage area is used to store the amount of stored data copied from the first storage area each time the corresponding node is selected as the target node. The stored data increment is the data increment between the current copied stored data and the previous copied stored data.

[0050] For example, node A has two containers in its memory: the addCountMap container (corresponding to the first storage area above) and the oldAddCountMap container (corresponding to the second storage area above). The container structure is related to the description information of the stored statistical data. For example, the description information includes the map name, label name, set time, and count, and the corresponding container structure is <map name, <label name with time information, count>>. When each node processes a piece of data, it refreshes the statistical information of the corresponding label to the memory container addCountMap, that is, it increments the count value of the statistical point in addCountMap by 1 based on the key that matches the current data description information (map name, label name with time information).

[0051] For example, after node 1 processes a data point with the label "user" and timestamp "20240918" in graph A, addCountMap["graphA+user+20240918"] changes from 100 to 101; after node 2 processes a data point with the label "order" and timestamp "20240918" in graph A, addCountMap["graphA+order+20240918"] changes from 50 to 51.

[0052] As can be seen above, when the nodes process data in parallel, each node will immediately add the statistical count of that type of data (statistical point) in its own memory container addCountMap according to the dimension of "graph name + data type + timestamp" after completing the processing of a single data item, thereby achieving real-time, accurate and efficient statistics on various types of data.

[0053] In one implementation, a target node can be selected by setting a distributed global lock (referred to as the target lock). Each node can preempt the target lock, but only one node can successfully acquire it; the target node is the node that acquires the target lock. After completing step S23 to update the real-time statistical data, the target lock can be released to allow for the selection of a new target node in the next round of lock acquisition. Steps S21 to S24 are then repeated at a preset update frequency.

[0054] Of course, in other implementations, different nodes can take turns as target nodes and repeat steps S21 to S24 according to a preset update frequency.

[0055] It should be noted that offline statistical data needs to be updated at a certain frequency to correct the real-time statistical data of the statistical points at regular intervals. Of course, the update frequency of offline statistical data should be less than that of real-time statistical data. For example, real-time statistical data is updated every 10 seconds, while offline statistical data is updated once a week, such as on Saturdays and Sundays when it is not busy, to update the offline statistical data.

[0056] S12: For each statistical point, based on the time relationship between the offline statistical task execution period and the set time period corresponding to the statistical point, select one of the real-time statistical data and the offline statistical data of the statistical point as the target statistical data of the statistical point.

[0057] In one implementation scenario, the method of this embodiment is used in databases other than graph databases. In this implementation scenario, statistical information (statistical data) of each statistical point can be stored in the form of statistical tables.

[0058] In another implementation scenario, the method of this embodiment is used for a graph database. Graph databases generally support information retrieval of graph structures using graph language. To facilitate querying using graph language, in this implementation scenario, a statistical graph can be constructed during the initialization of the distributed database before obtaining the real-time and offline statistical data for each statistical point in step S12. The statistical graph includes a statistical graph model containing metadata fields for statistical information of each statistical point, which is used to store each statistical point.

[0059] It should be noted that a statistical graph can be understood as a graph structure in a graph database that is independent of the business graph. Its core component is the statistical point, and each statistical point carries metadata fields that describe the statistical information itself. These metadata fields are the core attributes of the statistical point (the several attribute information associated with the statistical point mentioned above), which are used to clarify which type of data in the business graph the statistical point is counting, as well as the corresponding statistical dimension.

[0060] In one embodiment, the attribute information associated with a statistical point includes the statistical quantity of that statistical point. Before determining the target statistical data for a statistical point, the statistical graph is used to store the real-time statistical data for that statistical point. After determining the target statistical data for a statistical point, the real-time statistical data stored in the statistical graph can be adjusted to match the target statistical data for that statistical point. To prevent the loss of statistical data for each statistical point in the statistical graph, the statistical graph can be persistently stored in the storage engine of the graph database.

[0061] In one embodiment, the statistical graph is a statistical graph model used to store each statistical point, and the statistical points are entity points with a graph structure, that is, the statistical points are stored in the form of a graph; for example Figure 3In the statistical graph model shown, both people and peer edges are statistical points in the statistical graph, and both statistical points are entity points associated with several attribute information (or metadata fields). The several attribute information associated with each statistical point includes: the fixed label name of the statistical point (used to identify the entity point as a statistical point), the graph name, the label name, whether it is an edge, the statistic, and the set time period, etc.

[0062] It should be noted that since statistical points are entity points with a graph structure, they belong to graph data and can therefore be queried using graph language. After retrieving the statistical data of a statistical point, the relationship between the value of the statistical data and a preset threshold can be used to determine whether to control the statistical point and / or set corresponding control permissions.

[0063] Furthermore, the attribute information differs across different statistical points. For example, please refer to the documentation for the attribute information related to a specific time period. Figure 3 The first statistical point (person) is set to a time period of 20241023. If other attribute information (excluding statistical quantities) of the node (person) exists, it will be considered as follows: Figure 3 If the attribute information of the first statistical point is the same, but the time period is different, such as 20241024, then the two are different statistical points. In order to facilitate understanding of the data storage volume of a node (person) in different time periods, the data storage volume of a node (person) in different time periods can be associated in the statistical graph. For example, relationship edges can be established for the data storage volume of a node (person) in different time periods to form a data chain.

[0064] In some embodiments, in order to quickly query the statistical information of each statistical point and the statistical information corresponding to different combinations of attribute information, after constructing the statistical graph as described above, the set time period corresponding to each statistical point, the description information of the set stored data, and the current statistical result can be used as the attribute information of the corresponding statistical point; then, an inverted index is built for the attribute information of each statistical point, so as to quickly query the corresponding statistical point using graph language based on the inverted index; wherein, the current statistical result is the latest target statistical data of the statistical point.

[0065] Please refer to the following: Figure 3 Among the several attribute information associated with the statistical points, the graph name (business graph name), label name, and whether it is an edge are description information of the set storage data of the above statistical points, and the statistic is the current statistical result of the above statistical points.

[0066] Among them, the method of establishing an inverted index for several attribute information of statistical points is beneficial to using graph language to quickly query the statistical quantity corresponding to each statistical point or the statistical data corresponding to different attribute combinations based on the inverted index. For example, at least some of the attribute information associated with the statistical points can be combined in a dynamic and configurable way to obtain attribute information combinations, and then the attribute information combinations can be used as query conditions to query the corresponding data based on the attribute information combinations using graph language.

[0067] For example, Gremlin statements (a graph language) can be used to query data based on combinations of attribute information. For instance, at least two of the following can be combined to form diverse query conditions: graph name, schemaLabelName, timestamp, isEdge, and count. This allows for the rapid acquisition of the required statistical information through the corresponding Gremlin statements, which is beneficial for real-time insight into data dynamics and in-depth mining of historical data value.

[0068] For example, the Gremlin statement "statistics.traversal().V().has('graphName','graph').values('count').sum()" can be used to query the data volume of the business graph "graph"; the Gremlin statement "statistics.traversal().V().has('graphName','graph').has('schemaLabelName','person').values('count').sum()" can be used to query the number of "person" entity data in the business graph "graph"; the Gremlin statement "statistics.traversal().V().has('graphName','graph').has('isEdge',true).values('count').sum()" can be used to query the number of relation (edge) data in the business graph "graph"; the statement " The function `statistics.traversal().V().has('graphName','graph').has('schemaLabelName','hasPersonSamePeer').has('timestamp',P.gte(20241001)).has('timestamp',P.lte(20241031)) .values('count').sum()` queries the business graph `graph` to see how many peer relationships (edges) were added in October 2024 using the `hasPersonSamePeer` function. Here, `statistics` is the fixed label name of the statistical point, used to identify that the entity point is a statistical point.

[0069] It should be noted that, compared to the existing method of building an index for all point edge labels in a graph database, the method of building an index for several attribute information of statistical points in this application can significantly reduce the amount of index, and thus significantly reduce the amount of data written to the index engine. In addition, the above method of building an index by using the statistical quantity of the set storage data corresponding to the statistical point as one of the attribute information can directly query the relevant statistical quantity of the statistical point through the statistical quantity attribute information, without having to traverse the data in the business graph to perform data statistics, so the statistical quantity of the statistical point can be obtained quickly.

[0070] Furthermore, as mentioned earlier, before determining the target statistical data for each statistical point, the statistical chart is used to store the real-time statistical data for that point. After determining the target statistical data, the real-time statistical data stored in the statistical chart can be adjusted to match the target statistical data. Further, considering the large amount of data in the statistical chart, to better manage data storage and avoid situations where the statistical chart is empty for a period of time due to simultaneous updates to the statistical data of all statistical points, the following steps can be performed sequentially on each statistical point in the chart: based on the time relationship between the offline statistical task execution period and the corresponding set time period, select either the real-time statistical data or the offline statistical data as the target statistical data for that point, and adjust the real-time statistical data stored in the statistical chart to match the target statistical data.

[0071] Furthermore, as described above, if the execution time of the offline statistics task overlaps with the set time of the statistics point, the accuracy of the offline statistics data of the corresponding statistics point cannot be guaranteed. Therefore, in this case, the real-time statistics data of the statistics point can be used as the standard, that is, the real-time statistics data of the statistics point can be used as the corresponding target statistics data.

[0072] Furthermore, if the set time period corresponding to a statistical point is after the execution period of the offline statistical task, then there will be no offline statistical data for that statistical point. In this case, the real-time statistical data of the statistical point will be selected as the corresponding target statistical data. For example, if the set time for statistical point C is the entire day of February 12th, while the offline statistical task is on February 10th, then when performing offline statistics, the data for statistical point C has not yet been stored in the database, so there will be no offline statistical data for statistical point C. In this case, the real-time statistical data of statistical point C can be used as the corresponding target statistical data. Subsequently, after the next round of offline statistics, based on the time relationship between the execution period of the next offline statistical task and the set time period corresponding to the statistical point, one of the real-time statistical data and the offline statistical data of the statistical point will be selected as the target statistical data for the statistical point.

[0073] Of course, if the set time period corresponding to the statistical point is before the offline statistical task execution time, it means that the stored data of the set time period corresponding to the statistics has been fully scanned. In this case, the offline statistical data of the statistical point can be selected as the target statistical data of the statistical point.

[0074] In summary, step S12 includes the following situations: Scenario 1: If the set time period corresponding to the statistical point overlaps with the execution time period of the offline statistical task, or if the set time period corresponding to the statistical point is after the execution time period of the offline statistical task, select the real-time statistical data of the statistical point as the target statistical data of the statistical point. Scenario 2: If the set time period corresponding to the statistical point is before the offline statistical task execution period, select the offline statistical data of the statistical point as the target statistical data of the statistical point.

[0075] It should be noted that in some implementation scenarios, business users may delete the stored data of certain statistical points before the offline statistical task execution period due to business needs, or the database may automatically delete the stored data of some expired statistical points before the offline statistical task execution period. In this scenario, these deleted data will not be scanned during offline statistics, so there will be no offline statistical data for that statistical point. However, since the stored data in the database is statistically analyzed in real time before deletion, there will be real-time statistical data for that statistical point.

[0076] Considering this, during the process of selecting offline statistical data for a statistical point as the target statistical data for that statistical point before the execution period of the offline statistical task, the following steps can be further performed: check whether there is corresponding offline statistical data for the statistical point; if there is, select the offline statistical data for the statistical point as the target statistical data for the statistical point; if there is no corresponding offline statistical data, delete the statistical point in the statistical chart that does not have corresponding offline statistical data, or adjust the real-time statistical data of that statistical point in the statistical chart to 0 or invalid.

[0077] The above scheme selects either real-time or offline statistical data for a statistical point as its target statistical data based on the temporal relationship between the offline statistical task execution period and the set time period corresponding to the statistical point. Compared to simply merging offline statistical data before and online statistical data after a fixed time point to obtain the final statistical result, this application's method of selecting target statistical data based on temporal relationship fully combines the timeliness of real-time statistical data, the comprehensive accuracy of offline statistical data, and the coverage of the set time period for the statistical point and the offline statistical task execution period. This allows for a more accurate match to the real-time statistical needs of the statistical points, ultimately obtaining the target statistical data for each statistical point efficiently and accurately.

[0078] Furthermore, the above method of constructing statistical points and statistical graphs into graph structure information (graph data) facilitates the use of graph language to query graph data, thereby quickly obtaining statistical data of statistical points and statistical data related to statistical points; the specific data obtained is determined based on the combination of input attribute information.

[0079] In some embodiments, after obtaining the real-time statistical data and offline statistical data of each statistical point in step S11, or after selecting one of the real-time statistical data and offline statistical data of the statistical point as the target statistical data of the statistical point in step S12, the method further includes the following step of clearing expired statistical points: when executing each statistical point clearing task according to the preset execution frequency, the earliest corresponding statistical boundary period is determined based on the preset life cycle of each statistical point and the current time; and the statistical points whose corresponding set time period is earlier than the earliest statistical boundary period are deleted.

[0080] The preset execution frequency is, for example, once a day. The specific frequency can be determined based on the preset lifecycle of the statistical points and the set time of each statistical point. For example, if the set time of the statistical points is in days, then the preset execution frequency can be to clean up on a daily basis.

[0081] In one implementation scenario, the database is a distributed graph database. The cleanup of statistical points in the statistical graph is performed by one of the multiple nodes in the distributed graph database, for example, by using lock contention to determine the node to perform the cleanup.

[0082] To facilitate understanding of this application, a brief introduction is given below using the distributed graph database (JanusGraph) as an example: At the initial stage of the graph database cluster (distributed graph database), a statistical graph is constructed specifically for persistently storing statistical information of the statistical points. Subsequently, when data is accessed in parallel, each JanusGraph node receives a corresponding data write request. These nodes quickly and accurately save the statistical mapping information corresponding to the statistical points into memory. Then, every 10 seconds, an asynchronous refresh task starts on time, efficiently refreshing the statistical information temporarily stored in memory to the statistical graph, ensuring that the statistical graph reflects the latest data dynamics in a timely and accurate manner.

[0083] Furthermore, during off-peak hours each week, such as weekends, an offline statistical task is automatically triggered. This task leverages the powerful parallel processing capabilities of the HBase coprocessor, cleverly pushing the interpretation of underlying data down to the HBase level. This completely changes the inefficient traditional statistical method of pulling each piece of underlying serialized data to JanusGraph for interpretation and then statistics. The results of the offline statistics are integrated into the statistical graph, making the statistical results more accurate and providing solid and reliable data support for subsequent data analysis and decision-making.

[0084] Ultimately, for upper-level business users, obtaining graph statistics becomes extremely convenient and flexible. They can directly use Gremlin statements to dynamically configure and input diverse query conditions such as label names and time ranges to quickly obtain the required statistical information, greatly improving the efficiency of business users in utilizing graph data and the depth of value mining.

[0085] For example, firstly, since the vertex and edge label data of a graph are continuously increasing, it is necessary to persist the statistics. Considering performance and convenience, the concept of a graph can be directly used to construct the statistical storage model. During the initialization of the graph database cluster, a statistical graph containing metadata fields for vertex and edge label statistics (corresponding to the attribute information mentioned above) is constructed, specifically as follows: Figure 3 As shown, the data statistics for each statistical point are abstracted into a statistical point entity with a fixed label name (e.g., M or Statistics). This fixed label name M represents the statistical point, and a suitable index is created to facilitate querying the statistical quantity of the statistical point. The fixed label name of the statistical point is M, and it also contains five attributes related to data statistics information: graph name (in actual business scenarios, there may be multiple business graphs; this attribute can distinguish between business graphs for data volume statistics), label name (the label name of the point or edge), whether it is an edge (indicating whether the statistical point is an edge, facilitating the calculation of the data volume of all points or edges during querying), count of statistics, and the corresponding set time period (timeStamp, for example, a statistical point with an attribute value of 20241024 indicates the data volume of the statistical point on that day). Furthermore, an inverted index is created for these attributes of the statistical point (graph name, label name, whether it is an edge, and set time period), allowing for fast data volume statistics based on the dimensions of: graph name, label name, whether it is a point or edge, and time range.

[0086] Then, online statistics (real-time statistics) are performed during data access. Specifically, each Janusgraph node initializes an addCountMap container and an oldAddCountMap container in memory when it starts up (both container structures are...). <graphName,<labelNameWithTime,count> The container structure is `<graph name, <label name with time information, count value>>`. A task is created to update statistical values ​​into the graph every 10 seconds, and another task is created to clean up expired statistical points daily at midnight (each midnight, expired statistical points with lifecycles are cleaned up by calculating the timestamp of the earliest statistical point based on the current time and lifecycle, and then deleting the expired statistical points based on the label and timestamp). When each Janusgraph node receives data in parallel, after processing a data point, the statistical information of the corresponding label is refreshed into the memory container `addCountMap` (i.e., the count value in `addCountMap` is accumulated based on the combination key of the graph name, data label, and set time for the current data), and then the write task returns a successful write to the client. The server-side backend executes a scheduled statistical update task every 10 seconds. When new data is detected for a certain label (statistical point), it directly compares the difference between addCountMap and oldAddCountMap for a specific label in a given graph to see if it is greater than 0. If new data is detected, the statistical graph information update task is immediately triggered. It first tries to acquire the ZooKeeper distributed global lock (corresponding to the target lock mentioned above). If it fails to acquire the distributed lock, it waits for the next asynchronous update task in the next cycle to try again. If it acquires the distributed lock, it deeply copies a container currentAddCountMap from addCountMap (because addCountMap receives statistical data at any time, it directly deep copies the current content to avoid concurrent conflicts). The statistical increment C1 for a specific label in a given graph on a given day is then the node's currentAddCountMap minus oldAddCountMap. At this point, the Gremlin statement (statistics.traversal().V().has(GRAPH_NAME, graphName).has(SCHEMA_LABEL_NAME, schemaLabelName).has(IS_EDGE, isEdge).has(TIMESTAMP, ...) is then used. dateDay). <long>The function `values(COUNT)` retrieves the statistic C2 for that label on the chart for that day. The increment that needs to be updated in the chart is C3 = C2 + C1. The new statistic can then be refreshed onto the chart using Gremlin's property operator. Finally, the contents of `currentAddCountMap` are deeply copied into `oldAddCountMap`, the global lock is released, and the process waits 10 seconds to begin the next asynchronous update cycle.

[0087] During off-peak hours each week, such as early morning on weekends, an offline statistics task is automatically triggered. This task relies on the powerful parallel processing capabilities of the HBase coprocessor, cleverly pushing the interpretation of underlying data down to the HBase level, completely changing the inefficient traditional statistical model that requires pulling each piece of underlying serialized data to JanusGraph for interpretation and then statistics.

[0088] When the offline statistics task is completed, the process of merging offline statistical data into the statistics graph begins. First, the ZooKeeper distributed global lock is acquired (to prevent updates to real-time statistical data due to multiple nodes). Then, the statistical data in the statistics graph and the offline statistical data are compared. The statistical points in the statistics graph are traversed, checking if corresponding offline statistical data exists for statistical points outside the offline statistics task execution period (data other than Saturday and Sunday). If it exists, the offline statistical data for that point is used as the corresponding target statistical data and overwritten onto that statistical point in the statistics graph; if it does not exist, the statistical point is deleted from the statistics graph. After traversing all statistical points in the statistics graph according to the above process, an expiration cleanup task is triggered to clean up expired statistical points (to prevent interference from expired data during the offline statistics task execution), and then the ZooKeeper distributed global lock is released.

[0089] Finally, the query module for statistical charts can directly use Gremlin statements to dynamically and configurably input diverse query conditions such as business chart name, label name, and time range to quickly obtain the required statistical information.

[0090] The following specific embodiment further illustrates the statistical method for storing data, demonstrating the entire process of real-time statistics, expired statistical points, and offline statistical fusion calibration.

[0091] The database will begin accepting data on October 1, 2024, with 10 entries for entities (tag names) and 20 entries for edges (tag names) being accepted daily. For example... Figure 4 The data entry statistics for November 6, 2024 (Wednesday) and November 7, 2024 are displayed. On November 7, 2024, the user changed the lifecycle of the entity (person) to 1 month.

[0092] Figure 5 The data shows the process of clearing expired statistics points. The earliest entity statistics point on November 8, 2024 (Friday) was 20241009. On that day, due to user error, duplicate data was entered twice. On November 9, 2024 (Saturday), the system automatically triggered an offline statistics task. The statistics chart on November 10 shows that the earliest entity statistics point was 20241011 (after automatically clearing expired statistics points every morning, it meets the 31-day lifecycle requirement). However, the statistics point 20241108 still had 20 duplicate records.

[0093] Figure 6 This section demonstrates the offline statistical calibration chart. The red portion in the upper left block is because the offline task was triggered on November 9, 2024. When the offline statistical task scanned business data on November 9, the data had not expired. However, by November 10, the business data from October 9, 2024, had expired. Since the statistics from October 9 were still available, the offline statistical results show two statistics. The yellow portion in the upper left block is because some business data was inserted into the database on November 9 and November 10, 2024, falling into regions already scanned by the offline statistical task. Therefore, it doesn't display 10 statistics. The light green portion in the lower left block represents that the statistical results during the offline statistical task period are based on the online statistical values, i.e., the statistical points already existing in the chart during the period [November 9, 2024, November 10, 2024]. The two parts of the left block are merged to form the right block. The red portion on the left was discarded during the merging process because it had expired. As shown in the final statistical results on the right, the number of duplicate data entries in 20241108 has been calibrated, and the results are completely consistent with the actual data entry situation.

[0094] It should be noted that, Figure 4-6 The example table is for the convenience of explaining the above content. In fact, the statistical information of the statistical points in this application is in the form of a graph, not a table.

[0095] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 70 includes a memory 71 and a processor 72 coupled to each other.

[0096] The memory 71 stores program instructions, and the processor 72 executes the program instructions stored in the memory 71 to implement the steps of any of the above-described method embodiments. In a specific implementation scenario, the electronic device 70 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 70 may also include mobile devices such as laptops and tablets, which are not limited here.

[0097] Specifically, processor 72 controls itself and memory 71 to implement the steps of any of the above embodiments. Processor 72 may also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 72 may be implemented using integrated circuit chips.

[0098] Please see Figure 8 , Figure 8 This is a schematic diagram of the framework of the computer-readable storage medium provided in this application. The computer-readable storage medium 80 of this application embodiment stores program instructions 81, which, when executed, implement the methods provided in any embodiment and any non-conflicting combination of the above methods. The program instructions 81 can form a program file and be stored in the computer-readable storage medium 80 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 80 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0099] The above scheme selects either real-time or offline statistical data for a statistical point as its target statistical data based on the temporal relationship between the offline statistical task execution period and the set time period corresponding to the statistical point. Compared to simply merging offline statistical data before and online statistical data after a fixed time point to obtain the final statistical result, this application's method of selecting target statistical data based on temporal relationship fully combines the timeliness of real-time statistical data, the comprehensive accuracy of offline statistical data, and the coverage of the set time period for the statistical point and the offline statistical task execution period. This allows for a more accurate match to the real-time statistical needs of the statistical points, ultimately obtaining the target statistical data for each statistical point efficiently and accurately.

[0100] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0101] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.< / long>

Claims

1. A method for storing statistical data, characterized in that, The method includes: Obtain real-time and offline statistical data for each statistical point; wherein, the statistical point is a statistical entity that represents the statistical analysis of a set of stored data stored within a set time period, and the offline statistical data is obtained by performing a full scan of the stored data currently belonging to the statistical point during the execution period of the offline statistical task; For each statistical point, based on the time relationship between the offline statistical task execution period and the set time period corresponding to the statistical point, one of the real-time statistical data and the offline statistical data of the statistical point is selected as the target statistical data of the statistical point.

2. The method according to claim 1, characterized in that, The step of selecting either the real-time statistical data or the offline statistical data of the statistical point as the target statistical data of the statistical point based on the time relationship between the offline statistical task execution period and the set time period corresponding to the statistical point includes: In response to the overlap between the set time period corresponding to the statistical point and the execution period of the offline statistical task, or the set time period corresponding to the statistical point being after the execution period of the offline statistical task, the real-time statistical data of the statistical point is selected as the target statistical data of the statistical point. In response to the set time period corresponding to the statistical point being before the execution period of the offline statistical task, the offline statistical data of the statistical point is selected as the target statistical data of the statistical point.

3. The method according to claim 2, characterized in that, The method can be used to count the amount of stored data corresponding to statistical points in a distributed database. Before obtaining the real-time and offline statistical data for each statistical point, the method further includes: When initializing the distributed database, a statistical graph is constructed; wherein, before determining the target statistical data for the statistical points, the statistical graph is used to store the real-time statistical data for the statistical points. After determining the target statistical data for the statistical points, the method further includes: The real-time statistical data of the statistical points stored in the statistical chart is adjusted to the target statistical data of the statistical points.

4. The method according to claim 3, characterized in that, For each statistical point, based on the time relationship between the offline statistical task execution period and the set time period corresponding to the statistical point, one of the real-time statistical data and the offline statistical data of the statistical point is selected as the target statistical data of the statistical point. The real-time statistical data of the statistical points stored in the statistical chart is adjusted to the target statistical data of the statistical points, including: The steps are as follows: sequentially apply the time relationship between the offline statistical task execution period and the set time period corresponding to the statistical point to each statistical point in the statistical chart; select one of the real-time statistical data and the offline statistical data of the statistical point as the target statistical data of the statistical point; and adjust the real-time statistical data of the statistical point stored in the statistical chart to the target statistical data of the statistical point. The step of selecting offline statistical data for the statistical point as the target statistical data for the statistical point in response to the set time period corresponding to the statistical point being before the execution period of the offline statistical task includes: Detect whether there are corresponding offline statistical data for the statistical point; If it exists, select the offline statistical data of the statistical point as the target statistical data of the statistical point; If it does not exist, delete the statistical point in the statistical chart that does not have corresponding offline statistical data.

5. The method according to claim 3, characterized in that, After constructing the statistical graph, the method further includes: The set time period corresponding to each statistical point, the description information of the corresponding set stored data, and the current statistical result are used as the attribute information of the corresponding statistical point. The current statistical result is the latest target statistical data of the statistical point. An inverted index is built for the attribute information of each statistical point, so that the corresponding statistical point can be queried using graph language based on the inverted index.

6. The method according to claim 1, characterized in that, The method can be used to count the amount of stored data corresponding to statistical points in a distributed database. The distributed database includes multiple nodes, and each node is used to count the amount of stored data for each statistical point stored through the node. The acquisition of real-time statistical data for each statistical point includes: From the plurality of nodes, select the target node for this real-time statistical data update; For each statistical point, obtain the incremental storage data of the statistical point within the target time period; the target time period is the time period between the target node's current role as the target node and its previous role as the target node. The sum of the stored data increment and the real-time statistical data updated in the previous time at the statistical point is used as the real-time statistical data updated at the statistical point this time. According to the preset update frequency, the process of selecting the target node for the current real-time statistical data update from the multiple nodes and subsequent steps is repeated.

7. The method according to claim 6, characterized in that, The target node is the node that will acquire the target lock. After the statistical point completes the update of the real-time statistical data, the target lock is released. And / or, each of the nodes is provided with a first storage area and a second storage area. The first storage area is used to store the amount of stored data for each statistical point counted by the corresponding node. The amount of stored data is obtained by the corresponding node when storing the data corresponding to the statistical point. The second storage area is used to store the amount of stored data copied from the first storage area each time the corresponding node acts as a target node. The increment of the stored data is the data increment between the amount of stored data copied this time and the amount of stored data copied in the previous time.

8. The method according to claim 1, characterized in that, The method can be used to count the amount of stored data corresponding to statistical points in a distributed graph database. The stored data corresponding to the statistical points are nodes or edges of graph data. The statistical points are defined by attribute information, which includes description information of the corresponding set stored data. The description information of the set stored data includes: graph name, label name corresponding to the point or edge, and whether it is an edge. And / or, after obtaining the real-time statistical data and offline statistical data of each statistical point, or selecting one of the real-time statistical data and the offline statistical data of the statistical point as the target statistical data of the statistical point, the method further includes: When executing each statistical point clearing task according to the preset execution frequency, the earliest statistical boundary period is determined based on the preset lifespan of each statistical point and the current time. Delete the statistical points whose corresponding set time period is earlier than the earliest statistical boundary period.

9. An electronic device, characterized in that, Including interconnected memory and processor, The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the method of any one of claims 1-8.

Images