Enterprise-level digital operation information multidimensional analysis system

Abstract

The application relates to the technical field of data processing, in particular to a multi-dimensional analysis system for enterprise-level digital operation information, which comprises a metadata extraction module, a pre-estimation analysis module and a decision execution module. The metadata extraction module is used for extracting the container type statistical vector and the key space distribution bitmap of a bitmap object in each time window within a specified query period in response to a query request. The pre-estimation analysis module is used for calculating the replacement rate of the query period, generating a query strategy instruction and determining a to-be-processed unit. The total benchmark overhead of the to-be-processed unit is calculated. The space turnover number, type fluctuation amount, penalty coefficient and estimated instruction consumption of the to-be-processed unit are obtained. The decision execution module is used for selecting and executing a corresponding calculation strategy based on the estimated instruction consumption to complete the query on the to-be-processed unit. The overall calculation efficiency and hardware utilization rate are maximized under the premise of ensuring the query timeliness.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a multi-dimensional analysis system for enterprise-level digital operation information. Background Technology

[0002] In Online Analytical Processing (OLAP) systems, Roaring Bitmap, as a highly efficient compressed bitmap indexing technology, is widely used in scenarios such as user profiling deduplication and behavioral analysis. Roaring Bitmap employs a hybrid storage strategy, selecting different types of containers for data storage based on data sparsity. In actual query operations, the system often needs to perform logical merging operations on multiple bitmap objects within a continuous time window to support complex OLAP queries and real-time analysis.

[0003] Existing resource estimation schemes typically estimate computational overhead based solely on the cardinality of the bitmap, i.e., the total number of elements it contains. This linear estimation method ignores the heterogeneity of the underlying container types. During the merging process, if the two bitmaps involved in the operation have containers of the same type at the same key-value position, the CPU can process them efficiently using SIMD instructions. However, if the container types do not match, or if the data is misaligned in the key-value space, the CPU will not be able to effectively utilize branch prediction and pipeline techniques, resulting in a significant increase in instruction execution cycles. This hidden high overhead caused by the complexity of the physical structure cannot be perceived or captured, leading to severe distortion in resource estimation and unreliable execution strategy decisions in OLAP query processing, which in turn causes computing node overload or query response timeouts. Summary of the Invention

[0004] To address the aforementioned technical challenges, a multi-dimensional analysis system for enterprise-level digital operational information is provided.

[0005] The solution to the technical problem presented in this application is to provide a multi-dimensional analysis system for enterprise-level digital operation information, the system comprising:

[0006] The metadata extraction module is used to extract the container type statistical vector and key space distribution bitmap of bitmap objects in each time window within the specified query period when responding to a query request. The vector records the number of containers of each container type in the bitmap object, and the key space distribution bitmap records the existence distribution of the bitmap object in the high-level key value space.

[0007] The prediction and analysis module is used to obtain the estimated command consumption, specifically:

[0008] Analyze the proportion of distribution differences between bitmaps with different key space distributions, calculate the replacement rate for the query period, generate query strategy instructions, and determine the units to be processed.

[0009] By using the number of containers of different container types in the container type statistics vector, the expected resource overhead consumed by performing logical merging operations on the unit to be processed is evaluated, and the total baseline overhead of the unit to be processed is calculated.

[0010] The distribution differences of the key space distribution bitmap between adjacent time windows are analyzed to obtain the spatial flip number of the unit to be processed; the type fluctuation of the unit to be processed is determined by the differences of the container type statistical vector between adjacent time windows, and the penalty coefficient of the unit to be processed is obtained by combining the spatial flip number.

[0011] Based on the total baseline overhead and penalty coefficient, the estimated instruction consumption of the unit to be processed is determined;

[0012] The decision execution module is used to select and execute the corresponding calculation strategy based on the estimated instruction consumption in order to complete the query of the unit to be processed.

[0013] Preferably, the calculation of the permutation rate for the query period includes: defining the time window preceding the first time window within the query period as a reference window; performing bitwise XOR and bitwise OR operations on the key space distribution bitmap between the last time window and the reference window within the query period, and counting the number of bits with a value of 1 in the operation results, which are respectively recorded as the first quantity and the second quantity; and using the ratio of the first quantity to the second quantity as the permutation rate for the query period.

[0014] Preferably, the step of generating query strategy instructions and determining the units to be processed includes: if the replacement rate is greater than a preset value, executing a sharded query strategy; otherwise, executing a whole-shard query strategy; wherein, the sharded query strategy: calculating the distance between the key space distribution bitmaps of two adjacent time windows, selecting the time window corresponding to the maximum distance within the query period, using it as the dividing point, dividing the query period into two sub-periods as two independent units to be processed, executing the query operation in parallel, and then merging the query results; the whole-shard query strategy: executing the query operation as a single unit to be processed for the entire query period.

[0015] Preferably, the total baseline overhead of the calculation unit to be processed includes:

[0016] For each time window, the number of containers of all container types in the container type statistical vector is weighted and summed using the preset unit cost corresponding to each container type as the weight to obtain the theoretical cost of each time window.

[0017] The total baseline cost is positively correlated with the theoretical cost.

[0018] Preferably, obtaining the spatial flip number of the unit to be processed includes: performing a bitwise XOR operation on the key space distribution bitmap of two adjacent time windows, obtaining the XOR result bitmap, counting the total number of bits with a value of 1, and positively merging the total number of all two adjacent time windows within the unit to be processed as the spatial flip number of the unit to be processed.

[0019] Preferably, determining the type fluctuation of the unit to be processed includes:

[0020] For two adjacent time windows, calculate the difference in the number of containers of the same container type between their container type statistical vectors, and use this as the quantity difference. Then, sum the quantity differences of all container types as the total difference between the two adjacent time windows.

[0021] The fluctuation type is the result of positive fusion of the total difference between all two adjacent time windows within the unit to be processed.

[0022] Preferably, the penalty coefficient is positively correlated with both the number of space flips and the type fluctuation.

[0023] Preferably, the estimated instruction consumption is positively correlated with both the total baseline overhead and the penalty coefficient.

[0024] Preferably, the selection and execution of the corresponding calculation strategy includes: if the estimated instruction consumption is greater than a preset first threshold, an approximate calculation strategy is executed; otherwise, an accurate calculation strategy is executed.

[0025] Preferably, the precise calculation strategy includes: if the penalty coefficient is less than or equal to a preset second threshold, then calling the SIMD vectorized merging operator to perform the merging operation; otherwise, switching to the jump merging operator to perform the merging operation.

[0026] This application has at least the following beneficial effects:

[0027] This application calculates the replacement rate of a query period, which has the advantage of accurately quantifying the macroscopic stability of the data distribution throughout the query period. Based on this indicator, it automatically decides whether to process the entire data segment or in segments, fundamentally avoiding the performance penalty caused by forcibly merging segments with huge differences in data distribution. When the replacement rate is high, segmented processing can decompose complex queries into multiple more stable and easier-to-process subtasks, significantly reducing the range of key-value space involved in a single processing, improving the CPU cache hit rate, and thus reducing the total computation time. Calculating the total baseline overhead of the units to be processed has the advantage of weighting the number of containers by distinguishing different container types and assigning them different unit operation overhead weights, thus truly quantifying and reflecting the execution logic. The basic computational load required for the merge operation provides an accurate static cost baseline for subsequent decisions. The spatial flip number of the unit to be processed quantifies the oscillation of data in the key-value space dimension; frequent appearances or disappearances of high-level keys increase the comparison and jump overhead of the merge operation. The type oscillation of the unit to be processed quantifies the oscillation of data in the container type composition dimension; frequent container type conversions disrupt the continuity of the processing pipeline and increase instruction overhead. The penalty coefficient of the unit to be processed comprehensively reflects the degree of data turbulence within the unit by introducing spatial and type oscillations; a high penalty coefficient indicates that the actual execution overhead will be significantly higher than the static baseline overhead. The system can accurately predict the high CPU consumption of such tasks and determine the estimated instruction consumption of the units to be processed. Its advantage lies in combining the static total baseline overhead with a dynamic penalty coefficient to generate an estimated value that closely approximates the actual runtime consumption. Furthermore, it utilizes a fixed 8KB key space distribution bitmap for XOR operations, the overhead of which is far lower than loading and decompressing the bitmap entity data. This allows the system to perform a full estimate of all query requests without affecting query latency. Based on the estimated instruction consumption, the system selects and executes corresponding calculation strategies to complete the query of the units to be processed. Its advantage lies in using accurate estimated instruction consumption as a decision-making basis; when the consumption is too high, it automatically degrades to an approximate calculation strategy, trading acceptable precision loss for accuracy. The significant performance improvement ensures timely query response. When the consumption is controllable, a precise calculation strategy is adopted, and the optimal merging operator is further selected by combining the penalty coefficient. When the data is compact, the SIMD vectorized merging operator is used to maximize parallel throughput. When the data is sparse and fragmented, the jump merging operator is switched to intelligently skip key-value ranges that do not need to be compared, avoiding meaningless SIMD loading, comparison and masking operations on empty or misaligned data. This strategy ensures that the computing engine always runs in the mode most suitable for the current physical form of the data, avoiding the performance degradation caused by the large masking overhead and low effective load of SIMD instructions when processing highly misaligned data. Under the premise of ensuring query timeliness, the overall computing efficiency and hardware utilization are maximized. Attached Figure Description

[0028] The following description, in conjunction with the accompanying drawings, provides a more detailed explanation of a multi-dimensional analysis system for enterprise-level digital operation information according to this application.

[0029] Figure 1 A block diagram of a multi-dimensional analysis system for enterprise-level digital operation information provided in one embodiment of this application;

[0030] Figure 2 This is a block diagram illustrating the implementation of a prediction analysis module provided in one embodiment of this application. Detailed Implementation

[0031] The following description, in conjunction with the accompanying drawings and embodiments, provides a more detailed explanation of the multi-dimensional analysis system for enterprise-level digital operation information proposed in this application.

[0032] Please see Figure 1 The diagram illustrates a block diagram of a multi-dimensional analysis system for enterprise-level digital operation information provided in one embodiment of this application. The system includes: a metadata extraction module, a predictive analysis module, and a decision execution module.

[0033] The metadata extraction module is used to extract the container type statistical vector and key space distribution bitmap of bitmap objects in each time window within the specified query period when responding to a query request. The vector records the number of containers of each container type in the bitmap object, and the key space distribution bitmap records the existence distribution of the bitmap object in the high-level key value space.

[0034] In big data applications such as internet advertising, user profiling analysis, and real-time recommendations, the core business logic typically involves efficient set operations on massive amounts of data. These operations are characterized by huge data scale, high real-time requirements, and frequent queries, placing extremely high demands on the storage efficiency and computational performance of the underlying data structure. Bitmap indexes are a classic technique for handling set operations. The basic idea is to allocate a bit to each possible element value and perform set operations through efficient bitwise operations by the CPU. However, if the data range is large but the actual elements are sparse, it will consume a large amount of memory, causing serious space waste. Storing only existing elements saves space, but batch operations between sets require complex traversal and search, resulting in low computational efficiency.

[0035] Roaring Bitmap, as a highly efficient compressed bitmap indexing technology, can simultaneously achieve extreme memory efficiency and computation speed on sparse large datasets. It primarily employs a hybrid storage strategy, dividing a 32-bit integer into high 16 bits and low 16 bits. The high 16 bits serve as the key, dividing the global data space into up to 65536 independently manageable containers. The low 16 bits represent the specific value within each container, with its storage and computation method determined by the type of the container. This key-value separation two-level index structure allows each container to process only a maximum of 65536 (2^36) values. 16 The number of elements greatly reduces the complexity of a single container, allowing each container to independently optimize storage and computation.

[0036] Secondly, based on the actual density of data within each bucket, the optimal storage container is selected. The container types include sparse array containers, dense bitmap containers, and run-length encoded containers. For sparse data, i.e., a small number of elements (usually ≤4096), sparse array containers are used, resulting in minimal space usage when the data is extremely sparse. For dense data, dense bitmap containers are used, providing the fastest bit operation speed. For continuous data, i.e., when there are a large number of continuous values ​​in the data, run-length encoded containers are used. This adaptability allows Roaring Bitmap to maintain high performance in various data distribution scenarios.

[0037] Based on the above analysis, the bitmap object for a single time window is obtained as follows:

[0038] The system initializes an empty Roaring Bitmap object, determines the fixed time window to which the received event timestamp belongs, calls the RoaringBitmap.add method to add the user ID from the event to the bitmap object of the corresponding window, and when the time window ends, the RoaringBitmap.serialize method serializes the bitmap object into a byte stream and persists it for storage; at the same time, it reads the bitmap object corresponding to each time window, scans it and traverses all its internal containers, and counts the number of containers of each container type;

[0039] In this embodiment, the duration of the fixed time window is 10 seconds. As for other implementation methods, the implementer can set it according to the actual situation. Secondly, when accessing the bitmap object in memory, all containers inside it are traversed through the container iteration interface getContainerIterator provided by the Roaring Bitmap library. For each container, its specific type is determined and its corresponding high 16-bit key value is recorded.

[0040] Generate a container type statistics vector based on the number of containers of all container types corresponding to the bitmap object;

[0041] Among them, the Statistical vector of container types for bitmap objects corresponding to each time window Represented as: ,in, Indicates the first Each time window corresponds to the number of sparse array containers in the bitmap object. Indicates the first Each time window corresponds to the number of dense bitmap containers in the bitmap object. Indicates the first Each time window corresponds to the number of run-code containers in the bitmap object;

[0042] Therefore, the container type statistics vector occupies very little storage space, usually three 32-bit integers, totaling 12 bytes, yet it completely preserves the type composition characteristics of the data within that time window.

[0043] Secondly, during the merging of multi-way bitmaps, the failure of CPU branch prediction mainly stems from the inconsistent distribution of the containers involved in the merging process in the memory address space, i.e., key-value misalignment. To quantify the topological characteristics of this memory layout, the existence of bitmap objects in the high 16 bits is mapped to extract the key space distribution bitmap, specifically:

[0044] The system introduces a fixed-length 65536-bit key space distribution bitmap. This bitmap records the existence of the high 16 bits of the key within a time window. By traversing all containers in the bitmap object, the high 16 bits of the key corresponding to each container are extracted. For each existing key... The first bit in the key space distribution bitmap A key value is 1 if the corresponding bit in the key space distribution bitmap is 0.

[0045] Among them, the The key space distribution bitmap of each time window corresponds to the bitmap object. for:

[0046]

[0047] Therefore, the size of the key space distribution bitmap is strictly fixed at 8KB. Compared to the original bitmap object size, which is typically in the MB or even GB range, 8KB of data can reside entirely in the CPU's L1 or L2 cache. This allows for subsequent... Bitwise operations will be executed at the speed of CPU registers in nanoseconds, eliminating the latency of accessing main memory.

[0048] To ensure that subsequent query steps can quickly read the above features without loading the original data, for the first... The statistical vectors of container types for each time window and the bitmap of key space distribution are combined and encapsulated into a unified metadata object. It is serialized and written to a separate storage area dedicated to metadata storage;

[0049] in, Logically, each bitmap object corresponds one-to-one with the original bitmap object, representing the data characteristics and complete content of the same time window. However, they are completely isolated in terms of physical storage path, so that only KB-level metadata is read during the query decision stage, and MB-level entity data is loaded on demand during the actual calculation stage, which ensures the efficiency of the estimation process from the I / O level.

[0050] Since performing a Roaring Bitmap query requires logical merging operations on bitmap objects across multiple time windows, and the actual computational overhead of the merging operation is highly dependent on the internal structural matching degree of the bitmap objects involved in the merging, it cannot be accurately predicted before the merging is executed. Therefore, by extracting lightweight metadata objects, the computational complexity of the merging operation is estimated, and a merging strategy is selected and executed accordingly.

[0051] Therefore, when responding to a query request, the metadata object for each time window within the query period is extracted by accessing a separate metadata storage area. The metadata object contains a container type statistical vector and a key space distribution bitmap.

[0052] It should be noted that the query period refers to the time range specified in the query request.

[0053] At this point, we have obtained the container type statistical vector and key space distribution bitmap for each time window within the query period.

[0054] The prediction and analysis module is used to obtain the estimated command consumption.

[0055] Furthermore, the implementation block diagram of the prediction and analysis module provided in this application is as follows: Figure 2 As shown.

[0056] (1) Analyze the distribution difference ratio between different key space distribution bitmaps, calculate the replacement rate of the query period, generate query strategy instructions and determine the unit to be processed; use the number of containers of different container types in the container type statistical vector to evaluate the expected resource overhead consumed by the logical merging operation of the unit to be processed, and calculate the total baseline overhead of the unit to be processed.

[0057] To detect whether there has been a large-scale migration of user groups in business traffic, the replacement rate is calculated by analyzing the differences in the container layout of the key space distribution bitmap within a time window. Specifically:

[0058] Define the time window preceding the first time window within the query period as the reference window; perform a bitwise XOR operation on the key space distribution bitmap between the last time window within the query period and the reference window, and count the number of bits with a value of 1 in the operation result, which is the first count;

[0059] Perform a bitwise OR operation on the key space distribution bitmap between the last time window and the reference window within the query period, and count the number of bits with a value of 1 in the operation result, which is the second count;

[0060] The ratio of the first quantity to the second quantity is used as the replacement rate for the query period;

[0061] It should be noted that both bitwise XOR and bitwise OR operations are well-known techniques and will not be elaborated here. Bitwise XOR operation means that if the value of the same bit in two key space distribution bitmaps is the same, the result of that bit is 0, otherwise it is 1. Bitwise OR operation means that as long as one of the values ​​of the same bit is 1, the result is 1. The closer the permutation rate is to 1, the more it indicates that the active key value spaces of the end time window and the reference window are almost completely non-overlapping, which indicates that direct merging will lead to an extremely low cache hit rate, and a sharding strategy is required.

[0062] When querying over a long time period, the memory access pattern is optimized using the replacement rate, specifically:

[0063] If the replacement rate is greater than the preset value, the sharded query strategy is executed; otherwise, the whole shard query strategy is executed.

[0064] The sharded query strategy calculates the distance between the key space distribution bitmaps of adjacent time windows based on the key space distribution bitmaps of all time windows within the query period. It then selects the time window with the largest distance within the query period and uses it as the dividing point to divide the original query period into two independent processing units. These two units execute query operations in parallel, and the results are then finally merged. This strategy significantly reduces the key-value space range involved in a single merging task, improves CPU cache hit rate, and thus reduces the overall computation time.

[0065] The whole-time query strategy treats the entire query period as a single unit to be processed and executes the query operation.

[0066] In this embodiment, the preset value is 0.8. In other implementations, the implementer can set it according to the actual situation. Secondly, the distance is calculated by measuring the Jaccard distance between the key space distribution bitmaps of two adjacent time windows. The calculation of the Jaccard distance is a well-known technique and will not be elaborated here. For ease of understanding, it is assumed that the query period is... , with dividing point Using this as a boundary, the query period is divided into two independent subquery tasks, where the first subquery covers a time range of... The second subquery covers a time range of 10 ... Each subquery's time range is then calculated separately, and the results are summed to represent the evaluation process for the entire query period.

[0067] It should be noted that if the replacement rate is greater than the preset value, it means that the active key-value spaces are almost completely non-overlapping. If the entire query period is directly merged, the computing engine needs to maintain a huge result set bitmap to accommodate two completely different key-value spaces. Moreover, the merging process needs to jump back and forth between two completely discontinuous key-value intervals, which will cause the L3 cache to become invalid.

[0068] Furthermore, before performing large-scale data merging operations, it is necessary to estimate the computational overhead of the Roaring Bitmap merging operation to optimize system performance and resource allocation. In Roaring Bitmap, the computational overhead differs for different types of containers when performing logical merging operations. Secondly, the varying number of elements within a single container also leads to fluctuations in computational overhead. Therefore, during system initialization and deployment or periodic maintenance, the focus is not on measuring the precise physical execution time of a single container, but rather on measuring the overhead of different container types under the current hardware environment in large-scale data processing scenarios. The specific testing process is as follows:

[0069] The system constructs a benchmark test set containing randomly generated data, which covers three container types: sparse array container, dense bitmap container, and run-length encoded container.

[0070] For each container type, the system performs no less than 1 million random logical merge operations, and records the number of CPU instruction cycles consumed for each operation by reading the hardware performance counter (PMC), and takes the average number of CPU instruction cycles for all operations under each container type as the unit cost.

[0071] It should be noted that unit overhead refers to the average instruction overhead required to process a container of a corresponding container type; in the hardware environment of this embodiment, the typical ratio of unit overhead for sparse array containers, dense bitmap containers, and run-length encoded containers is approximately... As another implementation method, the implementer can set it according to the actual hardware environment.

[0072] Secondly, based on the number of containers of different container types and the unit cost in the container type statistical vector of a single time window, the theoretical cost is calculated, specifically:

[0073] For each time window, the number of containers of all container types in the container type statistics vector is weighted and summed, with the unit cost of each container type as the weight, to obtain the theoretical cost of each time window.

[0074] Among them, the The theoretical cost of a time window The calculation formula is:

[0075]

[0076] in, For the unit overhead of sparse array containers, For the unit overhead of dense bitmap containers, Unit overhead for the run-code container;

[0077] The total baseline cost of the unit to be processed is positively correlated with the theoretical cost.

[0078] It should be noted that a positive correlation means that the dependent variable increases as the independent variable increases and decreases as the independent variable decreases.

[0079] In this embodiment, the sum of the theoretical overhead costs of all time windows within the unit to be processed is used as the total baseline overhead.

[0080] It should be noted that the greater the theoretical overhead, the more containers this time window contains, the more complex the data, the longer the expected processing time, and the more CPU cycles are needed to process the data in this window; the greater the total baseline overhead, the larger the amount of data involved in the query, the higher the total computational load in global bitmap merging or querying, and the more computational resources are required.

[0081] At this point, the total baseline cost of the unit to be processed is obtained.

[0082] (2) Analyze the distribution differences of the key space distribution bitmap between adjacent time windows to obtain the space flip number of the unit to be processed; determine the type fluctuation of the unit to be processed by the differences of the container type statistical vector between adjacent time windows, and obtain the penalty coefficient of the unit to be processed by combining the space flip number; determine the estimated instruction consumption of the unit to be processed based on the total baseline overhead and the penalty coefficient.

[0083] Furthermore, due to the fluctuating existence of containers in the memory address space, this fluctuation can lead to frequent "blank jumps" or "insertions" during key space traversal in merge operations, causing CPU pipeline interruptions. Therefore, we analyze the changes in container existence at the same bit position in the key space distribution bitmap between two adjacent time windows and calculate the space flip number, specifically:

[0084] Perform a bitwise XOR operation on the key space distribution bitmaps of two adjacent time windows to obtain the XOR result bitmap;

[0085] It should be noted that the bitwise XOR operation is a well-known technique and will not be elaborated here; if the value of the same bit in two key space distribution bitmaps is the same, the result of that bit is 0, otherwise it is 1.

[0086] The total number of bits with a value of 1 in the XOR result bitmap is counted, and the total number of all two adjacent time windows within the unit to be processed is positively merged to obtain the spatial flip number of the unit to be processed.

[0087] In this embodiment, the specific process of forward fusion is as follows: the sum of the total number of all two adjacent time windows in the unit to be processed is used as the spatial flip number. In other implementation methods, the implementer may also use the average of the total number of all two adjacent time windows in the unit to be processed as the spatial flip number.

[0088] It should be noted that when analyzing the spatial flip number, for the first time window in the unit to be processed, it is necessary to read the metadata object of the previous time window. If the metadata object of the previous time window does not exist, an empty metadata object is filled in.

[0089] It should be noted that the larger the total number of bits in the XOR result bitmap, the greater the change in the key space distribution between two adjacent time windows, meaning that more containers at key value positions have undergone existence flips. The larger the number of space flips, the greater the overall change in the key space distribution within the entire processing unit, meaning that more containers at key value positions have undergone existence flips between different time windows. This reflects a large fluctuation in the memory address space of the containers within the entire processing unit, which may lead to more CPU pipeline interruptions, thereby increasing computational overhead.

[0090] Secondly, the space flip number is insufficient to fully capture all scenarios that cause CPU branch prediction to fail. Even if the container at the key-value position always exists, a change in the container type can also cause CPU branch prediction to fail. For example, if a key-value position... In its first The first time window is a sparse array container, while the second time window is a sparse array container. Each time window is a dense bitmap container, at which point the key-value positions are... If a container always exists, the XOR result is 0. Therefore, although the space flip count can detect and reflect whether a container exists at a key position (i.e., a container appears from nothing or disappears), it cannot detect changes in container type. Changes in container type can lead to completely different merge logic, and such type mutations can also cause CPU branch prediction to fail. Based on this analysis, the type fluctuation is calculated by analyzing the changes in the number of container types between two adjacent time windows. Specifically:

[0091] For two adjacent time windows, calculate the difference in the number of containers of the same container type between their container type statistical vectors, and use this as the quantity difference. Then, sum the quantity differences of all container types as the total difference between the two adjacent time windows.

[0092] In this embodiment, the absolute value of the difference in the number of containers of the same container type is calculated as the quantity difference. The calculation process for the total difference between two adjacent time windows is as follows:

[0093]

[0094] in, For the first The first time window and the first The total difference between time windows For the first The number of sparse array containers within a given time window For the first The number of dense bitmap containers within a time window For the first The number of run-code containers within a time window.

[0095] The total difference between all two adjacent time windows within the unit to be processed is positively fused and used as the type fluctuation of the unit to be processed.

[0096] In this embodiment, the sum of the total differences between all two adjacent time windows within the unit to be processed is used as the type fluctuation.

[0097] It should be noted that the total difference indicates a larger change in container type between two adjacent time windows, reflecting a more drastic change in data structure; the greater the fluctuation in the obtained type, the larger the overall change in container type within the entire unit to be processed, reflecting that the data involved in the query is more unstable and the risk of branch prediction failure faced by the merge operation is higher.

[0098] Furthermore, the spatial flip number reflects the spatial location oscillation, and the type volatility reflects the container type oscillation. Therefore, based on the spatial flip number and the type volatility, the penalty coefficient is determined as follows:

[0099] The penalty coefficient of the unit to be processed is positively correlated with both the number of space flips and the type fluctuation.

[0100] In this embodiment, the calculation process of the penalty coefficient is as follows: the space flip number and the type fluctuation are normalized respectively, wherein the normalization process of the space flip number is as follows:

[0101] Perform a bitwise OR operation on the key space distribution bitmap of all time windows within the processing unit, and count the total number of bits that are 1 in the operation result as the total number of key space coverages.

[0102] It should be noted that bitwise OR operation is a well-known technique and will not be elaborated here. In this technique, if any one of the values ​​of the same bit is 1, the result is 1.

[0103] The ratio of the number of space flips to the total number of key space covers is used as the normalized number of space flips.

[0104] It should be noted that the total key space coverage reflects the total number of all bit positions where data has ever appeared.

[0105] The normalization process for type volatility is as follows: count the total number of containers in all time windows within the unit to be processed; and use the ratio of type volatility to the total number of containers as the normalized type volatility.

[0106] Based on the preset first weight factor and the preset second weight factor, the normalized spatial flip number and the normalized type fluctuation are weighted and summed, and the sum of the sum with the value of 1 is used as the penalty coefficient of the unit to be processed.

[0107] Since the negative impact of spatial misalignment on performance is slightly greater than that of type change, the preset first weight factor is set to 2 and the preset second weight factor is set to 1.5. As other implementation methods, implementers can set them according to the actual situation. Secondly, the purpose of adding the value 1 is to establish a baseline so that when the data is completely stable, that is, when there is no spatial misalignment or type fluctuation, the penalty coefficient is 1. At this time, the estimated cost degenerates to a linear baseline value. However, when the data is unstable, for every additional unit of spatial oscillation or type oscillation, the estimated calculation cost will increase by a corresponding multiple.

[0108] It should be noted that the larger the penalty coefficient, the higher the instability of the data, that is, there are more spatial oscillations or type oscillations, reflecting that the actual computational cost of the merging operation is higher than the theoretical cost.

[0109] Furthermore, based on the total baseline overhead and penalty coefficient, the estimated instruction consumption is determined as follows:

[0110] The estimated instruction consumption of the unit to be processed is positively correlated with the total baseline overhead and the penalty coefficient;

[0111] In this embodiment, the product of the total baseline overhead and the penalty coefficient is used as the estimated instruction consumption.

[0112] It should be noted that the greater the estimated instruction consumption, the more computing resources the query may require, the longer the execution time, and the greater the load pressure on the system.

[0113] At this point, the estimated instruction consumption of the unit to be processed is obtained.

[0114] The decision execution module is used to select and execute the corresponding calculation strategy based on the estimated instruction consumption in order to complete the query of the unit to be processed.

[0115] Furthermore, to prevent highly fragmented storage container layouts or excessively large query tasks from exhausting the CPU resources of the computing cluster, the system first performs circuit breaker protection judgment based on the estimated instruction consumption, specifically:

[0116] If the estimated instruction consumption is greater than the preset first threshold, then this query is a high-risk task and an approximate calculation strategy is executed; otherwise, an exact calculation strategy is executed.

[0117] The approximate calculation strategy involves using the HyperLogLog estimation algorithm to approximate the query results for the unit to be processed and returning a statistical result; thus reducing the complexity from... Reduced to ;

[0118] In this embodiment, the process of obtaining the preset first threshold is as follows: real-time calculation of the load water level threshold. As a preset first threshold, during the system cold start phase, It is defaulted to 80% of the maximum instructions per second (IPS) of a single core, for example, for a 3GHz CPU, initially During system operation, the monitoring daemon dynamically adjusts the CPU utilization based on the average CPU utilization over the past minute, using a PID feedback control algorithm. When the average CPU utilization exceeds 90%, reduce it proportionally. Tighten access controls; increase the threshold when the average CPU utilization is below 50%. To free up computing power, the PID feedback control algorithm is a well-known technology and will not be elaborated here.

[0119] It should be noted that if the estimated instruction consumption is less than or equal to the preset first threshold, it means that the query is within a safe range and the subsequent precise calculation process will continue.

[0120] When executing a precise computation strategy, the system needs to select the underlying merging algorithm that best suits the current data physical structure. Since the penalty coefficient reflects the degree of data interleaving in the memory key-value space, it determines the effective load rate of SIMD vectorized instructions. Therefore, the precise computation strategy includes:

[0121] If the penalty coefficient is less than or equal to the preset second threshold, the SIMD vectorized merging operator is invoked; otherwise, the skip merging operator is switched to.

[0122] In this embodiment, the preset second threshold value is 1.2. In other implementation methods, the implementer can set it according to the actual situation.

[0123] Then, based on the result of the final logical merging operation, the response result of the query request is output;

[0124] It should be noted that if the entire query period contains multiple pending units, the results of the logical merging operations corresponding to all pending units will be merged to output the response result of the query request.

[0125] It should be noted that if the penalty coefficient is less than or equal to the preset second threshold, it indicates that the data involved in the query is well aligned in the key-value space and has high consistency in container types. By calling a SIMD vectorized merging operator, such as the AVX-512 instruction set, this operator can utilize the continuity of data to process 512-bit (i.e., 16 32-bit integers) data blocks in parallel with a single instruction and with very little mask loading overhead, thus achieving extreme computational speedup. If the penalty coefficient is greater than the preset second threshold, it indicates that the data structure has severe interleaving and fragmentation. In this scenario, if SIMD vectorized merging is forcibly used... The merging operator leads to a large amount of empty data padding and invalid mask logic operations, and frequent type mismatches can disrupt the vector pipeline. The skip-scan merging operator, based on scalar code logic, utilizes key-value indexes in metadata to quickly skip sparse key-value regions that only exist in parts of the bitmap, directly jumping to the next common key-value. Although scalar instructions process less data per operation, in extremely sparse and misaligned data distributions, this on-demand access strategy avoids the invalid computation of full scans, and its actual execution efficiency is superior to SIMD operators. This strategy essentially switches the computation mode from throughput-first to latency-first to adapt to fragmented data patterns.

Claims

1. A multi-dimensional analysis system for enterprise-level digital operation information, characterized in that, The system includes: The metadata extraction module is used to extract the container type statistical vector and key space distribution bitmap of bitmap objects in each time window within the specified query period when responding to a query request. The vector records the number of containers of each container type in the bitmap object, and the key space distribution bitmap records the existence distribution of the bitmap object in the high-level key value space. The prediction and analysis module is used to obtain the estimated command consumption, specifically: Analyze the proportion of distribution differences between bitmaps with different key space distributions, calculate the replacement rate for the query period, generate query strategy instructions, and determine the units to be processed. By using the number of containers of different container types in the container type statistics vector, the expected resource overhead consumed by performing logical merging operations on the unit to be processed is evaluated, and the total baseline overhead of the unit to be processed is calculated. The distribution differences of the key space distribution bitmap between adjacent time windows are analyzed to obtain the spatial flip number of the unit to be processed; the type fluctuation of the unit to be processed is determined by the differences of the container type statistical vector between adjacent time windows, and the penalty coefficient of the unit to be processed is obtained by combining the spatial flip number. Based on the total baseline overhead and penalty coefficient, the estimated instruction consumption of the unit to be processed is determined; The decision execution module is used to select and execute the corresponding calculation strategy based on the estimated instruction consumption in order to complete the query of the unit to be processed.

2. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The calculation of the permutation rate for the query period includes: defining the time window preceding the first time window within the query period as the reference window; performing bitwise XOR and bitwise OR operations on the key space distribution bitmap between the last time window and the reference window within the query period, and counting the number of bits with a value of 1 in the operation results, which are respectively recorded as the first quantity and the second quantity; and using the ratio of the first quantity to the second quantity as the permutation rate for the query period.

3. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The process of generating query strategy instructions and determining the units to be processed includes: if the replacement rate is greater than a preset value, executing a sharded query strategy; otherwise, executing a whole-shard query strategy. The sharded query strategy involves calculating the distance between the key space distribution bitmaps of two adjacent time windows, selecting the time window corresponding to the maximum distance within the query period, using it as a dividing point to divide the query period into two sub-periods as two independent units to be processed, executing the query operation in parallel, and then merging the query results. The whole-shard query strategy treats the entire query period as a single unit to be processed and executes the query operation.

4. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The total baseline overhead for calculating the unit to be processed includes: For each time window, the number of containers of all container types in the container type statistical vector is weighted and summed using the preset unit cost corresponding to each container type as the weight to obtain the theoretical cost of each time window. The total baseline cost is positively correlated with the theoretical cost.

5. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The process of obtaining the spatial flip number of the unit to be processed includes: performing a bitwise XOR operation on the key space distribution bitmap of two adjacent time windows, obtaining the XOR result bitmap, counting the total number of bits with a value of 1, and positively merging the total number of all two adjacent time windows within the unit to be processed as the spatial flip number of the unit to be processed.

6. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The determination of the type fluctuation of the unit to be processed includes: For two adjacent time windows, calculate the difference in the number of containers of the same container type between their container type statistical vectors, and use this as the quantity difference. Then, sum the quantity differences of all container types as the total difference between the two adjacent time windows. The fluctuation type is the result of positive fusion of the total difference between all two adjacent time windows within the unit to be processed.

7. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The penalty coefficient is positively correlated with both the number of space flips and the type fluctuation.

8. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The estimated instruction consumption is positively correlated with both the total baseline cost and the penalty coefficient.

9. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 1, characterized in that, The selection and execution of the corresponding calculation strategy includes: if the estimated instruction consumption is greater than a preset first threshold, an approximate calculation strategy is executed; otherwise, an accurate calculation strategy is executed.

10. The multi-dimensional analysis system for enterprise-level digital operation information as described in claim 9, characterized in that, The precise calculation strategy includes: if the penalty coefficient is less than or equal to a preset second threshold, then the SIMD vectorized merging operator is invoked to perform the merging operation; otherwise, the merging operation is switched to the jump merging operator.

Images