Bitmap index container optimization method and system based on multi-dimensional dynamic decision

Abstract

The invention provides a bitmap index container optimization method and system based on a multi-dimensional dynamic decision, and relates to the technical field of databases, and the method comprises the steps: constructing a feature fusion vector by calculating a skewness coefficient value and fusing a data distribution feature vector, a data access rule vector and a change trend vector; a self-adaptive threshold value is obtained through iterative optimization of a double-strategy gradient algorithm, whether the container type is switched or not is judged, data deduplication, priority ranking and batch data migration are conducted through a temporary container, and a bidirectional index table is established to redirect an access request.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and particularly relates to a bitmap index container optimization method and system based on multi-dimensional dynamic decision. BACKGROUND

[0002] With the rapid development of big data application, bitmap index technology in database system is widely used to improve query efficiency and data processing performance, and bitmap index container as an important carrier for storing and managing bitmap index has an important influence on the overall system performance. In traditional database application, bitmap index is mainly used for fast retrieval of low cardinality data column, and with the expansion of application scenarios, bitmap index has been widely used in various environments, and the traditional bitmap index container design usually adopts a static fixed container type, which cannot be dynamically adjusted according to the changes of data characteristics and access mode. The distribution characteristics, access frequency and change trend of data will change significantly over time and business scenarios, and the fixed container type is difficult to adapt to the dynamic changing needs, especially in high concurrency and large-scale data processing scenarios, inappropriate container type selection will cause significant decline in system performance. SUMMARY

[0003] The embodiment of the present application provides a bitmap index container optimization method and system based on multi-dimensional dynamic decision, which can at least solve some problems in the prior art.

[0004] In a first aspect of the embodiment of the present application, a bitmap index container optimization method based on multi-dimensional dynamic decision is provided, comprising: obtaining data characteristic values of the bitmap index container and calculating skewness coefficient values based on the data characteristic values; processing the data characteristic values through a pre-set time sequence diagram structure to obtain a data distribution characteristic vector, determining a data access rule vector and a corresponding data change trend vector based on the data characteristic values, combining the data distribution characteristic vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimizing the decision parameters based on the feature fusion vector combined with a double strategy gradient algorithm to obtain an adaptive threshold for container switching; comparing the skewness coefficient values with the adaptive threshold and determining whether to switch the container type; When container type switching is needed, a temporary container is created to perform deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, the heat value and the correlation value of the deduplicated data are calculated, the weighted sum of the heat value and the correlation value is taken as the priority value, the deduplicated data is sorted to obtain a to-be-migrated data sequence, the to-be-migrated data sequence is written into the temporary container in batches, and a write state value is recorded; when the write state value indicates that the pre-commit is successful, a confirm commit is performed, a bidirectional index table is generated, and the bidirectional index table is used to redirect data access requests; The performance index value after switching is monitored, and when the performance index value exceeds a preset range, a rollback operation is performed and the decision parameter is updated.

[0005] In an optional implementation, The data characteristic value of the bitmap index container is obtained, and a skewness coefficient value is calculated based on the data characteristic value, which includes: The number of writes and the number of reads of the bitmap unit in the bitmap index container are counted within a preset first time window, the distribution density of the data storage location in the bitmap unit is calculated, and the data characteristic value is obtained; The data characteristic value is divided into a plurality of data segments according to a preset rule, the distribution characteristics of each data segment are calculated respectively, and a probability distribution sequence is obtained by combination; The skewness coefficient value corresponding to the bitmap index container is calculated based on the probability distribution sequence.

[0006] In an optional implementation, The data distribution feature vector is obtained by processing the data characteristic value through a pre-set timing diagram structure, the data access rule vector and the corresponding data change trend vector are determined based on the data characteristic value, and the feature fusion vector is obtained by combining the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions, which includes: The timing diagram structure is constructed based on a pre-set time interval, and a plurality of time nodes are recorded, the bitmap unit state parameters of each time node are calculated, the state transition matrix is constructed according to the bitmap unit state parameters, and the data distribution feature vector is obtained by feature decomposition, wherein the bitmap unit state parameters include the occupancy state, the modification time and the index information; The data blocks in the bitmap unit are obtained from the timing diagram structure, the standard access frequency of the data blocks within a preset time period and the continuous access frequency between the data blocks are recorded, the ratio of the continuous access frequency to the standard access frequency is constructed as an access pattern matrix, the normalized Laplacian matrix of the access pattern matrix is calculated, and the feature vector is extracted as the data access rule vector; According to a preset time length, a time window sequence is divided, a data operation frequency in each time window is recorded, a ratio of the data operation frequency to a total operation frequency in the time window is calculated to obtain an operation proportion parameter, the operation proportion parameter is multiplied by a corresponding preset weight coefficient and summed to obtain a change trend index, and time series analysis is performed on the change trend index to obtain a data change trend vector, wherein the data operation frequency includes a data write frequency, a data update frequency and a data deletion frequency. The data distribution feature vector, the data access rule vector and the data change trend vector are subjected to dimension alignment and normalization processing, and a feature fusion vector is obtained by weighted summation of the processing results.

[0007] In an optional implementation, Based on the feature fusion vector, a double strategy gradient algorithm is combined to iteratively optimize the decision parameter to obtain an adaptive threshold of container switching, including: A historical data sequence is constructed by acquiring historical data, a time decay coefficient sequence is constructed as a time series decay weight based on the length of the historical data sequence, a gradient of each feature in the feature fusion vector to a pre-set initial strategy function is calculated, and a two-norm of the gradient is multiplied by the time series decay weight and accumulated to obtain an importance score value of each feature; Based on the importance score value, a set of attention transformation matrices is constructed, a query vector, a key value vector and a numerical value vector are obtained by multiplying the feature fusion vector and the set of attention transformation matrices, a dot product of the query vector and the key value vector is calculated, and after softmax normalization, the numerical value vector is multiplied to obtain a reconstructed feature representation, one-dimensional convolution operation and ReLU function activation are performed on the reconstructed feature representation, a compressed feature representation is obtained through a full connection layer mapping, a feature selection mask is determined through a sigmoid function and an element-wise multiplication operation is performed on the compressed feature representation to obtain a selected feature representation; Based on a Gaussian distribution, a random noise vector is constructed, the selected feature representation and the random noise vector are connected to form a feature supplement input, a feature supplement value is generated by combining a nonlinear transformation, the feature supplement value and the selected feature representation are added to obtain a supplemented feature representation; Based on the initial strategy function, a strategy gradient corresponding to the supplemented feature representation is calculated, the strategy gradient is multiplied by a pre-set adaptive learning rate to update the parameters in the initial strategy function, and an adaptive threshold of container switching is calculated based on the updated strategy parameters.

[0008] In an optional implementation, When container type switching is needed, a temporary container is created to perform deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, a heat value and a correlation value of the deduplicated data are calculated, a weighted sum of the heat value and the correlation value is taken as a priority value, and the deduplicated data is sorted to obtain a to-be-migrated data sequence including: When container type switching is needed, a temporary container is created and the to-be-migrated data in the bitmap index container is input into the temporary container, a feature fingerprint of each piece of data in the to-be-migrated data is calculated by using a local sensitive hash algorithm, a data similarity between any two pieces of to-be-migrated data in the to-be-migrated data is calculated based on the feature fingerprint, the data similarity is compared with a preset similarity threshold, when the data similarity is greater than the preset similarity threshold, it is determined that the data is repeated and is removed, and deduplicated data is obtained; The access frequency of the deduplicated data in a second sliding time window is counted, a time weight is calculated by calculating a time interval from the last access to the current time, a survival time is calculated by calculating a time interval from the data generation time to the current time, and a heat value is calculated based on the access frequency, the time weight and the survival time; An association weight is calculated by calculating a reference relationship between each two pieces of data in the deduplicated data, a semantic similarity between each two pieces of data is calculated, the association weight is multiplied by the semantic similarity and an average value is calculated to obtain a correlation value; According to task requirements, weight coefficients corresponding to the heat value and the correlation value are respectively initialized and weighted sum is performed to obtain a priority value, and the deduplicated data is sorted in descending order based on the priority value to obtain a to-be-migrated data sequence.

[0009] In an optional implementation, The to-be-migrated data sequence is written into the temporary container in batches and a write state value is recorded, when the write state value indicates that pre-commit is successful, a confirmation commit is performed, a bidirectional index table is generated, and a data access request is redirected by using the bidirectional index table including: The to-be-migrated data sequence is grouped into multiple data batches according to a preset batch size, the multiple data batches are written into the temporary container, and a write state value is recorded for each data batch in the multiple data batches; A hash value of the data in each data batch is calculated and summed to obtain a data checksum, the data checksum after writing is compared with the data checksum before writing to obtain an integrity verification result, a pre-commit log record is generated based on the integrity verification result, and when the write state value indicates that pre-commit is successful, a confirmation commit operation is performed; obtaining an original data identifier of to-be-migrated data from the bitmap index container, obtaining a new data identifier of the to-be-migrated data from the temporary container, establishing a mapping relationship and adding a version number and a timestamp to generate metadata information, and constructing a bidirectional index table based on the original data identifier, the new data identifier and the metadata information; storing the mapping relationship between the original data identifier and the new data identifier into a redirection cache, receiving a data access request, preferentially searching for a target data position from the redirection cache, searching for the target data position in the bidirectional index table if the target data position is not found, judging whether a retry number exceeds a search number threshold if the search fails, retrying if the retry number does not exceed the search number threshold, accessing data in the bitmap index container if the retry number exceeds the search number threshold and a read operation is performed, and rejecting the data access request if the retry number exceeds the search number threshold and a write operation is performed.

[0010] In an optional implementation, monitoring a performance index value after switching, and performing a rollback operation and updating a decision parameter when the performance index value exceeds a preset range, including: obtaining a performance index value after switching, the performance index value including a service response time, a resource occupancy rate and a service success rate; judging whether the performance index value exceeds a preset performance threshold range, the preset performance threshold range being determined based on a statistical distribution of historical performance data, and triggering a rollback operation when a duration during which the performance index value continuously exceeds the preset performance threshold range is greater than a preset time threshold; performing the rollback operation and updating a decision parameter based on a performance fluctuation of the current switching, the decision parameter including a switching time window, a performance monitoring frequency and a performance threshold interval.

[0011] In a second aspect of the embodiment of the application, a bitmap index container optimization system based on multi-dimensional dynamic decision is provided, including: a first unit configured to obtain a data characteristic value of a bitmap index container and calculate a skewness coefficient value based on the data characteristic value; a second unit configured to process the data characteristic value through a pre-set timing diagram structure to obtain a data distribution feature vector, determine a data access rule vector and a corresponding data change trend vector based on the data characteristic value, combine the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimize a decision parameter based on the feature fusion vector and in combination with a double strategy gradient algorithm to obtain an adaptive threshold for container switching; a third unit configured to compare the skewness coefficient value with the adaptive threshold and judge whether to perform container type switching; The fourth unit is configured to create a temporary container, perform deduplication operation on the to-be-migrated data in the temporary container, calculate the heat value and the correlation value of the deduplicated data, take the weighted sum of the heat value and the correlation value as the priority value, sort the deduplicated data to obtain a to-be-migrated data sequence, write the to-be-migrated data sequence into the temporary container in batches and record the write state value, perform a confirm commit when the write state value indicates that the pre-commit is successful, generate a bidirectional index table and redirect a data access request by using the bidirectional index table. The fifth unit is configured to monitor the performance index value after the switching, perform a rollback operation and update the decision parameter when the performance index value exceeds a preset range.

[0012] In a third aspect, the embodiment of the present application provides an electronic device, comprising: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0013] In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method described above.

[0014] In the present application, the data characteristic value of the bitmap index container is obtained, the skewness coefficient value is calculated, the multi-dimensional feature vector obtained by processing the time sequence diagram structure is combined, the double strategy gradient algorithm is used to optimize the decision parameter, the intelligent decision of the container type switching is realized, the adaptability of the system to different data distribution scenarios is effectively improved, the deduplication operation is performed by creating a temporary container, the data priority is calculated based on the heat value and the correlation value, the ordered migration of data is realized, the bidirectional index table is introduced to redirect the access request, the smoothness of the container switching process and the continuity of the data access are ensured, the system switching risk is reduced, the performance monitoring and rollback mechanism are introduced, the rollback operation is automatically performed when the performance index value after the switching exceeds a preset range, the closed-loop optimization is formed by updating the decision parameter, the stability and reliability of the system are improved, and the self-adaptation ability of the system is enhanced by continuously learning and optimizing the decision model. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 FIG. 1 is a flowchart of a bitmap index container optimization method based on multi-dimensional dynamic decision according to an embodiment of the present application; Figure 2 FIG. 2 is a data migration logic flowchart of the bitmap index container optimization method based on multi-dimensional dynamic decision according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments can not be described again for the same or similar concepts or processes.

[0018] Figure 1 The flowchart of the bitmap index container optimization method based on multi-dimensional dynamic decision of the embodiments of the present application is shown in FIG. 1, which comprises the following steps. Figure 1 obtaining data characteristic values of the bitmap index container and calculating skewness coefficient values based on the data characteristic values; processing the data characteristic values through a pre-set timing diagram structure to obtain a data distribution feature vector, determining a data access rule vector and a corresponding data change trend vector based on the data characteristic values, combining the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimizing decision parameters based on the feature fusion vector in combination with a double strategy gradient algorithm to obtain an adaptive threshold value of container switching; comparing the skewness coefficient values with the adaptive threshold value and determining whether to switch the container type; when it is necessary to switch the container type, creating a temporary container to perform a deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, calculating a heat value and a correlation value of the deduplicated data, taking a weighted sum of the heat value and the correlation value as a priority value and sorting the deduplicated data to obtain a to-be-migrated data sequence, writing the to-be-migrated data sequence into the temporary container in batches and recording a write state value, performing a confirm commit when the write state value indicates that a pre-commit is successful, generating a bidirectional index table and redirecting a data access request by using the bidirectional index table; monitoring a performance index value after switching, and performing a rollback operation and updating decision parameters when the performance index value exceeds a preset range.

[0019] In an optional implementation, obtaining data characteristic values of the bitmap index container and calculating skewness coefficient values based on the data characteristic values comprises: counting the number of writes and reads of bitmap units in the bitmap index container within a preset first time window, calculating the distribution density of data storage positions in the bitmap units to obtain the data characteristic values; ​The data characteristic values are divided into multiple data segments according to a preset rule, the distribution characteristics of each data segment are calculated respectively, and a probability distribution sequence is obtained by combination; The skewness coefficient value corresponding to the bitmap index container is calculated based on the probability distribution sequence.

[0020] The read-write characteristics of the bitmap units in the bitmap index container are analyzed. In a preset first time window, the number of writes and the number of reads of each bitmap unit are recorded. The time window can be set to 30 minutes. In this time window, the operations of each bitmap unit in the bitmap index container are counted by the monitoring module. For example, a certain bitmap index container contains 1000 bitmap units. In a 30-minute time window, the number of writes of bitmap unit 1 is recorded as 150 times, and the number of reads is 300 times; the number of writes of bitmap unit 2 is 80 times, and the number of reads is 450 times, and so on.

[0021] The distribution density of the data storage location in the bitmap unit is calculated. The distribution density represents the distribution of data in different storage locations in the bitmap unit. The calculation method is to count the frequency of data in each storage location and divide by the total number of storage locations. Taking a bitmap unit containing 256 storage locations as an example, suppose the data frequency in positions 1-50 is 0.4, the data frequency in positions 51-150 is 0.3, and the data frequency in positions 151-256 is 0.2. Through statistical data, the data characteristic values of the bitmap index container can be obtained, including the write frequency, the read frequency and the data distribution density.

[0022] The obtained data characteristic values are divided into multiple data segments according to a preset rule. The division of data segments can be based on the distribution range of data characteristic values. For example, the write frequency is divided into a low frequency segment (0-50 times / 30 minutes), a medium frequency segment (51-200 times / 30 minutes) and a high frequency segment (more than 201 times / 30 minutes); the read frequency is divided into a low frequency segment (0-100 times / 30 minutes), a medium frequency segment (101-500 times / 30 minutes) and a high frequency segment (more than 501 times / 30 minutes); the data distribution density is divided into a low density segment (0-0.3), a medium density segment (0.31-0.7) and a high density segment (0.71-1.0).

[0023] The distribution characteristics of each data segment are calculated. The distribution characteristics include the proportion of the number of bitmap units in the data segment to the total number of bitmap units. For example, in the low frequency segment (0-50 times / 30 minutes) of the write frequency, there are 400 bitmap units, and the distribution characteristics of the data segment are 400 / 1000=0.4; in the medium frequency segment (51-200 times / 30 minutes) of the write frequency, there are 500 bitmap units, and the distribution characteristics of the data segment are 500 / 1000=0.5; in the high frequency segment (more than 201 times / 30 minutes) of the write frequency, there are 100 bitmap units, and the distribution characteristics of the data segment are 100 / 1000=0.1.

[0024] The distribution characteristics of each data segment are combined to form a probability distribution sequence. Taking the write frequency as an example, the probability distribution sequence is [0.4, 0.5, 0.1], indicating the distribution characteristics of the low frequency segment, the medium frequency segment and the high frequency segment. Similarly, the probability distribution sequence of the read frequency can be [0.3, 0.6, 0.1], and the probability distribution sequence of the data distribution density can be [0.2, 0.5, 0.3].

[0025] The skewness coefficient value corresponding to the bitmap index container is calculated based on the probability distribution sequence. The skewness coefficient is used to measure the symmetry of the data distribution. The calculation method is to divide the sum of the cubes of the differences between the elements in the probability distribution sequence and the median by the cube of the standard deviation, and then multiply by the proportion factor of the number of data segments. For the probability distribution sequence [0.4, 0.5, 0.1] of the write frequency, the median is 0.4, the standard deviation is about 0.2, and the number of data segments is 3, so the skewness coefficient is ((0.4-0.4)^3+(0.5-0.4)^3+(0.1-0.4)^3) / (0.2^3)*(3 / (3-1))=-0.5. Negative skewness indicates that the data distribution is biased to the high value side.

[0026] For example, assume that there is a bitmap index container containing 10000 bitmap units. In a 30-minute time window, statistics show that the write frequency of the bitmap units is distributed as follows: 0-50 times for 4000 units, 51-200 times for 5000 units, and more than 201 times for 1000 units; the read frequency is distributed as follows: 0-100 times for 2000 units, 101-500 times for 7000 units, and more than 501 times for 1000 units; the data distribution density is distributed as follows: 0-0.3 for 3000 units, 0.31-0.7 for 5000 units, and 0.71-1.0 for 2000 units.

[0027] According to the calculation, the probability distribution sequence of the write frequency is [0.4, 0.5, 0.1], the probability distribution sequence of the read frequency is [0.2, 0.7, 0.1], and the probability distribution sequence of the data distribution density is [0.3, 0.5, 0.2].

[0028] The skewness coefficient of the write frequency is -0.5, the skewness coefficient of the read frequency is -0.3, and the skewness coefficient of the data distribution density is -0.15. These skewness coefficient values indicate that the write frequency, the read frequency, and the data distribution density of the bitmap unit in the bitmap index container all present a slight negative skew distribution, meaning that the data is slightly concentrated in the high value area.

[0029] In this embodiment, by statistically analyzing the read and write times of the bitmap unit in real time and combining the distribution density analysis of the data storage location, the data access mode and the hotspot distribution can be accurately grasped, and the data access imbalance phenomenon can be found in time, thereby providing a basis for resource scheduling optimization. The analysis method combining data segmentation and probability distribution sequence can not only finely depict the local characteristics of data distribution, but also intuitively reflect the overall data tilt degree through the skewness coefficient, thereby providing a reliable decision basis for system storage optimization and query optimization. Through preset rules for data segmentation, the analysis process is standardized, and the engineering implementation is facilitated.

[0030] In an alternative embodiment, The data distribution feature vector is obtained by processing the data feature values through the pre-set time sequence diagram structure, the data access rule vector and the corresponding data change trend vector are determined based on the data feature values, and the feature fusion vector is obtained by combining the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions, including: The time sequence diagram structure is constructed based on the pre-set time interval, and a plurality of time nodes are recorded, the bitmap unit state parameters of each time node are calculated, the state transition matrix is constructed according to the bitmap unit state parameters, and the feature decomposition is performed to obtain the data distribution feature vector, wherein the bitmap unit state parameters include the occupation state, the modification time and the index information; The data blocks in the bitmap unit are obtained from the time sequence diagram structure, the standard access times of the data blocks in a preset time period and the continuous access times between the data blocks are recorded, the ratio of the continuous access times to the standard access times is constructed as an access mode matrix, the normalized Laplacian matrix of the access mode matrix is calculated, and the feature vector is extracted as a data access rule vector; According to a preset time length, a time window sequence is divided, a data operation frequency in each time window is recorded, a ratio of the data operation frequency to a total operation frequency in the time window is calculated to obtain an operation proportion parameter, the operation proportion parameter is multiplied by a corresponding preset weight coefficient and summed to obtain a change trend index, and time series analysis is performed on the change trend index to obtain a data change trend vector, wherein the data operation frequency includes a data write frequency, a data update frequency and a data deletion frequency. The data distribution feature vector, the data access rule vector and the data change trend vector are subjected to dimension alignment and normalization processing, and a feature fusion vector is obtained by weighted summation of the processing results.

[0031] A time series graph structure is constructed based on a pre-set time interval for data feature values. The time interval can be set to 5 minutes, and data feature values of the bitmap index container are collected within the time interval, and multiple time nodes are recorded. For example, within a monitoring period of one hour, data feature values of 12 time nodes can be obtained. At each time node, state parameters of the bitmap unit are calculated, including an occupancy state, a modification time and index information. The occupancy state indicates whether the bitmap unit is occupied by data, which can be represented by 0 for idle and 1 for occupied; the modification time records the latest modification timestamp of the bitmap unit; and the index information includes a pointer or reference to the actual data. For an index container containing 1000 bitmap units, at T1 time, there can be 700 units in the occupied state, the average modification time is 10 minutes ago, and the index information points to different tables and records in the database.

[0032] A state transition matrix is constructed according to the bitmap unit state parameters. The state transition matrix reflects the changes of the bitmap unit state between adjacent time nodes. The rows and columns of the matrix represent different states, and the matrix elements represent the probability of transition from one state to another. It is assumed that the occupancy state and the modification time are combined to define four states: newly created (occupied and modification time less than 5 minutes), active (occupied and modification time within 5-30 minutes), static (occupied and modification time greater than 30 minutes) and idle (not occupied). In the constructed 4x4 state transition matrix, each element represents the probability of transition from one state to another. For example, the value of matrix element (1, 2) is 0.3, indicating that the probability of transition from the "newly created" state to the "active" state is 0.3.

[0033] The state transition matrix is decomposed to obtain a data distribution eigenvector. The decomposition can reveal the main mode of state transition. The 4x4 state transition matrix described above is decomposed to obtain 4 eigenvalues and their corresponding eigenvectors. The eigenvector corresponding to the largest eigenvalue is selected as the data distribution eigenvector, for example, [0.2, 0.5, 0.2, 0.1], indicating the long-term distribution proportion of bitmap units in the four states of new creation, active, static, and idle.

[0034] The data blocks in the bitmap unit are obtained from the timing diagram structure, and the access of the data blocks in a preset time period is recorded. The preset time period can be set to 30 minutes, during which the standard access number of each data block and the continuous access number between data blocks are recorded. The standard access number refers to the total number of times a single data block is accessed; the continuous access number refers to the number of times a data block is accessed immediately after another data block is accessed. For example, data block A is accessed 100 times in 30 minutes, data block B is accessed 80 times, and the continuous access from A to B occurs 40 times, and the continuous access from B to A occurs 30 times.

[0035] The ratio of the continuous access number to the standard access number is constructed into an access pattern matrix. For the foregoing example, the ratio from A to B is 40 / 100=0.4, and the ratio from B to A is 30 / 80=0.375. The ratio forms the elements of the access pattern matrix, reflecting the access association strength between data blocks.

[0036] The normalized Laplacian matrix of the access pattern matrix is calculated and the eigenvector is extracted. The normalized Laplacian matrix can reveal the relative association structure between data blocks. For n data blocks, an n x n normalized Laplacian matrix is constructed, which is calculated by calculating the degree matrix and the adjacency matrix according to the definition of the normalized Laplacian matrix. The calculated normalized Laplacian matrix is decomposed, and the eigenvectors corresponding to the k smallest non-zero eigenvalues are selected to form a data access rule vector. For example, for 5 data blocks, 2 eigenvectors are selected to obtain a 5x2 matrix as the data access rule vector.

[0037] According to a preset time length, a time window sequence is divided, and the number of data operations in each time window is recorded. The time length can be set to 10 minutes, and multiple time windows are divided in the monitoring period. In each time window, the number of data write, data update, and data deletion is recorded. For example, in the first 10-minute window, the number of write is recorded as 200 times, the number of update is recorded as 300 times, the number of deletion is recorded as 50 times, and the total number of operations is recorded as 550 times.

[0038] The operation proportion parameter is calculated by dividing the number of data operations by the total number of operations in the time window. For the preceding example, the write operation proportion is 200 / 550 = 0.364, the update operation proportion is 300 / 550 = 0.545, and the delete operation proportion is 50 / 550 = 0.091.

[0039] The change trend indicator is obtained by multiplying the operation proportion parameter by the corresponding preset weight coefficient and summing them up. The preset weight coefficient can be set according to the importance of the operation type, for example, the weight of the write operation is 0.5, the weight of the update operation is 0.3, and the weight of the delete operation is 0.2. For the above example, the change trend indicator is 0.364 x 0.5 + 0.545 x 0.3 + 0.091 x 0.2 = 0.364.

[0040] The data change trend vector is obtained by performing time series analysis on the change trend indicator. Time series analysis can use the sliding window method to calculate the difference and change rate of the change trend indicator between adjacent time windows. For example, if the change trend indicators of the last 6 time windows are 0.364, 0.382, 0.395, 0.410, 0.425, and 0.435, respectively, the 5 differences [0.018, 0.013, 0.015, 0.015, 0.01] and change rates [4.9%, 3.4%, 3.8%, 3.7%, 2.4%] can be calculated. The difference and change rate are combined to form the data change trend vector.

[0041] The data distribution feature vector, data access rule vector, and data change trend vector are subjected to dimension alignment and normalization processing. Dimension alignment can be achieved by principal component analysis or dimension reduction techniques, so that the three vectors have the same dimension. Normalization processing can use the maximum and minimum value normalization method to map the elements of each vector to the [0, 1] interval. For example, the data distribution feature vector [0.2, 0.5, 0.2, 0.1] is still [0.2, 0.5, 0.2, 0.1] after normalization, a column in the data access rule vector [0.1, 0.3, 0.5, 0.2] is normalized to [0.2, 0.6, 1.0, 0.4], and the data change trend vector [0.018, 0.013, 0.015, 0.015] is normalized to [1.0, 0.722, 0.833, 0.833].

[0042] The feature fusion vector is obtained by weighted sum of the processing results. The weights of the three vectors can be set as 0.4, 0.4, and 0.2, respectively. The elements at corresponding positions are multiplied and summed to obtain the feature fusion vector.

[0043] In this embodiment, through the time sequence analysis and matrix decomposition of the bitmap unit state parameter, the dynamic change characteristics of the data distribution can be captured, the state transition matrix is constructed and feature decomposition is performed, which can not only reflect the instantaneous state of the data distribution, but also reveal the evolution law of the data distribution, providing a more comprehensive basis for storage optimization. Combined with the feature vector extraction of the standardized Laplacian matrix, the internal law of data access can be deeply mined, effectively guiding the optimization of cache strategy and the improvement of data pre-reading mechanism. The time window sequence and operation proportion parameter are combined to realize the fine quantization of data operation behavior, which has important guiding significance for resource scheduling and load balancing. Through dimension alignment and normalization processing, the feature vectors of multiple dimensions are fused into a unified feature representation, which not only retains the important information of each dimension, but also realizes the effective integration of features, which can significantly improve the overall performance and stability of the system.

[0044] In an optional implementation manner, Based on the feature fusion vector, the decision parameters are iteratively optimized combined with the double strategy gradient algorithm to obtain the adaptive threshold of container switching. The historical data is acquired to construct a historical data sequence, a time decay coefficient sequence is constructed as a time sequence decay weight based on the length of the historical data sequence, the gradient of each feature in the feature fusion vector to a pre-set initial strategy function is calculated, the two-norm of the gradient is multiplied by the time sequence decay weight and accumulated to obtain an importance score value of each feature. Based on the importance score value, a set of attention transformation matrices is constructed, the feature fusion vector is multiplied by the set of attention transformation matrices to obtain a query vector, a key value vector and a numerical value vector, the dot product of the query vector and the key value vector is calculated, and after softmax normalization, the numerical value vector is multiplied to obtain a reconstructed feature representation, one-dimensional convolution operation and ReLU function activation are performed on the reconstructed feature representation, the compressed feature representation is obtained through full connection layer mapping, the feature selection mask is determined through the sigmoid function and the element-wise multiplication operation is performed on the compressed feature representation to obtain the selected feature representation. Based on the Gaussian distribution, a random noise vector is constructed, the selected feature representation and the random noise vector are connected to form a feature supplement input, and a feature supplement value is generated combined with a nonlinear transformation, the feature supplement value and the selected feature representation are added to obtain a supplemented feature representation. Based on the initial strategy function, the strategy gradient corresponding to the supplemented feature representation is calculated, the strategy gradient is multiplied by the pre-set adaptive learning rate to update the parameters in the initial strategy function, and the adaptive threshold of container switching is calculated based on the updated strategy parameters.

[0045] The historical data is obtained to construct a historical data sequence, and the historical data includes operation records, performance indicators and other information of the bitmap index container in the past period of time. For example, the data of the last 30 days is collected, and 24 samples are taken per day to form a historical data sequence with a length of 720. A time decay coefficient sequence is constructed based on the length of the historical data sequence as a time decay weight. The time decay coefficient reflects the timeliness of the historical data, and the more recent the data, the higher the weight. For a historical data sequence with a length of 720, an exponential decay method can be used to construct the time decay weight, and the weight of the most recent data point is 1.0, and the weight of the earliest data point is about 0.01.

[0046] The gradient of each feature in the feature fusion vector to the pre-set initial strategy function is calculated, and the initial strategy function can be a multi-layer perception network, the input of which is the feature fusion vector, and the output is the operation decision of the bitmap index container. Assuming that the dimension of the feature fusion vector is 10, the initial strategy function includes an input layer (10 nodes), a hidden layer (20 nodes) and an output layer (5 nodes). The gradient of each feature to the output of the strategy function is calculated by the back propagation algorithm. For example, for the feature fusion vector [0.3, 0.7, 0.2, 0.5, 0.8, 0.4, 0.6, 0.1, 0.9, 0.5], the calculated gradient vector can be [0.05, 0.12, 0.03, 0.08, 0.15, 0.07, 0.09, 0.02, 0.18, 0.08].

[0047] The two-norm of the gradient is multiplied by the time decay weight and accumulated to obtain the importance score value of each feature. The two-norm represents the length of the gradient vector, and the calculation method is the square root of the sum of squares of the elements of the gradient. For the above gradient vector, the two-norm is about 0.31. Multiply the two-norm value by the time decay weight at the corresponding time (such as 0.9) to get 0.28. Repeat this calculation for all times in the historical data sequence and accumulate to obtain the importance score value of each feature. For example, the importance score values of 10 features can be [12.5, 28.3, 8.7, 19.2, 35.6, 16.8, 21.4, 5.3, 41.2, 18.9].

[0048] An attention transformation matrix group is constructed based on the importance score value. The attention transformation matrix group includes three matrices: a query transformation matrix, a key-value transformation matrix and a numerical transformation matrix, and the elements of the matrix are proportionally distributed according to the feature importance score value. For example, for a feature fusion vector with a dimension of 10, three 10x8 transformation matrices are constructed, and the features with high importance score values have higher weights in the matrix.

[0049] The feature fusion vector is multiplied by the attention transformation matrix group to obtain the query vector, the key value vector and the numerical value vector. Assuming that the feature fusion vector is [0.3, 0.7, 0.2, 0.5, 0.8, 0.4, 0.6, 0.1, 0.9, 0.5], by multiplying with the corresponding transformation matrix, three 8-dimensional vectors are obtained: the query vector [0.62, 0.54, 0.48, 0.71, 0.39, 0.56, 0.67, 0.43], the key value vector [0.58, 0.49, 0.52, 0.65, 0.44, 0.51, 0.62, 0.47], and the numerical value vector [0.60, 0.52, 0.50, 0.68, 0.41, 0.54, 0.64, 0.45].

[0050] The dot product of the query vector and the key value vector is calculated, and after softmax normalization, the numerical value vector is multiplied to obtain the reconstructed feature representation. The dot product operation calculates the product of the corresponding elements of the query vector and the key value vector and sums them up to obtain a scalar value. Repeat the operation for the key value vectors at different positions to obtain a set of attention scores. Normalize these scores into a probability distribution through the softmax function, multiply the corresponding elements of the numerical value vector, and sum them up to obtain the reconstructed feature representation. The reconstructed feature representation is [0.59, 0.53, 0.49, 0.69, 0.42, 0.53, 0.65, 0.44].

[0051] Perform one-dimensional convolution operation and ReLU function activation on the reconstructed feature representation. The one-dimensional convolution operation uses 3 convolution kernels, each with a width of 3 and a step of 1, to obtain a 6-dimensional output. The ReLU function sets negative values to 0 and keeps positive values unchanged. For example, the feature representation after convolution and ReLU activation is [1.23, 0.98, 0.87, 1.45, 0.76, 1.02].

[0052] Map through the fully connected layer to obtain the compressed feature representation. The fully connected layer maps the 6-dimensional input to a 4-dimensional output, and the parameter matrix size is 6x4. After mapping through the fully connected layer, the compressed feature representation is obtained, for example, [0.85, 1.32, 0.74, 1.17].

[0053] Determine the feature selection mask through the sigmoid function and perform element-wise multiplication operation with the compressed feature representation to obtain the selected feature representation. The sigmoid function maps the value to 0 to 1 as the feature selection mask. For the compressed feature representation, the feature selection mask calculated through the additional weight matrix may be [0.92, 0.78, 0.65, 0.88]. Element-wise multiplication of the mask and the compressed feature representation obtains the selected feature representation [0.78, 1.03, 0.48, 1.03].

[0054] The selected feature representation is connected with a random noise vector to form a feature augmented input. The elements of the random noise vector are sampled from a Gaussian distribution with a mean of 0 and a standard deviation of 0.1, and the dimension is the same as the selected feature representation, for example, [0.05, -0.08, 0.12, -0.03]. The selected feature representation is connected with the random noise vector to form an 8-dimensional feature augmented input [0.78, 1.03, 0.48, 1.03, 0.05, -0.08, 0.12, -0.03].

[0055] The feature augmentation value is generated by combining a nonlinear transformation. The nonlinear transformation uses a two-layer neural network, the input is an 8-dimensional feature augmented input, the hidden layer has 6 nodes, and the output layer has 4 nodes corresponding to the feature augmentation value. After nonlinear transformation, the feature augmentation value [0.15, 0.22, -0.08, 0.17] is obtained.

[0056] The feature augmentation value is added to the selected feature representation to obtain the augmented feature representation. Element-wise addition is performed on the two vectors to obtain the augmented feature representation [0.93, 1.25, 0.40, 1.20].

[0057] The policy gradient corresponding to the augmented feature representation is calculated based on the initial policy function. The policy gradient represents the partial derivative of the policy function output with respect to the parameters. The augmented feature representation is used as input to calculate the policy function output through forward propagation, and then the parameter gradient is calculated through back propagation. For example, the calculated gradient for a certain parameter may be 0.12.

[0058] The policy gradient is multiplied by the pre-set adaptive learning rate to update the parameters in the initial policy function. The adaptive learning rate can be calculated using the Adam optimizer, with an initial value of 0.01. The gradient is multiplied by the learning rate to obtain the parameter update amount, for example, 0.12 x 0.01 = 0.0012. The parameter update amount is added to the original parameter to complete the parameter update.

[0059] The adaptive threshold for container switching is calculated based on the updated policy parameters. The container switching threshold determines the conditions under which the bitmap index container is switched or reconstructed. The updated policy function is used to input the current feature representation and output a threshold value, for example, 0.75. When the performance indicator of the bitmap index container is lower than the threshold, the container switching operation is triggered.

[0060] In this embodiment, by introducing a time decay coefficient sequence to weight the historical data, and combining the importance score calculated by the two-norm of the feature gradient, the key features that truly have an impact on strategy decision can be accurately identified, not only considering the immediate importance of the features, but also reflecting the importance of the features in the historical evolution process, improving the accuracy of feature selection. Through the interactive operation of the query vector, key-value vector and numerical vector, the deep correlation between features is captured. A random noise vector based on Gaussian distribution is introduced to supplement the features, and a feature supplement value is generated through nonlinear transformation, enhancing the robustness and generalization ability of feature representation. Not only can it make up for the information loss caused by feature selection, but also can improve the ability of the model to deal with uncertainty. Through the strategy gradient method combined with adaptive learning rate to update the strategy parameters, the dynamic adjustment of the container switching threshold is realized, improving the adaptability and decision accuracy of the system.

[0061] In an optional embodiment, When container type switching is needed, a temporary container is created to perform deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, the heat value and the correlation value of the deduplicated data are calculated, the weighted sum of the heat value and the correlation value is taken as the priority value, and the deduplicated data is sorted to obtain a to-be-migrated data sequence including: When container type switching is needed, a temporary container is created and the to-be-migrated data in the bitmap index container is input into the temporary container, the feature fingerprint of each piece of data in the to-be-migrated data is calculated through the local sensitive hash algorithm, the data similarity between any two pieces of to-be-migrated data in the to-be-migrated data is calculated based on the feature fingerprint, and the data similarity is compared with a preset similarity threshold. When the data similarity is greater than the preset similarity threshold, it is determined as duplicate data and removed to obtain deduplicated data; The access frequency of the deduplicated data in a second sliding time window is counted to obtain an access frequency, the time interval from the last access to the current time is calculated to obtain a time weight, and the time interval from the data generation time to the current time is calculated to obtain a survival time. The heat value is calculated based on the access frequency, the time weight and the survival time; The association weight between each two pieces of data in the deduplicated data is calculated to obtain an association weight, and the semantic similarity between each two pieces of data is calculated. The association weight and the semantic similarity are multiplied and averaged to obtain an association value; According to the task requirements, the weight coefficients corresponding to the heat value and the correlation value are respectively initialized and weighted summed to obtain a priority value, and the deduplicated data is sorted in descending order based on the priority value to obtain a to-be-migrated data sequence.

[0062] When the performance of the bitmap index container falls below a preset threshold or it is determined based on an adaptive strategy that container type switching is needed, a temporary container is created and the data to be migrated in the bitmap index container is input into the temporary container. The temporary container can be a storage structure of the same structure as the target container type but smaller in scale, used to temporarily store the data to be migrated. For example, when the original bitmap index container is a bitmap structure implemented based on an array and the target container type is a bitmap index container implemented based on a tree structure, a temporary container of the tree structure needs to be created. Assuming that there are 10,000 pieces of data in the original bitmap index container, according to the storage space and performance requirements, it can be necessary to migrate all or part of the data to the new container type.

[0063] The feature fingerprint of each piece of data to be migrated is calculated by using a local sensitive hashing algorithm. In this embodiment, the MinHash algorithm is used as an implementation manner of the local sensitive hashing. For each piece of data to be migrated, its key field is extracted as a feature, for example, the identifier, type, content digest, and the like of the data, and then the MinHash algorithm is applied to generate a feature fingerprint. The feature fingerprint is usually a fixed-length integer array, for example, an integer array of length 128. For a data record {ID: 1001, Type: "transaction", Content: "payment processing", Timestamp: 1629350400}, the feature fingerprint calculated by the MinHash algorithm can be [45, 78, 32, 91,..., 64].

[0064] The data similarity between any two pieces of data to be migrated is calculated based on the feature fingerprints. The data similarity can be measured by calculating the Jaccard similarity coefficient of the two feature fingerprints. The Jaccard similarity coefficient is the size of the intersection of two sets divided by the size of the union. In the case of using the MinHash algorithm, the Jaccard similarity coefficient can be approximated by calculating the proportion of values equal at the same position in the two feature fingerprints. For example, for the feature fingerprints [45, 78, 32, 91,..., 64] and [45, 79, 32, 90,..., 64] of two pieces of data, if the values are equal at 112 positions out of 128 positions, the similarity is 112 / 128 = 0.875.

[0065] The data similarity is compared with a preset similarity threshold, and when the data similarity is greater than the preset similarity threshold, it is determined that the data is duplicate data and is removed, to obtain deduplicated data. The preset similarity threshold can be set according to business requirements, for example, set to 0.85. When the similarity of two pieces of data exceeds 0.85, they are considered to be duplicate data and only one of them is retained. Assuming that there are 10,000 pieces of data to be migrated, after deduplication processing, there are 8,500 pieces of deduplicated data left.

[0066] The access frequency is obtained by counting the number of accesses of the deduplicated data within a second sliding time window. The second sliding time window can be set to the last 24 hours, and the number of accesses of each deduplicated data within this time window is counted. The access frequency represents the number of accesses of the data per unit time, which can be calculated by dividing the number of accesses by the length of the time window. For example, if a piece of data is accessed 120 times within 24 hours, its access frequency is 120 / 24=5 times / hour.

[0067] The time weight is obtained by calculating the time interval from the last access to the current time. The time weight reflects the timeliness of the data, and is calculated using a decreasing function. The shorter the time interval, the greater the time weight. For example, if the last access was 2 hours ago, the time weight can be calculated as 1 / (2+1)=0.333; if the last access was 0.5 hours ago, the time weight is 1 / (0.5+1)=0.667.

[0068] The survival time is obtained by calculating the time interval from the data generation time to the current time. The survival time represents the length of time the data has existed since its creation. For example, if the data was created 720 hours ago, its survival time is 720 hours.

[0069] The heat value is calculated based on the access frequency, time weight and survival time. The heat value can be calculated by multiplying the access frequency and the time weight, and then dividing by the logarithmic value of the survival time (plus 1 to prevent the logarithm from being 0). It takes into account the access frequency, recent access and survival time of the data, and can comprehensively reflect the heat of the data. For example, for data with an access frequency of 5 times / hour, a time weight of 0.667 and a survival time of 720 hours, the heat value can be calculated as 5x0.667 / log(720+1)=1.11.

[0070] The association weight is obtained by calculating the reference relationship between each two pieces of deduplicated data. The reference relationship can be determined by analyzing the explicit reference (such as foreign key relationship) or implicit reference (such as operations in the same session) between the data. The association weight can be set to the ratio of the number of references between the two data to the total number of references. For example, if the number of references of data A to data B accounts for 30% of all references of A, the association weight of A to B is 0.3.

[0071] The semantic similarity between each two pieces of data is calculated. The semantic similarity can be calculated by comparing the content, label or attribute of the data. Cosine similarity, Word2Vec or other semantic analysis methods can be used. For example, for two data content vectors [0.2, 0.5, 0.3, 0.8] and [0.3, 0.4, 0.2, 0.7], the calculated cosine similarity is 0.92.

[0072] The correlation weight is multiplied by the semantic similarity and averaged to obtain a correlation value. The correlation value comprehensively considers the reference relationship and semantic similarity between the data. For example, for two data with a correlation weight of 0.3 and a semantic similarity of 0.92, the correlation value is 0.3*0.92=0.276. For the correlation value of a data with all other data, the average value is calculated as the overall correlation value of the data.

[0073] The weight coefficients corresponding to the hotness value and the correlation value are respectively initialized according to the task requirements, and a weighted sum is performed to obtain a priority value. Different tasks have different emphasis on hotness and correlation, which is reflected by setting different weight coefficients. For example, for a task that emphasizes data hotness, the hotness weight can be set to 0.7 and the correlation weight to 0.3; for a task that emphasizes data correlation, the hotness weight can be set to 0.3 and the correlation weight to 0.7. The priority value is calculated by multiplying the hotness value by the hotness weight and adding the correlation value multiplied by the correlation weight. For example, for data with a hotness value of 1.11 and a correlation value of 0.276, if the hotness weight is 0.7 and the correlation weight is 0.3, the priority value is 1.11*0.7+0.276*0.3=0.86.

[0074] The deduplicated data is sorted in descending order based on the priority value to obtain a sequence of data to be migrated. The higher the priority value, the more important the data, which should be migrated first. All deduplicated data is sorted in descending order of priority value to form a sequence of data to be migrated. For example, the priority values of the top three data after sorting may be 2.56, 2.34 and 2.15 respectively. In the actual migration process, the data can be migrated in this sequence to ensure that important data is processed first and improve the overall performance of the system.

[0075] In this embodiment, by creating a temporary container when switching the container type, and using the local sensitive hashing algorithm to generate feature fingerprints of the data to be migrated, calculate the data similarity and remove duplicate data, thereby reducing the amount of redundant data and reducing the migration overhead. The correlation weight is obtained by calculating the reference relationship between the data, and the correlation value is calculated by combining the semantic similarity, which realizes the accurate evaluation of the structural relationship and semantic correlation between the data. The weight coefficients of the hotness value and the correlation value are respectively initialized according to the task requirements, and a weighted sum is performed to obtain a priority value, and the deduplicated data is sorted in descending order of priority to generate a sequence of data to be migrated, thereby realizing the effect of transmitting high-value data first while ensuring the integrity of the correlation between the data in the migration process, effectively reducing the amount of data migration, shortening the migration time and improving the overall performance and business continuity of the container switching process.

[0076] In an alternative embodiment, The data sequence to be migrated is written into a temporary container in batches and a write state value is recorded, a confirm commit is performed when the write state value indicates a pre-commit success, a bidirectional index table is generated and data access request redirection is performed using the bidirectional index table, which comprises: The data sequence to be migrated is grouped into multiple data batches according to a preset batch size, the multiple data batches are written into a temporary container, and a write state value is recorded for each data batch; The hash values of the data in each data batch are calculated and summed to obtain a data checksum, the data checksum after writing is compared with the data checksum before writing to obtain an integrity verification result, a pre-commit log record is generated based on the integrity verification result, and a confirm commit operation is performed when the write state value indicates a pre-commit success; The original data identifier of the data to be migrated is obtained from the bitmap index container, the new data identifier of the data to be migrated is obtained from the temporary container, a mapping relationship is established and a version number and a timestamp are added to generate metadata information, and a bidirectional index table is constructed based on the original data identifier, the new data identifier and the metadata information; The mapping relationship between the original data identifier and the new data identifier is stored in a redirection cache, a data access request is received, the target data location is preferentially searched from the redirection cache, if not found, the target data location is searched in the bidirectional index table, if the search fails, it is judged whether the number of retries exceeds a search threshold, if not, the retry is performed, if it exceeds and it is a read operation, the data in the bitmap index container is accessed, if it exceeds and it is a write operation, the data access request is rejected.

[0077] For the data sequence to be migrated that has been sorted, multiple data batches are obtained by grouping according to a preset batch size. The preset batch size is a parameter set according to system performance and fault tolerance requirements, usually 100-1000 data. For example, for 8500 deduplicated data, if the batch size is set to 500, 17 data batches can be obtained. The original priority order is maintained during grouping, i.e. the first batch contains the 500 data with the highest priority, the second batch contains the 500 data with the second highest priority, and so on.

[0078] The multiple data batches are written into a temporary container, and a write state value is recorded for each data batch in the multiple data batches. The write operation is to convert each data in the data batch into a format supported by the temporary container and store it according to the data organization method of the temporary container. The write state value is used to identify the write state of the data batch, including "initialization", "writing", "writing completed", "pre-commit success", "confirm commit completed" or "write failed", etc. For example, for the 500 data in the first batch, the state is set to "writing" at the beginning of writing, and the state is updated to "writing completed" after writing is completed.

[0079] The hash values of the data in each batch are calculated and summed to obtain a data checksum. The hash value calculation can use common hash algorithms such as SHA-256 or MD5. For each piece of data in the batch, its hash value is calculated, and then all the hash values are added (or exclusive-OR operation is performed) to obtain the data checksum. For example, for a batch containing 500 pieces of data, the data checksum calculated before writing can be 0x8F72A1B3C5D4E6F7.

[0080] The post-write data checksum is compared with the pre-write data checksum to obtain the integrity verification result. After the data is written to the temporary container, the data checksum is calculated again and compared with the pre-write checksum. If the two checksums are equal, it indicates that the data integrity verification is passed; otherwise, it indicates that data corruption occurred during the writing process. For example, if the post-write data checksum is also 0x8F72A1B3C5D4E6F7, the integrity verification result is "passed".

[0081] A pre-commit log record is generated based on the integrity verification result. The pre-commit log record contains information such as batch identification, writing time, data count, checksum, verification result, etc. For example, for the first batch, the pre-commit log record can be {BatchID: 1, Time: "2023-08-10 14:30:25", Count: 500, Checksum: "0x8F72A1B3C5D4E6F7", Validation: "passed"}. The confirmation submission operation is performed when the write state value indicates that the pre-commit is successful. The confirmation submission operation updates the state of the current batch from "pre-commit successful" to "confirmation submission completed", indicating that the batch data has been safely written to the temporary container.

[0082] The original data identifier of the data to be migrated is obtained from the bitmap index container. The original data identifier is a unique identifier of the data in the original bitmap index container, which can be an integer ID or a hash value. For example, the identifier of a piece of data to be migrated in the original bitmap index container is "BM_123456".

[0083] The new data identifier of the data to be migrated is obtained from the temporary container. The new data identifier is a unique identifier of the data in the temporary container (future new container). For example, the identifier of the same piece of data in the temporary container can be "TR_789012".

[0084] A mapping relationship is established and version number and timestamp are added to generate metadata information. The mapping relationship records the correspondence between the original data identifier and the new data identifier. The version number is used to identify the version of the data, and is initially set to 1. The timestamp records the time of creating the mapping relationship. The metadata information may be like {OriginalID: "BM_123456", NewID: "TR_789012", Version: 1, Timestamp: 1628590225000}.

[0085] A bidirectional index table is constructed based on the original data identifier, the new data identifier, and the metadata information. The bidirectional index table contains two directions of mapping: the mapping from the original data identifier to the new data identifier, and the mapping from the new data identifier to the original data identifier, and the bidirectional index structure supports fast lookup and reverse lookup. For example, a record in the bidirectional index table may be {Forward: {Key: "BM_123456", Value: "TR_789012"}, Backward: {Key: "TR_789012", Value: "BM_123456"}, Metadata: {Version: 1, Timestamp: 1628590225000}}.

[0086] The mapping relationship between the original data identifier and the new data identifier is stored in the redirection cache. The redirection cache is a cache structure used to speed up the data location lookup process. The cache adopts LRU (Least Recently Used) strategy management, and the capacity can be set to the 10000 most recently accessed mapping relationships. For example, an entry in the redirection cache may be {Key: "BM_123456", Value: "TR_789012", AccessTime: 1628590230000}.

[0087] A data access request is received, and the target data location is preferentially looked up from the redirection cache. The data access request contains the identifier of the target data and the operation type (read or write), and the lookup process checks the redirection cache to determine whether the requested data identifier is in the cache. For example, a request to read data "BM_123456" is received, and it is checked whether there is an entry with the key "BM_123456" in the redirection cache.

[0088] If not found, the target data location is looked up in the bidirectional index table. If there is no related mapping in the redirection cache, the bidirectional index table is continued to be looked up. For example, the entry with the key "BM_123456" is looked up in the bidirectional index table, and the corresponding new data identifier "TR_789012" is found.

[0089] If the search fails, it is determined whether the number of retries exceeds a search threshold. The search failure can be caused by the fact that the data has not been migrated or the mapping relationship has not been established. A retry mechanism is set to allow multiple attempts to search, and the retry interval can be set to 10 milliseconds, and the search threshold can be set to 3 times. For example, after the first search fails, the search is performed again after waiting for 10 milliseconds, and the search is performed until the data is found or the retry limit is reached.

[0090] If the threshold is not exceeded, the retry is performed, and if the threshold is exceeded and the operation type is a read operation, the data in the bitmap index container is accessed. If the number of retries does not exceed the threshold, the search continues in the redirection cache and the bidirectional index table. If the threshold is exceeded and the operation type is a read, the data is read from the original bitmap index container. For example, if three searches fail and the request is a read operation, the data corresponding to "BM_123456" is read directly from the original bitmap index container.

[0091] If the threshold is exceeded and the operation type is a write operation, the data access request is rejected. If the number of retries exceeds the threshold and the operation type is a write, the write request is rejected, and an error message "Data migration, write operation not supported temporarily" is returned.

[0092] In this embodiment, the hash value summation is introduced during the migration process to generate a data checksum and compare it with the checksum before writing, which realizes the integrity verification of the data batch, ensures the accuracy and reliability of the migrated data during the writing process, and through the pre-commit log mechanism, the submission is confirmed after the verification is passed, effectively reducing the risk of data loss or damage caused by writing errors. By obtaining the original data identifier and the new data identifier and establishing a mapping relationship, combining the version number and the timestamp to generate metadata information and construct a bidirectional index table, the data location can be quickly located after migration and the forward and reverse tracing is supported, ensuring the consistency and manageability of the data. The bidirectional index table is used as a secondary retrieval mechanism to improve the success rate of search, and the retry mechanism is used to improve the robustness of access, and after the search threshold is exceeded, the read is retraced or the write is rejected according to the operation type, thereby realizing efficient access control and fault handling during the migration process while ensuring data consistency and integrity.

[0093] Figure 2 The data migration logic flowchart of the bitmap index container optimization method based on multi-dimensional dynamic decision-making of the embodiment of the application.

[0094] In an optional implementation, The performance index value after switching is monitored, and when the performance index value exceeds a preset range, a rollback operation is performed and the decision parameter is updated, including: The performance index value after switching is obtained, and the performance index value includes service response time, resource occupancy rate and business success rate; determine whether the performance index value exceeds a preset performance threshold range, the preset performance threshold range being determined based on a statistical distribution of historical performance data, and trigger a fallback operation when a duration that the performance index value continuously exceeds the preset performance threshold range is greater than a preset time threshold; perform the fallback operation and update a decision parameter based on a performance fluctuation of the current switching, the decision parameter including a switching time window, a performance monitoring frequency, and a performance threshold interval.

[0095] After the bitmap index container switching is completed, the performance of the system after the switching is continuously monitored, and abnormalities are found in a timely manner and corresponding measures are taken. The performance index value after the switching is obtained, and the obtained performance index value includes service response time, resource occupancy rate, and service success rate. The service response time measures the time required from receiving a request to returning a result, and the unit is millisecond. The resource occupancy rate reflects the use of system resources (such as CPU, memory, disk I / O, etc.), and is expressed in percentage. The service success rate represents the proportion of the number of successfully processed service requests to the total number of requests, and is also expressed in percentage. The performance index can be collected in real time through a performance monitoring module embedded in the bitmap index container, or obtained through interfacing with an external monitoring system. For example, within the first hour after the switching, performance data is collected every minute, and the following index value sequences are obtained: service response time [15 ms, 16 ms, 14 ms, 17 ms,...], resource occupancy rate [45%, 47%, 46%, 48%,...], and service success rate [99.8%, 99.7%, 99.9%, 99.8%,...].

[0096] determine whether the performance index value exceeds a preset performance threshold range, the preset performance threshold range being determined based on a statistical distribution of historical performance data, and trigger a fallback operation when a duration that the performance index value continuously exceeds the preset performance threshold range is greater than a preset time threshold;

[0097] When the duration that the performance indicator value continuously exceeds the preset performance threshold range is greater than the preset time threshold, the fallback operation is triggered. The preset time threshold is a time period set according to the service tolerance and system stability requirements, usually 5-15 minutes. For example, if the preset time threshold is set to 5 minutes and the monitoring frequency is once per minute, when the performance indicator value of 5 consecutive monitoring exceeds the threshold range, the fallback operation is triggered.

[0098] The fallback operation is performed, which includes stopping writing new data to the new container, obtaining the mapping relationship of the migrated data from the bidirectional index table, rewriting the data to the original in-place index container according to the mapping relationship, updating the routing information, and redirecting the access request to the original in-place index container. In order to ensure the reliability of the rollback process, batch processing is also required, and checksum comparison is also required. For example, for the 8500 data that has been migrated, it is rewritten in batches of 500, and checksum comparison is performed after each batch of rewriting to ensure data consistency.

[0099] The decision parameters are updated based on the performance fluctuation of this switching, including the switching time window, the performance monitoring frequency and the performance threshold interval. The switching time window refers to the best time period for performing container switching, usually selected in the time period when the system load is low. For example, if this switching is performed in a period of high system load and causes performance problems, the switching time window needs to be adjusted to a period of lower load, such as from afternoon to night or weekend. The performance monitoring frequency refers to the time interval for collecting performance indicators, which needs to be adjusted according to the response speed and fluctuation characteristics of the system. For example, if the performance fluctuates greatly after this switching, the monitoring frequency needs to be increased from once per minute to once every 30 seconds, so as to discover problems more timely. The performance threshold interval is the standard for judging whether the system performance is abnormal, which needs to be dynamically adjusted according to the actual situation. For example, if the fluctuation range of service response time after this switching is larger than expected, the threshold interval of response time may need to be adjusted from [6ms, 18ms] to [5ms, 20ms] to reduce false positives.

[0100] The decision parameters are updated based on statistical characteristics of performance fluctuations, for the switching time window, the system load in different time periods is analyzed, and the time period with the lowest load is selected as the new switching time window. For example, by analyzing the system load every hour in a week, it is found that the load from 2am to 4am is the lowest, and this time period is set as the new switching time window. For the performance monitoring frequency, it can be adjusted according to the change rate of the performance indicators. For example, if the change rate of the performance indicators is high, it means that the system state changes rapidly, and the monitoring frequency needs to be increased; on the contrary, if the change rate is low, the monitoring frequency can be appropriately reduced to reduce system overhead. For the performance threshold interval, it can be updated based on the statistical distribution of historical data. For example, if the mean of the performance indicators in the recent period is 14ms and the standard deviation is 3ms, the threshold interval of the response time can be updated to [5ms, 23ms] (mean plus or minus 3 times the standard deviation).

[0101] In this embodiment, by real-time acquisition of performance indicators such as service response time, resource occupancy rate and business success rate after container type switching, and comparison with the preset performance threshold range determined based on the statistical distribution of historical performance data, when it is detected that the performance indicator value continuously exceeds the threshold range and the duration exceeds the preset time threshold, the rollback operation is automatically triggered, thereby quickly recovering to a stable state when the performance is abnormal, reducing the impact of switching on business continuity and user experience, and dynamically updating the decision parameters according to the performance fluctuation of this switching after rollback, which can adaptively optimize the subsequent switching strategy, continuously improve the accuracy and stability of the switching decision, and realize active prevention and continuous optimization of performance risks.

[0102] In a second aspect of the embodiment of the application, a bitmap index container optimization system based on multi-dimensional dynamic decision is provided, comprising: A first unit is configured to obtain data characteristic values of the bitmap index container and calculate skewness coefficient values based on the data characteristic values; A second unit is configured to process the data characteristic values through a pre-set timing diagram structure to obtain a data distribution feature vector, determine a data access rule vector and a corresponding data change trend vector based on the data characteristic values, combine the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimize the decision parameters based on the feature fusion vector and in combination with a double strategy gradient algorithm to obtain an adaptive threshold for container switching; A third unit is configured to compare the skewness coefficient values with the adaptive threshold and determine whether to perform container type switching; The fourth unit is configured to, when a container type switch is needed, create a temporary container, perform deduplication operation on the to-be-migrated data in the temporary container and the bitmap-indexed container to obtain deduplicated data, calculate a heat value and a correlation value of the deduplicated data, take a weighted sum of the heat value and the correlation value as a priority value, sort the deduplicated data according to the priority value to obtain a to-be-migrated data sequence, write the to-be-migrated data sequence into the temporary container in batches and record a write state value, perform a confirm commit when the write state value indicates that a pre-commit is successful, generate a bidirectional index table, and redirect a data access request by using the bidirectional index table. The fifth unit is configured to monitor a performance index value after the switch, perform a rollback operation and update a decision parameter when the performance index value exceeds a preset range.

[0103] In a third aspect, an electronic device is provided, including: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0104] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0105] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0106] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. Bitmap index container optimization method based on multi-dimensional dynamic decision, characterized in that, The method comprises the following steps: obtaining a data feature value of a bitmap index container and calculating a skewness coefficient value based on the data feature value; processing the data feature value through a pre-set timing diagram structure to obtain a data distribution feature vector, determining a data access rule vector and a corresponding data change trend vector based on the data feature value, combining the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimizing a decision parameter based on the feature fusion vector combined with a double strategy gradient algorithm to obtain an adaptive threshold value for container switching; comparing the skewness coefficient value with the adaptive threshold value and determining whether to switch the container type; when it is necessary to switch the container type, creating a temporary container to perform a deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, calculating a heat value and a correlation value of the deduplicated data, taking a weighted sum of the heat value and the correlation value as a priority value and sorting the deduplicated data to obtain a to-be-migrated data sequence, writing the to-be-migrated data sequence into the temporary container in batches and recording a write state value, performing a confirmation commit when the write state value indicates that a pre-commit is successful, generating a bidirectional index table and redirecting a data access request by using the bidirectional index table; monitoring a performance index value after switching, and performing a rollback operation and updating a decision parameter when the performance index value exceeds a preset range.

2. The method of claim 1, wherein, The method comprises the following steps: counting the number of writes and reads of a bitmap unit in a bitmap index container within a preset first time window, calculating the distribution density of a data storage location in the bitmap unit, and obtaining a data feature value; dividing the data feature value into a plurality of data segments according to a preset rule, calculating the distribution characteristics of each data segment respectively, and combining to obtain a probability distribution sequence; calculating the skewness coefficient value corresponding to the bitmap index container based on the probability distribution sequence.

3. The method of claim 1, wherein, The method comprises the following steps: constructing a timing diagram structure based on a pre-set time interval and recording a plurality of time nodes, calculating bitmap unit state parameters of each time node, constructing a state transition matrix based on the bitmap unit state parameters and performing feature decomposition to obtain a data distribution feature vector, wherein the bitmap unit state parameters include an occupancy state, a modification time and index information; obtaining data blocks in the bitmap unit from the timing diagram structure, recording the standard access frequency of the data blocks within a preset time period and the continuous access frequency between the data blocks, constructing the ratio of the continuous access frequency to the standard access frequency as an access pattern matrix, calculating a standardized Laplacian matrix of the access pattern matrix and extracting a feature vector as a data access rule vector; According to a preset time length, a time window sequence is divided, the number of data operations in each time window is recorded, the operation ratio parameter is calculated by calculating the ratio of the number of data operations to the total number of operations in the time window, the operation ratio parameter is multiplied by the corresponding preset weight coefficient and summed to obtain the change trend index, and the change trend index is analyzed in time sequence to obtain a data change trend vector, wherein the number of data operations includes the number of data writes, the number of data updates and the number of data deletions; The data distribution feature vector, the data access rule vector and the data change trend vector are dimensionally aligned and normalized, and the weighted sum of the processing results is obtained to obtain a feature fusion vector.

4. The method of claim 1, wherein, Based on the feature fusion vector, the decision parameter is iteratively optimized by combining a double strategy gradient algorithm to obtain an adaptive threshold for container switching, including: The historical data is obtained to construct a historical data sequence, a time decay coefficient sequence is constructed based on the length of the historical data sequence as a time sequence decay weight, the gradient of each feature in the feature fusion vector to the pre-set initial strategy function is calculated, the two-norm of the gradient is multiplied by the time sequence decay weight and accumulated to obtain the importance score value of each feature; Based on the importance score value, a set of attention transformation matrices is constructed, the feature fusion vector is multiplied by the set of attention transformation matrices to obtain a query vector, a key value vector and a numerical value vector, the dot product of the query vector and the key value vector is calculated, and after softmax normalization, the numerical value vector is multiplied to obtain a reconstructed feature representation, one-dimensional convolution operation and ReLU function activation are performed on the reconstructed feature representation, and through a fully connected layer mapping, a compressed feature representation is obtained., the feature selection mask is determined by the sigmoid function and the compressed feature representation is executed element by element multiplication operation to obtain the selected feature representation; Based on the Gaussian distribution, a random noise vector is constructed, the selected feature representation and the random noise vector are connected to form a feature supplement input, and a feature supplement value is generated by combining a nonlinear transformation, the feature supplement value and the selected feature representation are added to obtain a supplemented feature representation; Based on the initial strategy function, the strategy gradient corresponding to the supplemented feature representation is calculated, the strategy gradient is multiplied by the pre-set adaptive learning rate to update the parameters in the initial strategy function, and the adaptive threshold for container switching is calculated based on the updated strategy parameters.

5. The method of claim 1, wherein, When container type switching is needed, a temporary container is created to perform deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data, the hotness value and the correlation value of the deduplicated data are calculated, the weighted sum of the hotness value and the correlation value is taken as the priority value, and the deduplicated data is sorted to obtain a to-be-migrated data sequence, including: When container type switching is needed, a temporary container is created and the to-be-migrated data in the bitmap index container is input into the temporary container, a feature fingerprint of each piece of data in the to-be-migrated data is calculated by a local sensitive hash algorithm, a data similarity between any two pieces of to-be-migrated data in the to-be-migrated data is calculated based on the feature fingerprint, the data similarity is compared with a preset similarity threshold, when the data similarity is greater than the preset similarity threshold, it is determined that the data is duplicate data and is removed, and de-duplicated data is obtained; The access frequency of the de-duplicated data in a second sliding time window is counted, a time weight is calculated by calculating a time interval from a last access to a current time, and a survival time is calculated by calculating a time interval from a data generation time to the current time, and a heat value is calculated based on the access frequency, the time weight and the survival time; A correlation weight is calculated by calculating a reference relationship between each two pieces of data in the de-duplicated data, a semantic similarity between each two pieces of data is calculated, the correlation weight is multiplied by the semantic similarity and an average value is calculated to obtain a correlation degree value; According to task requirements, weight coefficients corresponding to the heat value and the correlation degree value are respectively initialized and weighted summation is performed to obtain a priority value, and the de-duplicated data is sorted in descending order based on the priority value to obtain a to-be-migrated data sequence.

6. The method of claim 1, wherein, The to-be-migrated data sequence is written into the temporary container in batches and a write state value is recorded, when the write state value indicates that pre-committing is successful, a confirmation commit is performed, a bidirectional index table is generated, and a data access request is redirected using the bidirectional index table, including: The to-be-migrated data sequence is grouped into multiple data batches according to a preset batch size, the multiple data batches are written into the temporary container, and a write state value is recorded for each data batch in the multiple data batches; The hash values of the data in each data batch are calculated and summed to obtain a data checksum, the data checksum after writing is compared with the data checksum before writing to obtain an integrity verification result, a pre-committing log record is generated based on the integrity verification result, and a confirmation commit operation is performed when the write state value indicates that pre-committing is successful; The original data identifier of the to-be-migrated data is obtained from the bitmap index container, the new data identifier of the to-be-migrated data is obtained from the temporary container, a mapping relationship is established, a version number and a timestamp are added to generate metadata information, and a bidirectional index table is constructed based on the original data identifier, the new data identifier and the metadata information; The mapping relationship between the original data identifier and the new data identifier is stored in a redirection cache, a data access request is received, a target data location is preferentially searched from the redirection cache, if the target data location is not found, the target data location is searched in the bidirectional index table, if the search fails, it is judged whether the number of retries exceeds a search threshold, if not, the retry is performed, if yes and it is a read operation, the data in the bitmap index container is accessed, and if yes and it is a write operation, the data access request is rejected.

7. The method of claim 1, wherein, When the performance index value after switching exceeds a preset range, a rollback operation is performed and a decision parameter is updated, including: Obtain the performance index value after switching, the performance index value including service response time, resource occupancy rate and service success rate; Determine whether the performance index value exceeds a preset performance threshold range, the preset performance threshold range being determined based on statistical distribution of historical performance data, and trigger a rollback operation when the performance index value continuously exceeds the preset performance threshold range for a duration greater than a preset time threshold; Perform the rollback operation and update decision parameters based on performance fluctuation of the current switching, the decision parameters including switching time window, performance monitoring frequency and performance threshold interval.

8. Bitmap index container optimization system based on multi-dimensional dynamic decision making for implementing the method of any of the preceding claims 1-7, characterized in that, Comprise: A first unit for obtaining a data feature value of a bitmap index container and calculating a skewness coefficient value based on the data feature value; A second unit for processing the data feature value through a pre-set timing diagram structure to obtain a data distribution feature vector, determining a data access rule vector and a corresponding data change trend vector based on the data feature value, combining the data distribution feature vector, the data access rule vector and the data change trend vector across dimensions to obtain a feature fusion vector, and iteratively optimizing decision parameters based on the feature fusion vector combined with a double strategy gradient algorithm to obtain an adaptive threshold for container switching; A third unit for comparing the skewness coefficient value with the adaptive threshold and determining whether to switch the container type; A fourth unit for creating a temporary container to perform deduplication operation on the to-be-migrated data in the bitmap index container to obtain deduplicated data when container type switching is needed, calculating a heat value and an association value of the deduplicated data, taking a weighted sum of the heat value and the association value as a priority value and sorting the deduplicated data to obtain a to-be-migrated data sequence, writing the to-be-migrated data sequence into the temporary container in batches and recording a write state value, executing a confirm commit when the write state value indicates that pre-commit is successful, generating a bidirectional index table and redirecting data access requests using the bidirectional index table; A fifth unit for monitoring the performance index value after switching, and performing a rollback operation and updating decision parameters when the performance index value exceeds a preset range.

9. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 7.

Images