A data processing method, device, storage medium, and program product

Abstract

This application provides a data processing method, comprising: processing the data to be stored using a target data processing model to obtain target feature points corresponding to target data in the data to be stored; the value of the feature points corresponding to the data to be stored is determined based on the value and storage order of the data to be stored; determining target parameters based on the target relational expression representing the distribution relationship of the data to be stored and the target feature points; calculating the size of the basic data based on the parameters of the target data processing model and the target step size; determining the storage offset and flag bits of the data to be stored based on the target parameters and the value of the data to be stored; associating the target feature points with an index data area; associating the offset and flag bits with a compressed data area; and storing the offset, flag bits, and target feature points in the target storage area based on the size of the basic data, thus solving the problem of redundant computational and storage overhead. This application also provides a data processing device, storage medium, and program product.

Description

Technical Field

[0001] This application relates to data processing technology in the field of computer technology, and more particularly to a data processing method, device, storage medium and program product. Background Technology

[0002] The storage engine is a crucial component of a Database Management System (DBMS), responsible for storing, retrieving, and managing data. Columnar storage is a data storage method in databases, allowing data of the same column (of the same type) to be stored in a contiguous storage space; therefore, columnar storage provides data homogeneity. Furthermore, storage engines incorporate complex and diverse indexing mechanisms to improve data retrieval efficiency for users handling massive datasets, depending on the query scenario. Therefore, modern database storage engines all possess two basic functions: data compression and data indexing. To improve data retrieval efficiency and reduce storage costs, many lightweight compression algorithms are embedded within the storage engine, significantly reducing the disk space required for data storage. Regarding data indexing, hash indexes are introduced to accelerate point-to-point queries, and a series of B+ tree variants, such as B* trees and distributed B+ trees, have been developed to handle range queries. Log merging trees have also been designed for write optimization scenarios. To balance query acceleration and reduced storage costs, the storage engine introduced an index building (index locating data) process and a data compression (decompression) process. However, index building and compression are independent, resulting in redundant computational and storage overhead, which limits write performance. Summary of the Invention

[0003] This application provides a data processing method, device, storage medium, and program product, which solves the problem of redundant computational and storage overhead caused by independent index building and compression building in related technologies, and improves data writing performance.

[0004] The technical solution of this application embodiment is implemented as follows: A data processing method, the method comprising: A target data processing model is used to process the data to be stored to obtain target feature points corresponding to the target data in the data to be stored; wherein, each data to be stored corresponds to one feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order; Target parameters are determined based on the target relation and the target feature points; wherein, the target relation characterizes the distribution relationship between the data to be stored; The size of the basic data is calculated based on the parameters of the target data processing model and the target step size; Based on the target parameters and the values ​​of the data to be stored, the storage offset and flag bit of each piece of data to be stored are determined; wherein, the flag bit represents the positional relationship between the data to be stored and the target data; The target feature point is associated with the index data region, and each offset value and each flag bit are associated with the compressed data region. Each offset value, each flag bit, and the target feature point are stored in the target storage region based on the base data size.

[0005] In the above scheme, determining the target parameters based on the target relation and the target feature points includes: Based on the adjacency relationship of the target data corresponding to multiple target feature points in the storage order, the multiple target feature points are divided into multiple feature point sets; wherein, each feature point set includes two feature points corresponding to two target data that are adjacent in the storage order. For each feature point set, the coordinate values ​​of two feature points in the feature point set are calculated based on the target relation to obtain the first parameter and the second parameter corresponding to each feature point set; wherein, the target parameter includes the first parameter and the second parameter.

[0006] In the above scheme, determining the storage offset and flag bit of the data to be stored based on the target parameter and the value of the data to be stored includes: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the storage offset value of each piece of data to be stored is determined.

[0007] In the above scheme, determining the flag bit of each piece of data to be stored based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored includes: The target value is obtained by calculating the first coordinate value, the first parameter, and the second parameter of the feature point corresponding to each data to be stored. Based on the relationship between the target value and the second coordinate value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; wherein, the first coordinate value is determined based on the value of the data to be stored, and the second coordinate value is determined based on the storage order of the data to be stored.

[0008] The method in the above scheme further includes: After receiving a data read request, a reference feature point is determined from the target feature point based on the value of the data to be read; The reference parameters are determined based on the values ​​of the reference feature points and the target relation. The reference position is determined based on the reference parameters, the value of the data to be read, and the error of the target data processing model; Based on the reference location, the basic data size, the reference parameters, and the target step size, the target storage location of the data to be read is determined; Based on the target storage location, the data to be read is determined from the target storage area.

[0009] In the above scheme, determining the target storage location of the data to be read based on the reference location, the basic data size, the reference parameters, and the target step size includes: The number of bytes to be read is determined based on the number of reference positions, the size of the basic data, and the target step size; The target storage location is determined based on the number of bytes to be read, the basic data size, the reference position, the reference parameters, and the target step size.

[0010] In the above scheme, determining the target storage location based on the number of bytes to be read, the basic data size, the reference position, the reference parameters, and the target step size includes: Based on the number of bytes to be read, the basic data size, and the reference position, determine the reference flag bit and reference offset value corresponding to each reference position from the target storage area; Based on the reference parameters, the reference position, the value of the data to be read, and the target step size, determine the reference flag bit and reference offset value corresponding to each reference position; The target storage location is determined from the reference location based on the reference flag, the reference offset value, the reference flag, and the reference offset value.

[0011] A data processing apparatus, the apparatus comprising: The first processing unit is used to process the data to be stored using a target data processing model to obtain target feature points corresponding to the target data in the data to be stored; wherein, each piece of data to be stored corresponds to one feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order; The first determining unit is used to determine target parameters based on the target relation and the target feature points; wherein the target relation characterizes the distribution relationship between the data to be stored; The first processing unit is also used to calculate the size of the basic data based on the parameters of the target data processing model and the target step size; The second determining unit is used to determine the storage offset value and flag bit of each piece of data to be stored based on the target parameter and the value of the data to be stored; wherein the flag bit represents the positional relationship between the data to be stored and the target data; The second processing unit is used to associate the target feature point with the index data area, and associate each offset value and each flag bit with the compressed data area, and store each offset value, each flag bit and the target feature point in the target storage area based on the basic data size.

[0012] A data processing device, the device comprising: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the data processing program in the memory to implement the steps of the above-described data processing method.

[0013] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data processing method described above.

[0014] A computer program product comprising a computer program that, when executed by a processor, implements the aforementioned data processing method.

[0015] The data processing method, device, storage medium, and program product provided in this application embodiment can use a target data processing model to process the data to be stored to obtain target feature points corresponding to the target data in the data to be stored; each data to be stored corresponds to one feature point; the value of the feature point is determined based on the value and storage order of the data to be stored; target parameters are determined based on the target relation and target feature points, the target relation characterizing the distribution relationship between the data to be stored; the basic data size is calculated based on the parameters of the target data processing model and the target step size; the storage offset value and flag bit of each data to be stored are determined based on the target parameters and the value of the data to be stored, the flag bit characterizing the positional relationship between the data to be stored and the target data; the target feature points are associated with the index data area, and each offset value and each flag bit are associated with the compressed data area, based on the basic data size. By storing each offset value, each flag bit, and the target feature point in the target storage area, the target data processing model can be used to process the data to be stored to obtain the target feature points corresponding to the target data. Based on the target parameters determined by the target feature points and the target relation, and the value of the data to be stored, the offset value and flag bit of the data to be stored are determined. Then, the target feature points are mapped to the index data, and the offset values ​​and flag bits are mapped to the compressed data. Based on the size of the basic data to be calculated, only the offset value, flag bit, and target feature point of the data to be stored need to be stored, instead of storing all the compressed data to be stored as in related technologies. Moreover, the information in the index data area and the information in the compressed data area are related, which solves the problem of redundant computation and storage overhead caused by the independent construction of index and compression in related data storage technologies, and improves data writing performance. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0018] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0019] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0020] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0021] This application provides a data processing method, which can be applied to a data processing device, as described above. Figure 1 As shown, the data processing method may include the following steps: Step 101: Use the target data processing model to process the data to be stored to obtain the target feature points corresponding to the target data in the data to be stored.

[0022] Each piece of data to be stored corresponds to a feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order.

[0023] Specifically, the target data processing model can refer to a model that employs a spline interpolation algorithm based on a greedy algorithm. It should be noted that the target data processing model can be trained using a model training module based on a pre-defined model error, employing a greedy spline interpolation method to capture the distribution patterns of the underlying data, and automatically adjusting the model's structure and parameters to adapt to data changes. The model outputs N feature points.

[0024] In one feasible implementation, a set of integer values ​​is used to simulate the data to be stored. For example, if the set of values ​​is 10000001, 10000003, 10000005, 10000006, 10000008, 10000017, 10000019, 10000020, 10000022, and 10000025, the storage order is to assign each value an integer value starting from 0, forming 10 coordinate values: (10000001, 0), (10000003, 1), ... (10000025, 9) (i.e., the feature points corresponding to the data to be stored). The target data processing model has an error set of 2, and the model identifies two target feature points: (10000001, 0) and (10000025, 9).

[0025] Step 102: Determine the target parameters based on the target relation and target feature points.

[0026] The target relation characterizes the distribution relationship between the data to be stored.

[0027] The target relation can include two, and correspondingly, the target parameters can include two parameters. Each target relation can yield one parameter. Specifically, the target parameters can be obtained by substituting the values ​​of the target feature points into the target relation.

[0028] It should be noted that if there are N target feature points, then N-1 sets of target parameters can be determined based on the target relation and the N target feature points.

[0029] Step 103: Calculate the basic data size based on the parameters of the target data processing model and the target step size.

[0030] The parameters of the target data processing model can refer to the error of the target data processing model. It should be noted that the target step size and the error of the target data processing model can be pre-set and can be configured according to actual needs and application scenarios.

[0031] Specifically, the basic data size can be obtained by calculating the error of the target data processing model and the target step size, or it can refer to the bit width, which represents the size of a compressed data.

[0032] In one feasible implementation, a formula can be used. Calculate the size of the basic data ;in, This represents the error of the target data processing model. Indicates the target step size. If It is 2. The value is 8, so the calculated value is... It is 5, meaning 5 bits represent one compressed data.

[0033] Step 104: Determine the storage offset and flag bit for each piece of data to be stored based on the target parameter and the value of the data to be stored.

[0034] The flag bit represents the positional relationship between the data to be stored and the target data.

[0035] Specifically, the storage flag for each piece of data can be determined based on the target parameters and the value of the feature point corresponding to each piece of data to be stored. Furthermore, the storage offset value of each piece of data to be stored is determined based on the target parameters, the value of the feature point corresponding to each piece of data to be stored, and the target step size. .

[0036] In one feasible implementation, if It's 5. So, 4 bits represent the offset value, and 1 bit represents the flag bit.

[0037] Step 105: Associate the target feature point with the index data area, and associate each offset value and each flag bit with the compressed data area. Store each offset value, each flag bit and the target feature point in the target storage area based on the base data size.

[0038] Specifically, the target feature points can be used as the index data area, and the offset values ​​and flag bits can be used as the compressed data area. All target feature points and the offset values ​​and flag bits of all data to be stored can be stored in the target storage area according to the size of the basic data.

[0039] It should be noted that all target feature points, offsets, and flag bits of all data to be stored can be stored according to the target storage format. In one feasible implementation, the flag bits and offset values ​​can be stored as data in the compressed data area, as shown in Table 1 below. The compressed data is packed into bytes using a bit-packing method. x0 is 10000001, y0 is 0, x1 is 100000025, y1 is 9, offset is 0, and size is the size of 10 compressed data bytes. The size is 7, which can be calculated using 10*5 / 8. Furthermore, as shown in Table 1, the area before byte x0 is the compressed data area, and the area starting from word x0 and continuing from x0 (including x0) is the index data area.

[0040]

[0041] Table 1 In other embodiments of this application, data storage schemes in related technologies require at least 40 bytes for 10 numerical bytes; however, the number of bytes required to store data in this application is significantly reduced. For example, assuming there are N feature point sets, this application's scheme requires storing N+1 feature points. For int type column values, the number of bytes is (N+1)*(4+4), while the size of other types of target parameters requires at least N*(8+8).

[0042] Based on the foregoing embodiments, in other embodiments of this application, step 102 can be implemented in the following ways: A1. Based on the adjacency relationship of the target data corresponding to multiple target feature points in the storage order, divide the multiple target feature points into multiple feature point sets.

[0043] Each feature point set includes two feature points corresponding to two adjacent target data in the storage order.

[0044] Specifically, multiple target feature points can be sorted according to the storage order of the data to be stored corresponding to all target feature points from front to back, and two target feature points that are adjacent in the sorting order can be divided into a set to obtain multiple feature point sets.

[0045] In one feasible implementation, if the target feature points are (10000001, 0) and (10000025, 9), then these two target feature points directly form a feature point set.

[0046] A2. For each feature point set, calculate the coordinates of two feature points in the feature point set based on the target relation to obtain the first parameter and the second parameter corresponding to each feature point set.

[0047] The target parameters include the first parameter and the second parameter.

[0048] Specifically, for each feature point set, the coordinate values ​​of two feature points in the feature point set can be substituted into the corresponding target relation to obtain the first parameter and the second parameter.

[0049] In this embodiment of the application, the target relation may include a first relation. Second Relation ;in, Indicates the first parameter. Indicates the second parameter, ( , ) represents the coordinates of the feature point stored first in the storage order. , ) represents the coordinates of the feature point stored later in the storage order; if ( , (10000001, 0), , If the given value is (10000025, 9), then the calculation yields... =0.375, =-3750000.375.

[0050] In other embodiments of this application, step 104 described above can be implemented in the following ways: B1. Based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored, determine the flag bit of each piece of data to be stored.

[0051] In this context, the value of a feature point corresponding to each piece of data to be stored can refer to the coordinate value of the feature point; the set of feature points corresponding to each piece of data to be stored can be determined based on the relationship between the coordinate values ​​of the feature points corresponding to each piece of data to be stored and the coordinate values ​​of the feature points in the set of feature points; specifically, if the second coordinate value of the feature point corresponding to the data to be stored is within the range of the second coordinate values ​​of two feature points in a set of feature points, then the set of feature points is determined to be the set of feature points corresponding to the data to be stored.

[0052] It should be noted that the flag bit of each piece of data to be stored can be obtained by calculating the first parameter, the second parameter of the feature point set corresponding to each piece of data to be stored, and the coordinate value of the feature point corresponding to each piece of data to be stored.

[0053] B2. Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, determine the offset value of each piece of data to be stored.

[0054] Specifically, for each piece of data to be stored, the offset value can be obtained by calculating the first parameter, the second parameter, the coordinate value of the feature point, and the target step size of its corresponding feature point set.

[0055] In the embodiments of this application, formulas can be used. Calculate the offset value; where, This represents the first coordinate value of the feature point corresponding to each piece of data to be stored. This represents the second coordinate value of the feature point corresponding to each piece of data to be stored; taking the feature points corresponding to the data to be stored as (10000001, 0), (10000003, 1)...(10000025, 9) as an example, according to the formula... The calculated offset values ​​can be 0000, 0010, 0100, 1001, 1011, 1000, 0110, 0001, 0001, and 0000, respectively.

[0056] In other embodiments of this application, determining the flag bit of each piece of data to be stored based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored includes: The target value is obtained by calculating the first coordinate value, the first parameter, and the second parameter of the feature point corresponding to each data to be stored. Based on the relationship between the target value and the second coordinate value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined.

[0057] The first coordinate value is determined based on the value of the data to be stored, and the second coordinate value is determined based on the storage order of the data to be stored.

[0058] Specifically, the first coordinate value can refer to the value of the data to be stored, and the second coordinate value can refer to the storage order of the data. It should be noted that the target value can be obtained by multiplying the first coordinate value by the first parameter and adding the product to the second parameter. This can be expressed using a formula. The target value is calculated. Furthermore, the flag can be either 0 or 1. If the second coordinate value corresponding to the data to be stored is greater than the target value, the flag is set to 1; if the second coordinate value corresponding to the data to be stored is less than the target value, the flag is set to 0.

[0059] In other embodiments of this application, the flag bits and offset values ​​of the data to be stored for the above set of feature points (10000001, 0), (10000003, 1)...(10000025, 9) can be as shown in Table 2.

[0060]

[0061] Table 2 It should be noted that the coordinates of N target feature points can be used as the metadata of the index data; the index data generated by the data in the previous example consists of two coordinate points, namely (10000001, 0) and (10000025, 9), and N-1 sets of target parameters are used as the metadata of the compressed data. It can be seen that the metadata used by the index data and the compressed data are both feature points learned and trained by the storage engine from the data. The learning storage engine strongly associates the indexing and compression functions.

[0062] In other embodiments of this application, reference is made to Figure 2 As shown, the method may further include: Step 106: After receiving the data read request, determine the reference feature points from the target feature points based on the value of the data to be read.

[0063] Specifically, from all target feature points, the feature point whose first coordinate value is closest to the value of the data to be read, and whose first coordinate value is greater than or equal to the value of the data to be read, and whose first coordinate value is less than or equal to the value of the data to be read, can be used as a reference feature point.

[0064] In one feasible implementation, assume the value of the data to be read is... The first coordinate of the determined reference feature point is 10000005. and Then it needs to satisfy The reference feature points that meet the conditions can be (10000001, 0) and (10000025, 9).

[0065] Step 107: Determine the reference parameters based on the values ​​of the reference feature points and the target relationship.

[0066] The value of the reference feature point can refer to its coordinates. Specifically, the reference parameters can be calculated by substituting the first and second coordinates of the reference feature point into the first and second equations; the reference parameters may include two parameters. and .

[0067] In this embodiment of the application, if the reference feature points are (10000001, 0) and (10000025, 9), then the reference parameters =0.375, =-3750000.375.

[0068] Step 108: Determine the reference position based on the reference parameters, the value of the data to be read, and the error of the target data processing model.

[0069] Specifically, the values ​​of the reference parameters and the data to be read can be used to calculate the result, and the reference position can be determined based on the error between the calculation result and the target data processing model.

[0070] In one feasible embodiment, a formula can be used. Obtain the calculation result; This represents the result of the operation; since the error is 2, the reference position can be determined as [0, 4).

[0071] Step 109: Determine the target storage location of the data to be read based on the reference location, basic data size, reference parameters, and target step size.

[0072] Specifically, the number of bytes to be read can be determined first based on the reference position, the basic data size, and the target step size. Then, the target storage location can be determined based on the number of bytes to be read, the basic data size, the reference position, the reference parameters, and the target step size.

[0073] Step 110: Based on the target storage location, determine the data to be read from the target storage area.

[0074] Specifically, after determining the target storage location of the data to be read in the target storage area, the data to be read can be obtained from the target storage location in the target storage area.

[0075] In other embodiments of this application, step 109 described above can be implemented in the following ways: C1. Determine the number of bytes to be read based on the number of reference positions, the size of the basic data, and the target step size.

[0076] Specifically, the number of reference positions, the basic data size, and the target step size can be calculated to obtain the number of bytes to be read. In one feasible embodiment, the number of reference positions can be multiplied by the basic data size, and the product divided by the target step size to obtain the number of bytes to be read; if the reference positions are [0, 4), then the number of reference positions is 4, and the basic data size is... The target step size is 5. If the value is 8, then the calculated number of bytes to be read is 3.

[0077] C2. Determine the target storage location based on the number of bytes to be read, the size of the basic data, the reference position, the reference parameters, and the target step size.

[0078] Specifically, the reference flag and reference offset can be determined based on the number of bytes to be read, the size of the basic data, and the reference position; then, the target storage location can be determined based on the reference flag, reference offset, reference parameters, reference position, and target step size.

[0079] In other embodiments of this application, determining the target storage location based on the number of bytes to be read, the basic data size, the reference position, the reference parameter, and the target step size includes: Based on the number of bytes to be read, the size of the base data, and the reference position, determine the reference flag and reference offset value corresponding to each reference position from the target storage area.

[0080] Specifically, data can be read from the target storage area based on the number of bytes to be read, and the flag bits and offset values ​​corresponding to each reference position can be extracted according to the basic data size. In one feasible implementation, in 3 bytes, based on a basic data size of 5, as shown in Table 3 below, 4 flag bits and offset values ​​can be extracted sequentially.

[0081]

[0082] Table 3 Based on the reference parameters, reference position, the value of the data to be read, and the target step size, determine the reference flag and reference offset value corresponding to each reference position.

[0083] Specifically, for each reference position, the reference flag and reference offset value corresponding to each reference position can be obtained by calculating the reference parameters, reference position, value of the data to be read, and target step size.

[0084] In feasible embodiments of this application, the value of the data to be read can be used as... Each reference position is used as ,parameter As , refer to As Substitute into the formula In the process, the reference offset value corresponding to each reference position is calculated, and then... The value and The reference flags corresponding to each reference position are obtained by comparison.

[0085] The target storage location is determined from the reference location based on the reference flag, reference offset, reference flag, and reference offset.

[0086] Specifically, for each reference position, its corresponding reference flag and reference offset value can be compared with its corresponding base flag and base offset value, and the reference position where the reference flag and base flag are the same and the reference offset value are the same is determined as the final target storage position.

[0087] In one feasible embodiment, the reference position ranges from [0, 4). Given the reference position and a known column value of 10000005, the calculated value is... , and Substitute into the formula The reference offset value is 4. The binary representation of 4 is 0100, which does not match the offset value in Table 3, indicating that the data to be read, 10000005, is not at position 1. Then, the reference position is 2, which is substituted into the formula. The calculated reference offset value is 0100, and the reference flag is 1, which perfectly matches the reference flag and reference offset value at position 2 in Table 3. This indicates that the actual location of the data to be read, 10000005, is in row 3. Furthermore, the learning-based storage engine in this application employs a query method without decompression logic, greatly improving query efficiency.

[0088] The data processing method provided in the embodiments of this application can use a target data processing model to process the data to be stored to obtain the target feature points corresponding to the target data. Based on the target parameters determined by the target feature points and the target relation, and the value of the data to be stored, the offset value and flag bit of the data to be stored are determined. Then, the target feature points are mapped to the index data, and the offset value and flag bit are mapped to the compressed data. According to the size of the basic data to be calculated, only the offset value, flag bit and target feature points of the data to be stored need to be stored, instead of storing all the compressed data to be stored as in related technologies. Moreover, the information of the index data area and the information of the compressed data area are related, which solves the problem of redundant calculation and storage overhead caused by the independent index construction and compression construction in the data storage technology of related technologies, and improves the data writing performance.

[0089] Based on the foregoing embodiments, embodiments of this application provide a data processing apparatus that can be applied to... Figure 1 and Figure 2 In the data processing method provided in the corresponding embodiment, refer to Figure 3 As shown, the data processing device 2 may include: a first processing unit 21, a first determining unit 22, a second determining unit 23, and a second processing unit 24, wherein: The first processing unit 21 is used to process the data to be stored using a target data processing model to obtain the target feature points corresponding to the target data in the data to be stored; wherein, each data to be stored corresponds to one feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order; The first determining unit 22 is used to determine target parameters based on the target relation and target feature points; wherein, the target relation characterizes the distribution relationship between the data to be stored; The first processing unit 21 is also used to calculate the size of the basic data based on the parameters of the target data processing model and the target step size; The second determining unit 23 is used to determine the storage offset value and flag bit of each piece of data to be stored based on the target parameter and the value of the data to be stored; wherein the flag bit represents the positional relationship between the data to be stored and the target data; The second processing unit 24 is used to associate the target feature point with the index data area, and associate each offset value and each flag bit with the compressed data area, and store each offset value, each flag bit and the target feature point in the target storage area based on the basic data size.

[0090] In other embodiments of this application, the first determining unit 22 is further configured to perform the following steps: Based on the adjacency relationship of the target data corresponding to multiple target feature points in the storage order, the multiple target feature points are divided into multiple feature point sets; wherein, each feature point set includes two feature points corresponding to two target data that are adjacent in the storage order. For each feature point set, the coordinate values ​​of two feature points in the feature point set are calculated based on the target relation to obtain the first parameter and the second parameter corresponding to each feature point set; wherein, the target parameter includes the first parameter and the second parameter.

[0091] In other embodiments of this application, the second determining unit 23 is further configured to perform the following steps: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each data to be stored, the flag bit of each data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the storage offset value of each piece of data to be stored is determined.

[0092] In other embodiments of this application, the second determining unit 23 is further configured to perform the following steps: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each data to be stored, the flag bit of each data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the offset value of each piece of data to be stored is determined.

[0093] In other embodiments of this application, the second determining unit 23 is further configured to perform the following steps: The target value is obtained by calculating the first coordinate value, the first parameter, and the second parameter of the feature point corresponding to each data to be stored. Based on the relationship between the target value and the second coordinate value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; wherein, the first coordinate value is determined based on the value of the data to be stored, and the second coordinate value is determined based on the storage order of the data to be stored.

[0094] In other embodiments of this application, the second processing unit 24 is further configured to perform the following steps: After receiving a data read request, reference feature points are determined from the target feature points based on the values ​​of the data to be read; The reference parameters are determined based on the values ​​of the reference feature points and the target relationship. The reference position is determined based on the reference parameters, the values ​​of the data to be read, and the error of the target data processing model; Based on the reference location, the size of the basic data, the reference parameters, and the target step size, determine the target storage location of the data to be read; Based on the target storage location, determine the data to be read from the target storage area.

[0095] In other embodiments of this application, the second processing unit 24 is further configured to perform the following steps: The number of bytes to be read is determined based on the number of reference positions, the size of the base data, and the target step size; The target storage location is determined based on the number of bytes to be read, the size of the basic data, the reference position, the reference parameters, and the target step size.

[0096] In other embodiments of this application, the second processing unit 24 is further configured to perform the following steps: Based on the number of bytes to be read, the size of the basic data, and the reference position, determine the reference flag and reference offset value corresponding to each reference position from the target storage area; Based on the reference parameters, reference position, the value of the data to be read, and the target step size, determine the reference flag and reference offset value corresponding to each reference position; The target storage location is determined from the reference location based on the reference flag, reference offset, reference flag, and reference offset.

[0097] It should be noted that the specific implementation process of the steps performed by each unit in the embodiments of this application can be referred to Figure 1 and Figure 2 The implementation process of the data processing method provided in the corresponding embodiments will not be described in detail here.

[0098] The data processing apparatus provided in the embodiments of this application can use a target data processing model to process the data to be stored to obtain the target feature points corresponding to the target data. Based on the target parameters determined by the target feature points and the target relation, and the value of the data to be stored, the offset value and flag bit of the data to be stored are determined. Then, the target feature points are mapped to the index data, and the offset value and flag bit are mapped to the compressed data. According to the size of the basic data to be calculated, only the offset value, flag bit and target feature points of the data to be stored need to be stored, instead of storing all the compressed data to be stored as in related technologies. Moreover, the information of the index data area and the information of the compressed data area are related, which solves the problem of redundant calculation and storage overhead caused by the independent index construction and compression construction in the data storage technology of related technologies, and improves the data writing performance.

[0099] Based on the foregoing embodiments, embodiments of this application provide a data processing device that can be applied to... Figure 1 and Figure 2 In the data processing method provided in the corresponding embodiment, refer to Figure 4As shown, the data processing device 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein: Communication bus 33 is used to realize the communication connection between processor 31 and memory 32; The memory 32 is used to store computer programs that can run on the processor 31; The processor 31 is used to run computer programs to perform the following steps: Obtain the required resource information of the container to be scheduled, and obtain the available and used resource information of each node in the target service cluster; A target data processing model is used to process the data to be stored, thereby obtaining the target feature points corresponding to the target data in the data to be stored; wherein, each data to be stored corresponds to one feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order. Target parameters are determined based on the target relation and target feature points; whereby the target relation characterizes the distribution relationship between the data to be stored. The size of the base data is calculated based on the parameters of the target data processing model and the target step size; The storage offset and flag bit for each piece of data to be stored are determined based on the target parameters and the values ​​of the data to be stored; where the flag bit represents the positional relationship between the data to be stored and the target data. Associate the target feature points with the index data area, and associate each offset value and each flag bit with the compressed data area. Store each offset value, each flag bit, and the target feature point in the target storage area based on the base data size.

[0100] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Based on the adjacency relationship of the target data corresponding to multiple target feature points in the storage order, the multiple target feature points are divided into multiple feature point sets; wherein, each feature point set includes two feature points corresponding to two target data that are adjacent in the storage order. For each feature point set, the coordinate values ​​of two feature points in the feature point set are calculated based on the target relation to obtain the first parameter and the second parameter corresponding to each feature point set; wherein, the target parameter includes the first parameter and the second parameter.

[0101] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each data to be stored, the flag bit of each data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the storage offset value of each piece of data to be stored is determined.

[0102] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each data to be stored, the flag bit of each data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the offset value of each piece of data to be stored is determined.

[0103] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: The target value is obtained by calculating the first coordinate value, the first parameter, and the second parameter of the feature point corresponding to each data to be stored. Based on the relationship between the target value and the second coordinate value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; wherein, the first coordinate value is determined based on the value of the data to be stored, and the second coordinate value is determined based on the storage order of the data to be stored.

[0104] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: After receiving a data read request, reference feature points are determined from the target feature points based on the values ​​of the data to be read; The reference parameters are determined based on the values ​​of the reference feature points and the target relationship. The reference position is determined based on the reference parameters, the values ​​of the data to be read, and the error of the target data processing model; Based on the reference location, the size of the basic data, the reference parameters, and the target step size, determine the target storage location of the data to be read; Based on the target storage location, determine the data to be read from the target storage area.

[0105] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: The number of bytes to be read is determined based on the number of reference positions, the size of the base data, and the target step size; The target storage location is determined based on the number of bytes to be read, the size of the basic data, the reference position, the reference parameters, and the target step size.

[0106] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Based on the number of bytes to be read, the size of the basic data, and the reference position, determine the reference flag and reference offset value corresponding to each reference position from the target storage area; Based on the reference parameters, reference position, the value of the data to be read, and the target step size, determine the reference flag and reference offset value corresponding to each reference position; The target storage location is determined from the reference location based on the reference flag, reference offset, reference flag, and reference offset.

[0107] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 2 The data processing methods provided in the corresponding embodiments will not be described in detail here.

[0108] The data processing device provided in the embodiments of this application can use a target data processing model to process the data to be stored to obtain the target feature points corresponding to the target data. Based on the target parameters determined by the target feature points and the target relation, and the value of the data to be stored, the offset value and flag bit of the data to be stored are determined. Then, the target feature points are mapped to the index data, and the offset value and flag bit are mapped to the compressed data. According to the size of the basic data to be calculated, only the offset value, flag bit and target feature points of the data to be stored need to be stored, instead of storing all the compressed data to be stored as in related technologies. Moreover, the information of the index data area and the information of the compressed data area are related, which solves the problem of redundant calculation and storage overhead caused by the independent construction of index and compression in the data storage technology of related technologies, and improves the data writing performance.

[0109] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors 31 to implement... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the data processing method.

[0110] Based on the foregoing embodiments, embodiments of this application provide a computer program product, including a computer program that can be executed by a processor 31 to perform... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the data processing method.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: A target data processing model is used to process the data to be stored to obtain target feature points corresponding to the target data in the data to be stored; wherein, each data to be stored corresponds to one feature point; the value of the feature point is determined based on the value of the data to be stored and the storage order; Target parameters are determined based on the target relation and the target feature points; wherein, the target relation characterizes the distribution relationship between the data to be stored; The size of the basic data is calculated based on the parameters of the target data processing model and the target step size; Based on the target parameters and the values ​​of the data to be stored, the storage offset and flag bit of each piece of data to be stored are determined; wherein, the flag bit represents the positional relationship between the data to be stored and the target data; The target feature point is associated with the index data region, and each offset value and each flag bit are associated with the compressed data region. Each offset value, each flag bit, and the target feature point are stored in the target storage region based on the base data size.

2. The method according to claim 1, characterized in that, The determination of target parameters based on the target relation and the target feature points includes: Based on the adjacency relationship of the target data corresponding to multiple target feature points in the storage order, the multiple target feature points are divided into multiple feature point sets; wherein, each feature point set includes two feature points corresponding to two target data that are adjacent in the storage order. For each feature point set, the coordinate values ​​of two feature points in the feature point set are calculated based on the target relation to obtain the first parameter and the second parameter corresponding to each feature point set; wherein, the target parameter includes the first parameter and the second parameter.

3. The method according to claim 2, characterized in that, The step of determining the storage offset and flag bits of the data to be stored based on the target parameter and the value of the data to be stored includes: Based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; Based on the first parameter, the second parameter, the value of the feature point corresponding to each piece of data to be stored, and the target step size, the storage offset value of each piece of data to be stored is determined.

4. The method according to claim 3, characterized in that, The step of determining the flag bit of each piece of data to be stored based on the first parameter, the second parameter, and the value of the feature point corresponding to each piece of data to be stored, and the value of the feature point corresponding to each piece of data to be stored, includes: The target value is obtained by calculating the first coordinate value, the first parameter, and the second parameter of the feature point corresponding to each data to be stored. Based on the relationship between the target value and the second coordinate value of the feature point corresponding to each piece of data to be stored, the flag bit of each piece of data to be stored is determined; wherein, the first coordinate value is determined based on the value of the data to be stored, and the second coordinate value is determined based on the storage order of the data to be stored.

5. The method according to claim 1, characterized in that, The method further includes: After receiving a data read request, a reference feature point is determined from the target feature point based on the value of the data to be read; The reference parameters are determined based on the values ​​of the reference feature points and the target relation. The reference position is determined based on the reference parameters, the value of the data to be read, and the error of the target data processing model; Based on the reference location, the basic data size, the reference parameters, and the target step size, the target storage location of the data to be read is determined; Based on the target storage location, the data to be read is determined from the target storage area.

6. The method according to claim 5, characterized in that, Determining the target storage location of the data to be read based on the reference location, the basic data size, the reference parameters, and the target step size includes: The number of bytes to be read is determined based on the number of reference positions, the size of the basic data, and the target step size; The target storage location is determined based on the number of bytes to be read, the basic data size, the reference position, the reference parameters, and the target step size.

7. The method according to claim 6, characterized in that, The process of determining the target storage location based on the number of bytes to be read, the basic data size, the reference position, the reference parameters, and the target step size includes: Based on the number of bytes to be read, the basic data size, and the reference position, determine the reference flag bit and reference offset value corresponding to each reference position from the target storage area; Based on the reference parameters, the reference position, the value of the data to be read, and the target step size, determine the reference flag bit and reference offset value corresponding to each reference position; The target storage location is determined from the reference location based on the reference flag, the reference offset value, the reference flag, and the reference offset value.

8. A data processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute a data processing program in memory to implement the steps of the data processing method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1 to 7.

Images