A data processing method, apparatus and device

Abstract

The application discloses a data processing method, device and equipment, and acquires at least one object, an object identification number of each object and bitmap data corresponding to each object identification number. First bitmap data in the bitmap data corresponds to the address of a first data bucket in a database, and the object identification number starts from 0 and increases one by one. A numerical label is determined, and label data of each object identification number under the numerical label is acquired, and the label data is a numerical value. A first array and a second array corresponding to the numerical label are constructed in the first data bucket. When the number of objects is less than or equal to a first number, the first array is used to store second bitmap data in the bitmap data corresponding to the object identification number, and the second array is used to store the label data under the numerical label corresponding to the object identification number. In this way, not only the object identification number is stored in the data bucket, but also the numerical value under the numerical label can be integrated in the second array only by using one numerical label, so that object circle selection is facilitated.

Description

A data processing method, apparatus and equipment Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, and device. Background Technology

[0002] To achieve efficient and accurate object analysis and selection with massive amounts of data, while reducing data storage space, it is common practice to combine selection labels with Bitmap to store data, and then use bit operations on the Bitmap to achieve object analysis and selection.

[0003] To further save data storage space, RoaringBitmap compressed bitmaps can be used for data storage; RoaringBitmap is an evolution of Bitmap. While RoaringBitmap saves data storage space, it also has limitations. For example, when using RoaringBitmap to store tag data related to numerical values ​​such as "whether it is a numerical value," the large number of numerical enumerations results in a large number of tags of that type, making the stored tag data quite scattered and inconvenient for subsequent object analysis and selection. Summary of the Invention

[0004] In view of this, this application provides a data processing method and apparatus that can improve data processing efficiency.

[0005] To solve the above problems, the technical solution provided in this application is as follows:

[0006] In a first aspect, this application provides a data processing method, the method comprising:

[0007] Obtain at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier; the bitmap data includes a first bitmap data and a second bitmap data, the first bitmap data corresponding to the address of a first data bucket in the database; the object identifiers start from 0 and increment sequentially;

[0008] Determine the numerical labels and obtain the label data under each object identifier for the numerical label; the label data under the numerical label is a numerical value;

[0009] Construct a first array and a second array corresponding to the numerical labels in the first data bucket;

[0010] When the number of objects of the at least one object is less than or equal to the first number, the second bitmap data corresponding to each object identifier is stored as an array element in the first array, and the tag data under the numerical tag of each object identifier is stored as an array element in the second array; the array elements with the same array index in the first array and the second array correspond to the same object identifier.

[0011] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0012] The first acquisition unit is used to acquire at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier; the bitmap data includes first bitmap data and second bitmap data, the first bitmap data corresponding to the address of a first data bucket in the database; the object identifier starts from 0 and increments sequentially;

[0013] The second acquisition unit is used to determine the numerical label and acquire the label data under the numerical label for each object identifier; the label data under the numerical label is a numerical value;

[0014] The first construction unit is used to construct the first array and the second array corresponding to the numerical labels in the first data bucket;

[0015] The first storage unit is configured to store the second bitmap data corresponding to each object identifier as an array element in the first array when the number of objects of the at least one object is less than or equal to a first quantity, and to store the tag data under the numerical tag of each object identifier as an array element in the second array; the array elements with the same array index in the first array and the second array correspond to the same object identifier.

[0016] Thirdly, this application provides an electronic device, comprising:

[0017] One or more processors;

[0018] Storage device, on which one or more programs are stored,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the descriptions.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any of the above.

[0021] Therefore, this application has the following beneficial effects:

[0022] This application provides a data processing method, apparatus, and device to acquire at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier. The bitmap data includes first bitmap data and second bitmap data. The first bitmap data corresponds to the address of a first data bucket in a database. The object identifiers start from 0 and increment sequentially. Numerical labels are determined, and label data under each object identifier's numerical label is acquired. The label data under the numerical label is a numerical value. Then, a first array and a second array corresponding to the numerical labels are constructed in the first data bucket. When the total number of objects is less than or equal to a first quantity, the first array stores the second bitmap data from the bitmap data corresponding to the object identifier, with each second bitmap data serving as an array element in the first array. The second array stores the label data under the numerical label corresponding to the object identifier, with each label data serving as an array element in the second array. Array elements stored at the same position in the first and second arrays correspond to each other; that is, the same array index in the first and second arrays corresponds to the same object identifier. In this way, a first array and a second array are constructed in the data bucket. This allows not only the object identifier to be stored in the first array, but also enables the integration of all tag data (i.e., values) under a single numeric label into the second array. Subsequent object analysis and selection based on the label data can then directly utilize the label data stored in the second array, facilitating subsequent object analysis and selection. Attached Figure Description

[0023] Figure 1 is a schematic diagram of a bitmap provided in an embodiment of this application;

[0024] Figure 2 is a flowchart of a data processing method provided in an embodiment of this application;

[0025] Figure 3 is a schematic diagram of a compressed bitmap provided in an embodiment of this application;

[0026] Figure 4 is a schematic diagram of another compressed bitmap provided in an embodiment of this application;

[0027] Figure 5a is a schematic diagram of an application scenario provided by an embodiment of this application;

[0028] Figure 5b is a schematic diagram of another application scenario provided by the embodiments of this application;

[0029] Figure 5c is a schematic diagram of another application scenario provided by the embodiments of this application;

[0030] Figure 5d is a schematic diagram of another application scenario provided by the embodiments of this application;

[0031] Figure 6 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0032] Figure 7 is a schematic diagram of the basic structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] To facilitate understanding and explanation of the technical solutions provided in the embodiments of this application, the background technology of this application will be described first.

[0035] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0036] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0037] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0038] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0039] To achieve efficient and accurate object analysis and selection with massive amounts of data, while reducing data storage space, it is common practice to combine selection labels with Bitmap to store data, and then use bit operations on the Bitmap to achieve object analysis and selection.

[0040] In this context, selection refers to identifying objects that meet specific requirements from a large pool of objects. Object selection can include selecting products, among other things, and is not limited here. In practical applications, object analysis and selection can be implemented using a labeling platform or a selection platform.

[0041] Selection labels (or simply labels) can be understood as selection criteria. After identifying each object and setting its label, the bitmap corresponding to the label can be obtained based on the object's information. The bitmap contains consecutive binary bits, which represent the label data of the object under that label. For example, when a binary bit is 1, the label data of the object under that label is 1, indicating that the object matches the content of the label after analysis of its information (simply put, it matches the label). When a binary bit is 0, the label data of the object under that label is 0, indicating that the object does not match the content of the label after analysis of its information. Therefore, the bitmap structure enables the storage of label data, which in turn enables the storage of object information. Furthermore, using a bitmap structure for storage achieves data compression, reducing data storage costs.

[0042] Each label corresponds to a bitmap. By performing bitwise operations on the bitmaps corresponding to different labels, efficient and accurate object analysis and selection can be achieved. Bitwise operations include intersection, union, and combinations of intersection and union operations. The specific bitwise operations selected can be determined according to the selection requirements.

[0043] Referring to Figure 1, which is a schematic diagram of a bitmap provided in an embodiment of this application. As shown in Figure 1, the objects include objects 1 to 8, and labels "label i" and "label j" are set according to the actual selection requirements. The binary bits in the bitmap correspond to one object in sequence. For example, the first binary bit in the bitmap is the label data of object 1 under the corresponding label, the second binary bit in the bitmap is the label data of object 2 under the corresponding label, and so on.

[0044] The bitmap corresponding to "Label i" contains the binary bits "1, 0, 1, 0, 1, 1, 1, 1". This means that object 1 has a label data of 1 under "Label i", indicating that object 1 matches "Label i". Object 2 has a label data of 0 under "Label i", indicating that object 2 does not match "Label i", and so on. The bitmap corresponding to "Label j" contains the binary bits "0, 1, 0, 1, 0, 1, 0, 1". This means that object 1 has a label data of 0 under "Label j", indicating that object 1 does not match "Label j". Object 2 has a label data of 1 under "Label j", indicating that object 2 matches "Label b", and so on.

[0045] If the selection requirement is for objects matching "label i + label j", that is, objects that match both "label i" and "label j", then bitwise operations are performed on the bitmaps corresponding to "label i" and "label j" to obtain the objects to be selected. The bitwise operations can be determined based on the selection requirements. Specifically, the bitwise operation is the intersection of binary bits at each identical binary position (e.g., the first binary bit in the bitmap corresponding to "label i" and the first binary bit in the bitmap corresponding to "label j" are at the same binary position). Therefore, the intersection result is "0, 0, 0, 0, 0, 1, 0, 1", which indicates that the objects matching "label i + label j" are objects 6 and 8. Based on this, the selected objects are objects 6 and 8.

[0046] Understandably, in practical applications, label i can be red and label j can be a square, resulting in a selected object that is a red square.

[0047] To further save data storage space, RoaringBitmap compressed bitmaps can be used for data storage. RoaringBitmap is an evolved storage method of Bitmap. It is an open-source compressed bitmap data storage structure used in data compression, search engines, databases, and other fields, and can efficiently store and manipulate large amounts of integer data.

[0048] The following section will introduce the data storage method of RoaringBitmap.

[0049] In RoaringBitmap's data storage method, the database creates Container data buckets. These buckets are used to store data, and each bucket has a unique location number (or address), typically starting from 0 and increasing sequentially. RoaringBitmap currently has two storage versions, used to store 32-bit and 64-bit integers respectively. This application embodiment uses the storage of 32-bit integers as an example. In RoaringBitmap's storage method, the 32-bit integer is first divided into the first 16 bits (also called the high 16 bits) and the last 16 bits (also called the low 16 bits). The first 16 bits are used to determine the location number of the data bucket where the data will be stored, and then the last 16 bits are stored in the determined data bucket. That is, the last 16 bits are the data to be stored in the determined data bucket. For example, the value 31 has the hexadecimal value 0000001F, with the first 16 bits being 0000 and the last 16 bits being 001F. Based on the first 16 bits (0000), the corresponding data bucket position number is determined to be 0. The last 16 bits (001F) are then stored in the data bucket with position number 0 (also known as the 0th position). In practice, the last 16 bits are converted to the integer 31 for storage. It's important to note that data buckets are created only when needed, not by directly initializing all data buckets. This can be understood as the number 31 representing the identifier "31" in object 31. In practical applications, the identifier for each object is determined first.

[0050] RoaringBitmap incorporates various data storage methods (also known as data storage structures) to achieve bitmap compression. These include, for example, ArrayContainer (an array bucket, also known as an ArrayContainer data storage structure), BitmapContainer (a bitmap bucket, also known as a BitmapContainer data storage structure), and RunContainer (a run step bucket).

[0051] The ArrayContainer data storage structure is designed for sparse data storage and can be used to store double-byte (short) integer values ​​(see Figure 3 and section S203 below). Since the lower 16 bits of a 32-bit integer value are stored in a data bucket, and the binary length of a short type is exactly 16 bits, the lower 16 bits of a 32-bit integer value can be stored using the short type. To save storage space, the ArrayContainer data storage structure supports a maximum of 4096 elements (i.e., sparse data storage). Beyond 4096 elements, the ArrayContainer data storage structure is upgraded to a BitmapContainer data storage structure (i.e., dense data storage). In the ArrayContainer data storage structure, one element refers to a short integer value; 4096 elements mean 4096 short integer values. If an element (a short integer value) can represent an object identifier, then an ArrayContainer data storage structure can store 4096 object identifiers. In practical applications, the ArrayContainer data storage structure only stores the object identifiers of objects whose tags match the specified tags.

[0052] Understanding this, "upgrade" means that if a data bucket's data storage structure is an ArrayContainer, after the upgrade, the data storage structure of that data bucket will be updated from ArrayContainer to BitmapContainer. Therefore, data previously stored in the data bucket using the ArrayContainer data storage structure will be re-stored in the same data bucket using the BitmapContainer data storage structure.

[0053] Based on this, the storage capacity of ArrayContainer is (2 + 2 × c) B. In practical applications, in addition to storing integer values, an extra short integer, c, is also stored to represent the number of integer values ​​stored, i.e., the number of elements. Therefore, the first "2" in (2 + 2 × c) represents the number of bytes occupied by this extra short integer c, i.e., 2 bytes. Storing this extra short integer c facilitates subsequent deserialization. The second "2" indicates that if the stored integer value is of short type, one short integer value occupies 2 bytes. Since a total of c integer values ​​are stored, the total number of bytes occupied by the integer values ​​is (2 × c) B. It can be understood that the maximum value of c is 4096. Furthermore, the time complexity of ArrayContainer is O(log(n)).

[0054] The BitmapContainer data storage structure is designed for dense data storage and can be used to store long integer data types. The BitmapContainer data storage structure stores the lower 16 bits of a Bitmap (see Figure 4 and sections A1-A2 below). It can be seen that the lower 16 binary bits can represent 65536 (2 to the power of 16) values ​​(i.e., 65536 object identifiers). The BitmapContainer data storage structure supports a maximum of 1024 elements, each element being a long integer. A long element has 8 bytes = 64 bits, meaning 1024 long elements × 64 bits = 65536 bits. In BitmapContainer, each bit in a long element corresponds to an object identifier (simply called an object ID). Therefore, the BitmapContainer data storage structure can support a maximum of 65536 object identifiers (i.e., 65536 objects). It can be understood that object identifiers start from 0, representing objects 0 through 65535. It is also understandable that the value in each bit is the data of the corresponding object under the tag. Therefore, when storing data using the BitmapContainer data storage structure, it can store the tag data of a maximum of 65536 objects under the tag. The storage capacity of BitmapContainer is fixed at 8KB (1024 long elements, 65536 bits = 8KB), and the time complexity is O(log(1)). Thus, it can be seen that 4096 × 2 bytes = 8KB in ArrayContainer. Therefore, if the ArrayContainer data storage structure is used to store more than 4096 elements, it should be upgraded to the BitmapContainer data storage structure. It is understandable that 8KB of space can store 65536 (2 to the power of 16) binary bits (i.e., 65536 bits).

[0055] RunContainer is used to apply Run-Length Encoding (RLE) to consecutive sequences. For example, the sequence 11, 12, 13, 14, 15 will be compressed into (11, 4), where "4" indicates that there are four consecutive numbers after 11. In the extreme case, if the sequence is all consecutive, then only 4 bytes are needed. However, if the sequence is an odd or even sequence, not only will the storage space not be compressed, but the storage space may even double. RunContainer uses a variable-length array of Unsigned Short integers to store the RLE-compressed data. The storage capacity is related to the number of consecutive sequences run (r), specifically (2 + 4 × r) bytes, and the time complexity is O(log(n)).

[0056] In practical applications, Clickhouse combined with RoaringBitmap can be used for object analysis and selection in tagging platforms. Clickhouse is an open-source, column-oriented data management system for Online Analytical Processing (OLAP), characterized by high performance, scalability, high reliability, and support for SQL-like queries.

[0057] Specifically, the imported data is aggregated using Clickhouse's Bitmap aggregation engine to generate a RoaringBitmap. Then, the object is analyzed and selected using Clickhouse's SQL-like functions and the bitmap bit operation functions already implemented in Clickhouse (e.g., bitmapAnd(): bitmap intersection operation; bitmapOr(): bitmap union operation, etc.).

[0058] The applicant's research revealed that while RoaringBitmap compressed bitmaps are an improved storage method based on bitmaps, saving data storage space, they also have limitations. Like regular bitmaps, RoaringBitmap compressed bitmaps can only store one-dimensional data (essentially storing the result of whether an object matches a label). This limitation is more pronounced when using the RoaringBitmap data storage structure to store numeric data, such as amounts (e.g., sales revenue) and quantities. For example, using the label "Is it 0?", the bitmap for objects 1-8 under this label is "0, 1, 0, 0, 0, 0, 0, 0", indicating that object 2's value is 0. Therefore, the bitmap under the label "Is it 0?" stores the numeric data type of object 2 having a value of 0. In other words, the storage of numeric data is achieved through the aforementioned label and bitmap settings.

[0059] Currently, when using RoaringBitmap to store numeric data, there are two storage methods, both of which highlight the inherent limitations of RoaringBitmap's data storage approach. It's understandable that a RoaringBitmap compressed bitmap can theoretically be converted to a regular bitmap. Since both compressed and regular bitmaps exhibit this limitation, we will use the converted regular bitmap to illustrate its limitation in storing one-dimensional data.

[0060] The first method generates a label for each value, and each label corresponds to a bitmap in the RoaringBitmap data storage structure. The label can represent "whether it is a value xx"; a binary bit of 1 indicates it is a value xx, and a binary bit of 0 indicates it is not a value xx. Since there are many possible values, the number of labels will be large, resulting in a large number of bitmaps.

[0061] For example, when the numerical range is 0-500 (e.g., the sales amount range is 0-500), and the objects include objects 1-8 (e.g., objects can be merchants), the generated labels include 500 labels such as "Is it 0", "Is it 1", ..., "Is it 500", etc. The bitmap under the label "Is it 0" can be "0, 1, 0, 0, 0, 0, 0", indicating that the value of object 2 is 0 (it should be understood that this is the representation of a normal bitmap after conversion, only used to illustrate the current limitations of the RoaringBitmap data storage structure. In practice, when using ArrayContainer to store data, the value "2" will be stored directly, indicating that the label data of object 2 under the label is 1); the bitmap under the label "Is it 1" can be "0, 0, 0, 0, 0, 0, 0, 0" (the representation of a normal bitmap), indicating that no object has a value of 1, and so on.

[0062] RoaringBitmap's data storage method stores one-dimensional data in each bitmap under each label (label data is 1 or 0, indicating whether the object matches or does not match the label). This presents the following problems: the values ​​of objects 1-8 may only be 8 out of 500, scattered across the label data under 500 labels, making the data relatively discrete. Furthermore, it consumes considerable storage space when the number of values ​​is large. Additionally, when selecting objects based on a range of values ​​greater than or less than (e.g., objects with values ​​greater than 100), it requires a union operation on multiple RoaringBitmap data storage structures such as "whether it is 100", "whether it is 101", ..., "whether it is 500", resulting in relatively low efficiency.

[0063] The second method involves dividing the numerical value into intervals. For example, if the value is 0-500, the intervals could be [0, 99], [100, 200], or [200, 500]. A label is generated for each interval, such as "Is the value within 0-99?" or "Is the value within 100-200?". The bitmap under each label is then retrieved. For example, the bitmap under the label "Is the value within 0-99?" would be "0, 0, 0, 1, 0, 0, 0, 0", indicating that the value of object 4 is within 0-99 (this is also a regular bitmap converted from a RoaringBitmap compressed bitmap, for the sake of understanding its limitations).

[0064] In this storage method, because the numerical range is divided manually, when the value changes and the range needs to be added, deleted, or modified, the labels need to be reset according to the new range and the corresponding RoaringBitmap needs to be retrieved again. Furthermore, when performing a greater than or less than numerical range judgment, it cannot accurately retrieve objects that meet the selection requirements. For example, when selecting objects with values ​​greater than 150, since the numerical range [100, 200] includes both greater than 150 and less than or equal to 150, it can only retrieve objects whose values ​​conform to the numerical range [100, 200], but cannot retrieve objects with values ​​greater than 150 within the numerical range [100, 200].

[0065] Based on the above analysis, the applicant found that the RoaringBitmap data storage structure can only be used to store one-dimensional data, which causes inconvenience in subsequent data processing when using the RoaringBitmap data storage structure to store the above-mentioned numerical data.

[0066] Based on this, embodiments of this application provide a data processing method, apparatus, and device to obtain at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier. The bitmap data includes first bitmap data and second bitmap data. The first bitmap data corresponds to the address of a first data bucket in the database. The object identifiers start from 0 and increment sequentially. Numerical labels are determined, and label data under each object identifier's numerical label is obtained; the label data under the numerical label is a numerical value. Furthermore, a first array and a second array corresponding to the numerical labels are constructed in the first data bucket. When the total number of all object identifiers is less than or equal to a first quantity, the first array is used to store the second bitmap data from the bitmap data corresponding to the object identifier, with each second bitmap data serving as an array element in the first array. The second array is used to store the label data under the numerical label corresponding to the object identifier, with each label data serving as an array element in the second array. Array elements stored at the same position in the first and second arrays correspond to each other; that is, the same array index in the first and second arrays corresponds to the same object identifier. In this way, a first array and a second array are constructed in the data bucket. This allows not only the object identifier to be stored in the first array, but also enables the integration of all tag data (i.e., values) under a single numeric label into the second array. Subsequent object analysis and selection based on the label data can then directly utilize the label data stored in the second array, facilitating subsequent object analysis and selection.

[0067] It is understandable that the shortcomings of the above solutions are the result of the applicant's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application below should be considered contributions made by the applicant to the embodiments of this application.

[0068] To facilitate understanding of this application, a data processing method provided by an embodiment of this application will be described below with reference to the accompanying drawings.

[0069] Based on the above, referring to Figure 2, which is a flowchart of a data processing method provided in an embodiment of this application, the method can be executed by any terminal device and / or server, without limitation. Specifically, it can also be executed by a tagging platform or selection platform in any terminal device and / or server. As shown in Figure 2, the method may include S201-S204:

[0070] S201: Obtain at least one object, the object identifier of each object, and the bitmap data corresponding to each object identifier; the bitmap data includes the first bitmap data and the second bitmap data, the first bitmap data corresponds to the address of the first data bucket in the database; the object identifier starts from 0 and increases one by one.

[0071] The object can be a product, etc. The object identifier is used to identify the object. When the object is a product, the object identifier can be a product identifier, etc. There is no restriction here.

[0072] Object identifiers are represented numerically. In some optional examples, object identifiers can be represented by integer values, such as decimal integers. Furthermore, object identifiers typically start from 0 and increment by 1, i.e., they increase sequentially. For example, object 0, object 1, object 2, object 3, ... Here, the "0" in "object 0" is the object identifier, and so on. It is understandable that the actual object referred to by each object identifier, such as "0, 1, 2, 3, ..., can be determined based on the specific circumstances. For example, when the object is a product, "object 0" could refer to shoes from one brand, "object 1" could refer to shoes from another brand, and so on.

[0073] Bitmap data is data represented using binary bits, where each binary bit is 0 or 1, and one binary bit occupies 1 bit. As an optional example, the bitmap data in this embodiment can be 32-bit binary (i.e., the bitmap data is stored as a 32-bit integer) or 64-bit binary (i.e., the bitmap data is stored as a 64-bit integer). When the object identifier is represented as a decimal integer, it can be converted into bitmap data to obtain the bitmap data corresponding to that object identifier. For example, if the object identifier is the decimal value 31, its corresponding 32-bit binary bitmap data is 000000000000 0000 0000 0000 0001 1111.

[0074] Bitmap data can be divided into first bitmap data and second bitmap data, which typically have the same number of bits. The first bitmap data is used to determine the address of the corresponding first data bucket in the database, and the second bitmap data is stored in that first data bucket. As an optional example, when the bitmap data is 32-bit binary bitmap data, the first bitmap data can be the high 16 bits, and the second bitmap data can be the low 16 bits. For example, if the object identifier is the decimal value 31, then the first bitmap data would be 0000 0000 0000 0000, and the second bitmap data would be 000000000001 1111.

[0075] Each data bucket in the database has a corresponding location number (or address). Based on the first bitmap data corresponding to the object identifier, the address of the data bucket storing the second bitmap data corresponding to that object identifier can be determined. This data bucket can be called the first data bucket. It's understandable that different object identifiers correspond to different bitmap data, therefore the first bitmap data corresponding to different object identifiers will be different, and consequently, the address of the data bucket corresponding to the first bitmap data will be different.

[0076] Understandably, the first bitmap data and the second bitmap data (lower 16 bits) can represent values ​​starting from 0, with a maximum value of 65535 (i.e., all 16 bits are 1), for a total of 65536 values ​​(2 to the power of 16). Since the first bitmap data ranges from 0 to 65535, the data bucket positions are typically numbered starting from 0 (decimal) and increasing sequentially. The data bucket positions can be represented in decimal or hexadecimal; no specific limitation is made here. Furthermore, since the second bitmap data also ranges from 0 to 65535, the second bitmap data in each data bucket starts from 0 and ends at 65535. Therefore, when the data bucket position number is 0, the second bitmap data stored in the data bucket is the object identifier itself; thus, the object identifier typically starts from 0.

[0077] For example, if the first bitmap data is 0000 0000 0000 0000, its corresponding decimal value is 0, then its corresponding data bucket position number is 0. Therefore, the second bitmap data 0000 0000 0001 1111 (corresponding to the decimal value 31) is stored in the data bucket with position number 0; this data bucket can be called the first data bucket. As another example, if the object identifier is 65536, and its 32-bit binary bitmap data is 0000 0000 0000 0001 0000 0000 0000 0000, then the high 16 bits are 0000 0000 0000 0001, and the low 16 bits are 0000 0000 0000 0000. Thus, the decimal value of the high 16 bits (first bitmap data) is 1, and the decimal value of the low 16 bits (second bitmap data) is 0. The corresponding first data bucket position number is 1, and the low 16 bits 0000 0000 0000 0000 are stored in the data bucket with position number 1. This data bucket can be called the first data bucket.

[0078] S202: Determine the numeric labels and obtain the label data for each object identifier under the numeric labels; the label data under the numeric labels is a number.

[0079] Tags can be understood as selection criteria. All tag data under a tag is numerical; such tags are called numeric tags. For example, tags like "Sales amount on November 10, 2022" and "Total sales on November 10, 2022" are numeric tags, and their tag data is all numerical (e.g., when the selectable range for sales amount is 0-100, the tag data is a value within the range of 0-100). In related technologies, when the selectable range for sales amount is 0-100, the tags under RoaringBitmap need to be set to 100, such as "Is it 1?", "Is it 2?", ..., "Is it 100?", with tag data of 0 or 1. Taking object 1 as an example, if its sales amount is 50, the data under the numeric tags can directly store the value 50. However, in the RoaringBitmap data storage structure, the tag data under all tags except the "Is it 50?" tag needs to be set to 0, and the tag data under the "Is it 50?" tag needs to be set to 1 to achieve the storage of the numerical data of sales amount 50. Based on this, it can be seen that the numerical labels in this application embodiment have innovated the labels under RoaringBitmap. The numerical labels in this application directly represent "sales amount". The label data corresponding to the numerical labels is a specific numerical value, that is, the specific sales amount can be directly stored in the label data, rather than binary numbers such as 0 or 1.

[0080] In this embodiment, each object identifier can represent an object, and each object has corresponding tag data under a numeric label. The tag data (i.e., the value) under the numeric label for different objects (or different object identifiers) may be the same or different. For example, object 1 has a tag data of 1000 under the label "Sales amount on November 10, 2022," while object 2 has a tag data of 2000 under the same label. The unit for sales amount can be assumed to be "yuan," and this is not limited here.

[0081] It is understood that the embodiments of this application can store tag data under numerical tags corresponding to different object identifiers, and can also select objects based on the tag data and selection requirements. The selection requirements can consist of one or more selection conditions. When there are multiple selection conditions, the selection requirements also include the operational relationships between different selection conditions.

[0082] S203: Construct the first and second arrays corresponding to the numerical labels in the first data bucket.

[0083] In practical applications, bitmap data is stored using "arrays". For example, the bitmap "1, 0, 1, 0, 1, 1, 1, 1" corresponding to "label a" shown in Figure 1 is stored in an array.

[0084] Referring to Figure 3, which is a schematic diagram of a compressed bitmap provided in an embodiment of this application, Figure 3 illustrates the data storage method of the ArrayContainer in RoaringBitmap. The left side of Figure 3 shows a schematic diagram of data buckets, with the bucket positions numbered 0x0, 0x1, ..., 0xA, 0xFFFF. Here, "0x" represents hexadecimal, meaning the position number of the first data bucket shown in Figure 3 is represented by a hexadecimal number. It can be considered that the first data bucket created is the data bucket with position number 0x0. If storage is needed, a data bucket 0x1 is created, and so on.

[0085] As shown in Figure 3, taking data stored in the "0x0" data bucket as an example, if the data bucket uses the ArrayContainer data storage structure, then a one-dimensional array is used to store the data under the ArrayContainer data storage structure. To save storage space, this one-dimensional array stores the object identifier of the object whose label data matches the label. Specifically, it stores the second bitmap data corresponding to the object identifier of the object whose label data matches the label, that is, it stores the second bitmap data corresponding to the object identifier whose label data is 1 (for example, the label is "whether it is xx value"). As an optional example, the second bitmap data corresponding to the object identifier whose label data is 1 is converted into a short type integer before storage. Taking Figure 3 as an example, this one-dimensional array stores short type values ​​such as 3, 10, 18, 50, and 4095. Since the data is stored in the "0x0" data bucket, it means that all the first bitmap data is 0. Therefore, the values ​​3, 10, 18, 50, and 4095 represent the object identifier itself, that is, object 3, object 10, object 18, object 50, and object 4095.

[0086] In Figure 3, objects 3, 10, 18, 50, and 4096 have a label data of 1 under the label "Is it a value of xx" (for example only). This means that the value corresponding to objects 3, 10, 18, 50, and 4096 is that "xx value". It can be understood that the one-dimensional array in the ArrayContainer data storage structure shown in Figure 3 only stores the object identifiers with a label data of 1. This means that the label data corresponding to other object identifiers is 0. Storing only the object identifiers with a label data of 1 can save storage space to a certain extent.

[0087] Additionally, in data buckets other than "0x0", the object identifier is determined by the data bucket's position number and the value in the array. Specifically, the position number of the first data bucket and the value in the array are converted into high 16-bit binary values ​​and low 16-bit binary values, respectively. Then, a 32-bit binary value is obtained based on the high 16-bit binary value and the low 16-bit binary value, and the 32-bit binary value is converted into a decimal object identifier.

[0088] Understandably, in the RoaringBitmap data storage structure, for example, if the label is "whether it is a value of xx", then when the bit is 1, it means that the label data of the object identifier corresponding to this bit is 1, indicating that the number corresponding to the object identifier (the object) is the value of xx in the label. In the ArrayContainer data storage method, the ArrayContainer array stores the object identifier with label data of 1, meaning it can only store one-dimensional data and cannot store the value itself. Therefore, when there are many enumerated values ​​for the value xx, the number of labels (such as "whether it is a value of xx") is large, and the ArrayContainer array will be large. The number of labels and the array size are generally the same, which makes the RoaringBitmap data storage method have the inherent limitations mentioned above.

[0089] Based on this, this application provides a NumBitmap data storage structure, building upon RoaringBitmap, for storing numeric data. The NumBitmap data storage structure can store two-dimensional data of an object, namely, the object identifier (integer) and the tag data (integer) of the object identifier under the numeric tag.

[0090] As an optional example, the NumBitmap data storage structure includes the SkipArrayContainer data storage structure. The SkipArrayContainer data storage structure can be understood as an improvement on the ArrayContainer data storage structure in RoaringBitmap, and it is comparable to the ArrayContainer data storage structure.

[0091] Specifically, under the SkipArrayContainer data storage structure, a first array and a second array corresponding to the numeric labels need to be constructed in the first data bucket. The first data bucket is the data bucket obtained in S201 based on the first bitmap data corresponding to the object identifier. Then, the first array stores the object identifier (specifically, the second bitmap data corresponding to the object identifier), and the second array stores the label data of the object identifier under the numeric labels. This data storage structure, which relies on the first and second arrays to store two dimensions of data—the object identifier and the label data of the object identifier under the numeric labels—is called the SkipArrayContainer data storage structure.

[0092] S204: When the number of objects of at least one object is less than or equal to the first quantity, store the second bitmap data corresponding to each object identifier as an array element in the first array, and store the label data under the numeric label of each object identifier as an array element in the second array; the array elements with the same array index in the first array and the second array correspond to the same object identifier.

[0093] In the SkipArrayContainer data storage structure, when storing object identifiers in the first array, it's only necessary to store the second bitmap data from the bitmap data corresponding to the object identifier in the first array. The first bitmap data from the bitmap data corresponding to the object identifier already identifies the address of the first data bucket containing both the first and second arrays. Each element in the first array is the second bitmap data from the bitmap data corresponding to each object identifier. Additionally, the tag data under the numeric label for the object identifier is stored in the second array. Thus, each element in the second array is the tag data under the numeric label for each object identifier. Therefore, the first bitmap data corresponding to the object identifier is used to determine the address of the first data bucket, while the second bitmap data and the tag data under the numeric label for the object identifier are stored in the first data bucket.

[0094] Similar to RoaringBitmap's ArrayContainer, NumBitmap's SkipArrayContainer is also used to satisfy sparse data storage. Under the SkipArrayContainer data storage structure, its internal array can store a maximum of a first number of elements, which is the maximum number of array elements that the first or second array can store. Specifically, an element in the SkipArrayContainer data storage structure consists of one array element from the first array and one array element from the second array. It can be understood that the number of array elements in the first array is the same as the number of array elements in the second array.

[0095] The number of array elements to be stored in the first array and the number of array elements to be stored in the second array are determined by the number of objects (i.e., the number of object identifiers) of at least one object. For example, when the number of objects is 5, the first and second arrays can store the second bitmap data of the object identifiers of the 5 objects and the tag data of the 5 objects under the numeric labels. In practical applications, when the number of objects is less than or equal to the first number, the second bitmap data is stored as an array element in the first array, and the tag data of the object identifiers under the numeric labels is stored as an array element in the second array. In one possible implementation, this application embodiment provides a method for obtaining the first number, as detailed below.

[0096] As an alternative example, when storing the second bitmap data in the first array, the second bitmap data can first be converted into a decimal short integer before being stored in the first array. The data type of the tag data (i.e., the number) under the numeric tag for the object identifier stored in the second array can be an int integer. Therefore, the first array can be called a short array, and the second array can be called an int array.

[0097] The first array can be represented by 'a', where a = [a0, a1, a2, a3, ...], and a0, a1, a2, a3, etc. are array elements. Since a[0] = a0, the array index of element a0 is [0] (or can be directly written as 0). The rest are similar. The second array can be represented by 'b', where b = [b0, b1, b2, b3, ...], and b0, b1, b2, b3, etc. are array elements. Since b[0] = b0, the array index of element b0 is also [0]. The rest are similar. It can be seen that object identifiers usually start from 0, therefore the array indices of the first and second arrays also start from 0, increasing in increments of 1.

[0098] Referring again to Figure 3, if the first data bucket is the data bucket with position number "0x0", then the first and second arrays under the SkipArrayContainer data storage structure in Figure 3 are stored in the data bucket with position number "0x0". The first array stores five values: 3, 10, 18, 50, and 4095. This can be understood as follows: when the first bitmap data is the high 16 bits and the second bitmap data is the low 16 bits, the five values ​​3, 10, 18, 50, and 4095 stored in the first array are the decimal short integer data converted from the low 16 bits of the bitmap data. Since the first data bucket is "0x0", the high 16 bits are all 0. Therefore, "3, 10, 18, 50, 4095" stored in the first array represent objects 3, 10, 18, 50, and 4095. a=[3,10,18,50,4095], a0=3, a1=10, a2=18, a3=50, a4=4095.

[0099] Additionally, if the location number of the first data bucket is "0x1", the object identifier needs to be determined based on both the location number of the first data bucket and the value in the first array. Specifically, the location number of the first data bucket and the value in the first array are converted into high 16-bit binary values ​​and low 16-bit binary values, respectively. Then, a 32-bit binary value is obtained based on the high 16-bit binary value and the low 16-bit binary value, and finally, the 32-bit binary value is converted into a decimal object identifier. For example, if the location number of the first data bucket is "0x1", it means that the high 16 bits of the bitmap data are 0000000000000 0001; if one element in the first array is 0, it means that the low 16 bits of the bitmap data are 0000 0000000000000, thus the object identifier is 65536, and so on.

[0100] The second array stores 100, 201, 101, 202, and 102, representing that object 3 has a value of 100, object 10 has a value of 201, object 18 has a value of 101, object 50 has a value of 202, and object 4095 has a value of 102. b = [100, 201, 101, 202, 102], b0 = 100, b1 = 201, b2 = 101, b3 = 202, b4 = 102. That is, the SkipArrayContainer data storage structure can store the two-dimensional data of objects: (object 3, value 100), (object 10, value 201), (object 18, value 101), (object 50, value 202), and (object 4095, value 102). Among them, (object 3, value 100) can be called an element under the SkipArrayContainer data storage structure, and (object 10, value 201) and so on can also be called an element under the SkipArrayContainer data storage structure.

[0101] Array elements with the same array index in the first and second arrays correspond to the same object identifier. For example, "object 3" in the first array and "100" in the second array have the same array index, such as [0]. Therefore, the object identifiers corresponding to "object 3" in the first array and "100" in the second array are the same, both being "3". It is understood that Figure 3 is only used as an example for illustration and is not limited to this. In practical applications, the second bitmap data of the corresponding object identifier and the label data under the numeric label are usually stored starting from object 0, with the object identifiers incrementing one by one. For example, (object 0, value of object 0), (object 1, value of object 1), (object 2, value of object 2), etc., then the first array stores 0, 1, 2, etc., and the second array stores the value of object 0, the value of object 1, the value of object 2, etc. That is, in this case, the array index of the first array and the array element at that array index position (i.e., the object identifier) ​​are the same, and the array index of the second array and the array element at that array index position (the object's value) correspond to the same object identifier.

[0102] As described above, compared to the ArrayContainer data storage structure which uses a single array to store one-dimensional data, SkipArrayContainer uses two arrays—a first array and a second array—to store two-dimensional data: the object identifier and the label data of the object identifier under the numeric label. Thus, using only a single numeric label, all the values ​​of the object identifiers under that label can be retrieved, allowing the values ​​to be centrally stored in the second array. Furthermore, when the selection requirement involves judging a numerical range greater than or less than a certain value, the values ​​stored in the second array can be directly retrieved for the greater than or less than range judgment, accurately obtaining the object identifiers that meet the selection requirements, thereby identifying the objects that satisfy the selection criteria.

[0103] Based on the above-described S201-S204, this application provides a data processing method to obtain at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier. The bitmap data includes first bitmap data and second bitmap data. The first bitmap data corresponds to the address of a first data bucket in the database. The object identifiers start from 0 and increment sequentially. Numerical labels are determined, and label data under each object identifier is obtained; the label data under the numerical label is a numerical value. Furthermore, a first array and a second array corresponding to the numerical labels are constructed in the first data bucket. When the total number of objects is less than or equal to a first quantity, the first array stores the second bitmap data from the bitmap data corresponding to the object identifier, with each second bitmap data serving as an array element in the first array. The second array stores the label data under the numerical label corresponding to the object identifier, with each label data serving as an array element in the second array. Array elements stored at the same position in the first and second arrays correspond to each other; that is, the same array index in the first and second arrays corresponds to the same object identifier. In this way, by directly storing the object identifier and its numerical value under the numeric label in the data bucket, only one numeric label is needed to integrate all the label data (i.e., the value) under that numeric label into the second array. Subsequent object analysis and selection based on the label data can then directly access the label data stored in the second array, facilitating subsequent object analysis and selection.

[0104] As an optional example, the NumBitmap data storage structure also includes the ArrayNumContainer data storage structure. The NumBitmap data storage structure will be described below based on A1-A2.

[0105] Based on this, in one possible implementation, the data processing method provided in this application embodiment further includes the following steps:

[0106] A1: Construct the third array corresponding to the numerical labels in the first data bucket.

[0107] Referring to Figure 4, which is a schematic diagram of another compressed bitmap provided in an embodiment of this application, as shown in Figure 4, a one-dimensional array is also used to store data in the BitmapContainer data storage method. Specifically, if the BitmapContainer data storage structure is used, the one-dimensional array stores long type values. Long type values ​​can be converted into binary bits (one long element can be converted into 64 bits), and each binary bit corresponds to an object identifier. In the data bucket with position number 0x0, the object identifier is the value obtained after converting the second bitmap data into decimal. In the data bucket with position number 0x1, the position number of the data bucket is converted into the high 16 bits of binary value, the second bitmap data is used as the low 16 bits of binary value, and then a 32-bit binary value is obtained based on the high 16 bits and the low 16 bits of binary value, and then the 32-bit binary value is converted into a decimal object identifier.

[0108] In Figure 4, the one-dimensional array within the BitmapContainer stores the values ​​3, 5, 1, 10, 2, etc., which are long integers. Since one long integer corresponds to 8 bytes (64 bits), taking the value "3" as an example, its binary representation is 00000000 00000000 00000000 00000000 00000000 00000000 0000000000000000011. The leftmost bit is the first bit, and from right to left (from least significant bit to most significant bit), they are the 1st, 2nd, 3rd, and so on. We can see that the 1st and 2nd bits are 1, and the remaining bits are 0. The 1st bit corresponds to object 0, the 2nd bit to object 1, the 3rd bit to object 2, and so on. This means that the tag data for objects 0 and 1 under the label is 1 (for example, when the label is "whether it is a value of xx", then the value of objects 0 and 1 is the value of xx). Taking the value "5" in Figure 4 as another example, its binary representation is 00000000 000000000000000000000000 00000000 0000000 0000000 00000101. Since the value "3" in Figure 4 occupies the first 64 bits, the value "5" in Figure 4, from right to left, corresponds to the 65th bit, 66th bit, 67th bit, etc. Therefore, the 65th and 67th bits are 1, and the remaining bits are 0. The 65th bit corresponds to object 64, and the 67th bit corresponds to object 66. The tag data for objects 64 and 66 under the label is also 1. The other values ​​are similar and will not be elaborated further here.

[0109] Understandably, the BitmapContainer data storage method can only store one-dimensional data (i.e., values ​​like 3, 5, 1, 10, 2, etc.). This one-dimensional data, converted to binary, is actually the tag data (1 or 0) under the label "whether it is a certain value". Therefore, when there are many enumerated values, the number of tags (such as "whether it is a certain value") is large, and the BitmapContainer array will also be large. The number of tags and the number of arrays are generally the same. Similarly, this results in the inherent limitations of the RoaringBitmap data storage method described above.

[0110] Based on this, this application provides another ArrayNumContainer data storage structure in NumBitmap. The ArrayNumContainer data storage structure can be understood as an improvement on the BitmapContainer data storage structure, and the ArrayNumContainer data storage structure is comparable to the BitmapContainer data storage structure.

[0111] In practical implementation, under the ArrayNumContainer data storage structure, a third array corresponding to the numeric labels needs to be constructed in the first data bucket. The third array can be represented by c, c = [c0, c1, c2, c3, ...], where c0, c1, c2, c3, etc. are array elements, c[0] = c0, then the array index of array element c0 is [0] (or can be written directly as 0), and so on. As an optional example, since the object identifier usually starts from 0, the array index of the third array also starts from 0 and increases in increments of 1.

[0112] It is understandable that the first data bucket in this step is determined by the first bitmap data in the bitmap data corresponding to the object identifier in S201.

[0113] A2: When the number of objects is greater than the first quantity and less than or equal to the second quantity, store the label data of each object's identifier under the numeric label as an array element in the third array; the array index of the third array corresponds to the second bitmap data, and the second quantity is the maximum number of array elements stored in the third array.

[0114] Similar to BitmapContainer in RoaringBitmap, ArrayNumContainer in NumBitmap is a data storage structure used for dense data storage within NumBitmap. Within the ArrayNumContainer data storage structure, its internal array (the third array) can support storing more than a first number of elements. The maximum number of array elements stored in the third array is the second number. This second number represents the maximum number of object identifiers that can be represented under the ArrayNumContainer data storage structure. Since the lower 16 bits can represent 65536 (2 to the power of 16) values ​​(i.e., 65536 object identifiers, ranging from object identifier 0 to 65535), a second number of 65536 is required. An array element in the third array can be considered an element within the ArrayNumContainer data storage structure.

[0115] That is, in the example of A1-A2, when the number of objects is greater than the first quantity and less than or equal to the second quantity, the two-dimensional data of all objects (including the object identifier and the label data of the object identifier under the numeric label) can be stored using the NumBitmap data storage structure.

[0116] In the ArrayNumContainer data storage structure, an array (the third array) is used to represent data in two dimensions. Specifically, the label data of the object identifier under the numeric label is stored as array elements in the third array, and the array index of the third array corresponds to the second bitmap data (used to represent the object identifier). The array elements in the third array can be integers, so the third array can store a maximum of 65536 integer values. "The array index of the third array corresponds to the second bitmap data" can be understood as the array index of the array element being the integer value obtained after converting the second bitmap data corresponding to the object identifier. For example, if the second bitmap data is converted to a decimal integer, then the array index is that decimal integer.

[0117] Since the second bitmap data is used to represent the object identifier, and the array index of the third array corresponds to the second bitmap data, it can also be said that the array index of the third array is used to represent the object identifier. Specifically, there are two cases. First, if the position number of the first data bucket is "0x0", then based on the position number "0x0", the first bitmap data (e.g., the high 16 bits) is all 0. In this case, the decimal integer converted from the second bitmap data (i.e., the array index of the third array) is the object identifier itself, and the array index of the third array is the object identifier itself. Second, if the position number of the first data bucket is a position number other than "0x0", then the object identifier is determined based on the position number of the first data bucket and the array index of the third array. Specifically, the position number of the first data bucket and the array index of the third array are converted into high 16-bit binary values ​​and low 16-bit binary values ​​respectively. Then, a 32-bit binary value is obtained based on the high 16-bit binary value and the low 16-bit binary value, and the 32-bit binary value is converted into a decimal object identifier. For example, if the position number of the first data bucket is "0x1", it means that the high 16 bits of the bitmap data (i.e., the first bitmap data) are 0000 0000 0000 0001; if the array index of an array element in the third array is [0], it means that the low 16 bits of the bitmap data (i.e., the second bitmap data) are 0000 0000 0000 0000, and the object identifier is 65536. The rest are similar.

[0118] Referring again to Figure 4, the third array of ArrayNumContainer in Figure 4 is stored in the first data bucket with position number "0x0". The array index of the third array inside it is the object identifier itself. The third array stores values ​​such as 3, 10, 18, 50, and 4095. If the array index of the third array corresponding to "3" is [0] (i.e., the object identifier is 0), the array index of the third array corresponding to "10" is [1] (i.e., the object identifier is 1), the array index of the third array corresponding to "18" is [2] (i.e., the object identifier is 2), the array index of the third array corresponding to "50" is [3] (i.e., the object identifier is 3), and the array index of the third array corresponding to "4095" is [4] (i.e., the object identifier is 4), then it can be represented as (object 0, value 3), (object 1, value 10), (object 2, value 18), (object 3, value 50), (object 4, value 4095).

[0119] As described above, ArrayNumContainer uses a third array to store two dimensions of data: the object identifier and the label data of the object identifier under the numeric label. Thus, using only a single numeric label, the values ​​of all object identifiers under that label are retrieved, allowing the values ​​to be centrally stored in the third array. Furthermore, when the selection requirement involves judging a numerical range greater than or less than a certain value, the values ​​stored in the third array are directly retrieved for the greater than or less than range judgment, accurately obtaining the object identifiers that meet the selection requirements, thereby identifying the objects that satisfy the selection criteria.

[0120] Understandably, if the number of objects is greater than the first preset number but less than or equal to the second preset number, then the third array under the ArrayNumContainer data storage structure can be used directly to store the two-dimensional data. If the number of objects is less than or equal to the first preset number, then the first and second arrays under the SkipArrayContainer data storage structure can be used directly. If, based on the already stored two-dimensional data, several more objects are subsequently added, causing the total number of objects to exceed the first preset number, then the SkipArrayContainer data storage structure of the data bucket needs to be upgraded to an ArrayNumContainer data storage structure (all within the same data bucket). After the upgrade, the two-dimensional data stored in the first and second arrays under the SkipArrayContainer data storage structure is then re-stored in the third array under the ArrayNumContainer data storage structure.

[0121] For example, if the initial quantity is 16384, meaning there are 10000 objects, the data is stored in the first and second arrays of the SkipArrayContainer data storage structure within the data bucket, using two-dimensional data storage. If 20000 objects are subsequently added, exceeding 16384, the data storage structure within the same data bucket is upgraded (i.e., a third array is constructed). The two-dimensional data of the 10000 objects stored in the first and second arrays of the SkipArrayContainer data storage structure is then moved to the third array of the ArrayNumContainer data storage structure. Similarly, the two-dimensional data of the remaining 20000 objects is also stored in the third array.

[0122] Understandably, when the number of objects exceeds the second limit (e.g., 65536), it indicates that the data bucket is full. At this point, another data bucket can be created to store the additional two-dimensional data of the objects. Understandably, if the number of additional objects to be stored in the newly created data bucket is less than the first preset limit, the SkipArrayContainer data storage structure should still be used. If the number of additional objects to be stored in the newly created data bucket is greater than the first preset limit but less than or equal to the second limit, the ArrayNumContainer data storage structure should still be used.

[0123] For example, if there are a total of 65537 objects (greater than the second number of 65536), and the object IDs (i.e., object identifiers) are represented as id0-id65536, then the two-dimensional data corresponding to objects id0-id65535 are all stored in bucket 0 (position number 0x0), and stored using the third array under the ArrayNumContainer data storage structure. Furthermore, bucket 1 is created (position number 0x1), so the two-dimensional data corresponding to object id65536 is stored in bucket 1. Since bucket 1 contains only the two-dimensional data of one object (i.e., object id65536 and its corresponding value), this single data should be stored in bucket 1 using the un-upgraded SkipArrayContainer data storage structure. Therefore, the two-dimensional data corresponding to object id65535 is in bucket 0, and the two-dimensional data corresponding to object id65536 is in bucket 1. It can be understood that in bucket 1, the value stored for object id65536 in the first array under the SkipArrayContainer data storage structure is 0. Since the high 16 bits of 65536 are 1 and the low 16 bits are 0, the low 16 bits (representing the second bitmap data) in bucket 1 are 0. In one possible implementation, an embodiment of this application provides a method for determining a first quantity, comprising the following steps:

[0124] When the space storage capacity of the third array is a fixed target storage capacity, the first quantity is obtained according to the target storage capacity and the space storage capacity corresponding to a single group of elements.

[0125] The storage capacity of a single group of elements is the sum of the storage capacity of the data types of the array elements in the first array and the storage capacity of the data types of the array elements in the second array.

[0126] As an optional example, when the data type of the array elements in the first array is short integer and the data type of the array elements in the second array is integer, the storage capacity corresponding to a single array of elements is the sum of the storage capacity of the short integer data type and the storage capacity of the integer data type.

[0127] Therefore, the storage capacity of the SkipArrayContainer data storage structure is (2 + (2 + 4) × c) B, where c is the number of element groups. SkipArrayContainer also stores an additional short integer, c, which represents the number of element groups. The first "2" in (2 + (2 + 4) × c) represents the number of bytes occupied by this additional short integer c, which is 2 bytes. The second "2" indicates that when the array elements in the first array are of type short, the short integer values ​​occupy 2 bytes, and the "4" indicates that when the array elements in the first array are of type int, the int integer values ​​occupy 4 bytes. Therefore, a single element group occupies 6 bytes (i.e., the storage capacity corresponding to a single element group). The number of stored element groups is c, so the total number of bytes occupied is ((2 + 4) × c) B. The time complexity of the SkipArrayContainer data storage structure is O(log(n)).

[0128] When the data type of the array elements in the third array is integer, the target storage capacity is the product of the target power of 2 and the storage capacity of the integer data type; the target power is the number of bits in the second bitmap data.

[0129] Understandably, when the target exponent is 16 bits, the third array can contain a maximum of 2 to the power of 16 (i.e., 65536) elements. If the elements of the third array are represented by int type, then a maximum of 65536 int values ​​need to be stored, and the target storage capacity can be the product of 65536 and 4 bytes, i.e., 262144 bytes = 256KB.

[0130] Therefore, the storage capacity of the ArrayNumContainer data storage structure is fixed at 256KB, that is, the target storage capacity of the third array is 256KB (the storage capacity occupied by 65536 int type integers). In addition, the time complexity of the ArrayNumContainer data storage structure is O(log(1)).

[0131] Based on this, when the target storage capacity is 262144 bytes, the first quantity can be (262144-2) / 6 = 43690.

[0132] Understandably, in practical applications, the initial quantity can be adjusted to consider data storage efficiency. For example, the adjusted initial quantity could be 16384 elements. In the B1-B2 example, the array in SkipArrayContainer can store 16384 elements (i.e., both the first and second arrays have 16384 elements). After exceeding 16384 elements, it will be upgraded to ArrayNumContainer.

[0133] It is understandable that the maximum value of c in the SkipArrayContainer data storage structure is the first quantity.

[0134] As can be seen from the above, the NumBitmap data storage structure occupies more storage space than the RoaringBitmap data storage structure. Therefore, as an alternative example, the NumBitmap data storage structure can be temporarily used in memory, with its storage space released after use and subsequently reclaimed.

[0135] If persistent storage of data is based on the NumBitmap data storage structure, the stored data can be further compressed. Therefore, as another optional example, the array elements in the first and second arrays can be compressed and serialized based on Varint encoding.

[0136] Varint (Variable-length integer) encoding is a format for encoding integers whose length varies depending on the size of the integer. Smaller numbers are encoded using fewer bytes, while larger numbers are encoded using more bytes. Understandably, larger integers are relatively rare in practical applications, therefore, data compression through Varint encoding can achieve a high compression ratio and good compression effect, reducing storage and network bandwidth resources.

[0137] Understandably, the array elements in the third array can also be compressed and serialized based on Varint encoding.

[0138] The following section will introduce application scenarios for selecting objects by combining the RoaringBitmap data storage structure and the NumBitmap data storage structure provided in the embodiments of this application.

[0139] In practical applications, tags can also be non-numerical tags. Non-numerical tags do not involve numerical values ​​but rather some attribute information of the object itself. For example, if the object is a product, the object identifier is the product identifier, and the tag is "sports shoes," then "sports shoes" can be seen as a type of product, and type can be seen as an attribute of the product. Therefore, non-numerical tags can also be called attribute tags. For example, if there are three products: Product 1, Product 2, and Product 3, with product identifiers 1, 2, and 3 respectively, the tag data for the three product identifiers under the tag "sports shoes" are 1, 0, and 1 respectively, indicating that Product 1 and Product 3 are sports shoes, while Product 2 is not.

[0140] As an optional example, the bitmap data corresponding to each object identifier can also be divided into third bitmap data and fourth bitmap data. The third bitmap data and fourth bitmap data typically have the same number of bits. The third bitmap data corresponds to the address of the second data bucket in the database. It is understood that in this example, if the bitmap data corresponding to the object identifier is still 32 bits, then the third bitmap data and fourth bitmap data can both be 16 bits, corresponding to the high 16 bits and low 16 bits respectively. In this case, the second data bucket and the first data bucket obtained from the same object identifier are the same data bucket, the third bitmap data is the same as the first bitmap data, and the fourth bitmap data is the same as the second bitmap data (the following embodiments are all illustrated using this example).

[0141] Based on this, in one possible implementation, the data processing method provided in this application embodiment further includes the following steps:

[0142] C1: Determine the non-numeric labels and obtain the label data for each object identifier under the non-numeric labels; the label data under the non-numeric labels is non-numeric.

[0143] In this step, the object identifier is the object identifier described in S201. That is, for an object, a non-numeric label can be set to select the object, a numeric label can be set to select the object, or a combination of non-numeric labels and numeric labels can be used to select the object.

[0144] Each object identifier has a label of either 1 or 0 under the non-numeric label. "1" indicates that the object identifier satisfies / conforms to the non-numeric label, and "0" indicates that the object identifier does not satisfy / conform to the non-numeric label. Since the object identifier is used to represent an object, "1" essentially means that the object satisfies the non-numeric label, and "0" essentially means that the object does not satisfy the non-numeric label.

[0145] C2: Construct the fourth and fifth arrays corresponding to the non-numeric labels in the second data bucket.

[0146] For numeric tags, the NumBitmap data storage structure provided in this application provides a good method for storing tag data under numeric tags. For non-numeric tags, the RoaringBitmap data storage structure can be used to store tag data under non-numeric tags.

[0147] Specifically, a fourth and fifth array corresponding to non-numeric labels are constructed in the second data bucket. The fourth array is used to implement the ArrayContainer data storage structure, and the fifth array is used to implement the BitmapContainer data storage structure.

[0148] C3: When the number of objects whose tag data matches the non-numeric tag is less than or equal to the third number, store the fourth bitmap data corresponding to the object identifier of the non-numeric tag in the fourth array as an array element.

[0149] In the RoaringBitmap data storage structure, the third number is 4096.

[0150] Understandably, within the ArrayContainer data storage structure, only the object identifiers of objects whose tag data matches non-numeric tags are stored. Therefore, when the number of objects whose tag data matches non-numeric tags is less than or equal to the third quantity, the ArrayContainer data storage structure is used to store the data. The fourth bitmap data corresponding to the object identifiers of objects whose tag data matches non-numeric tags is stored in the fourth array. Specifically, if object identifier 31 satisfies the non-numeric tag condition (i.e., the tag data under the non-numeric tag is 1), then the fourth bitmap data (such as the lower 16 bits in a 32-bit system) corresponding to "31" is stored in the fourth array.

[0151] Specifically, the fourth bitmap data is converted into a short integer and then stored in the fourth array (refer to the introduction of ArrayContainer in S203 above). It can be understood that the array under ArrayContainer shown in Figure 3 is the fourth array, which stores object identifiers such as object 3, object 10, object 18, object 50, and object 4096, and the tag data corresponding to each object identifier under the non-numeric tag is all 1.

[0152] C4: When the number of objects whose tag data under the non-numeric tag matches the number of objects with the non-numeric tag is greater than the third quantity and the number of objects of at least one object is less than or equal to the fourth quantity, the tag data of each object identifier under the non-numeric tag is stored as an array element in the fifth array; the fourth quantity is the maximum number of array elements stored in the fifth array.

[0153] When the number of objects with non-numeric tags is greater than the third quantity and the number of objects with at least one non-numeric tag is less than or equal to the fourth quantity, the BitmapContainer data storage structure can be used to store the data. Here, the fourth quantity is the maximum number of array elements stored in the fifth array. For example, the fourth quantity is 65536 (see the above description of the BitmapContainer data storage structure).

[0154] The tag data under each object identifier's non-numeric tag is stored as an array element in the fifth array. Since each object identifier corresponds to one bit, in practice, every 64 bits can be converted into a long integer, and then the obtained long integer is stored as an array element in the fifth array. The fourth bitmap data corresponding to each object identifier is used to represent the object identifier. Specifically, the fourth bitmap data corresponding to each object identifier can be converted into a decimal integer. When the data bucket's position number is 0x0, this decimal integer is the object identifier. When the data bucket's position number is not 0x0, the actual corresponding object identifier can be obtained from the data bucket's position number and the fourth bitmap data (refer to the introduction of the BitmapContainer data storage structure in A1 above). It can be understood that the array under BitmapContainer shown in Figure 4 is the fifth array.

[0155] The ArrayContainer data storage structure supports a maximum of 4096 elements. Beyond 4096 elements, it will switch to the BitmapContainer data storage structure. The BitmapContainer data storage structure has a maximum of 1024 long elements, but can represent a maximum of 65536 object identifiers.

[0156] It is understandable that when the first bitmap data and the second bitmap data correspond to the high 16 bits and the low 16 bits respectively, and the third bitmap data and the fourth bitmap data also correspond to the high 16 bits and the low 16 bits respectively, the data stored in the NumBitmap data storage structure and the data stored in the RoaringBitmap data storage structure can be better combined to realize the selection of objects under the selection requirements, thereby expanding the object selection and analysis scenarios.

[0157] Based on this, when the data bits of the first bitmap data and the data bits of the third bitmap data are the same, the data processing method used in the embodiments of this application further includes the following steps:

[0158] D1: Construct bitmap data corresponding to non-numeric labels based on the array elements of the fourth or fifth array; each bit in the bitmap data corresponding to non-numeric labels is used to indicate whether the object identifier satisfies the non-numeric label requirement.

[0159] It is understandable that the array elements in the fourth array are short decimal integers, and the array elements in the fifth array are long decimal integers. Therefore, the array elements of the fourth or fifth array can be used to construct the bitmap data corresponding to the non-numeric label. The bitmap data consists of binary bits, and each bit in the bitmap data is used to indicate whether the object identifier satisfies the non-numeric label.

[0160] D2: According to the numerical selection requirements, obtain the bitmap data corresponding to the numerical label based on the array elements of the second array corresponding to the numerical label.

[0161] In this step, the numerical selection requirement can be something like "greater than," "less than," "greater than or equal to," or "less than or equal to a certain value." For example, the numerical selection requirement could be "select values ​​greater than 100" or "select values ​​greater than 100 and less than 200." It is clear that the numerical selection requirement is part of the selection criteria.

[0162] Since the array elements in the second array are all numerical values ​​of object identifiers under numeric labels, each array element in the second array can be matched against the numeric selection requirements to determine whether the array element meets the requirements. If the numeric element meets the requirements, the binary bit of the corresponding object identifier is set to 1; otherwise, the binary bit of the corresponding object identifier is set to 0. These binary bits are then used to form the bitmap data corresponding to the numeric label.

[0163] D3: Perform logical operations on the bitmap data corresponding to the numeric labels and the bitmap data corresponding to the non-numeric labels to obtain the target bitmap.

[0164] Logical operations are performed on the bitmap data corresponding to numeric labels and the bitmap data corresponding to non-numeric labels to obtain the target bitmap. Then, the objects to be selected can be determined based on the target bitmap. Specifically, the object identifier corresponding to each "1" bit in the target bitmap is determined; the objects corresponding to these object identifiers are the objects that meet the selection requirements.

[0165] Logical operations are also part of the selection requirements. Once the selection requirements are determined, they are implemented through numerical selection criteria and logical operations. Logical operations include logical AND and logical OR operations, and the specific logical operations can be determined based on the selection requirements.

[0166] It is understandable that step D2 above can also be applied to the third array.

[0167] To make it easier to understand how object selection is achieved by combining the NumBitmap and RoaringBitmap data storage structures in D1-D3, examples will be used to introduce D1-D3 below.

[0168] Referring to Figure 5a, which is a schematic diagram of an application scenario provided by an embodiment of this application. As shown in Figure 5a, the non-numerical tags include tag a, tag b, and tag c. The objects include objects 1 to 8. Since the number of object identifiers is less than 4096, the ArrayContainer data storage structure in RoaringBitmap is used to store the data. Furthermore, the data bucket is determined based on the object identifier. Since the objects include objects 1 to 8, it is the "0x0" data bucket. Three arrays are created in the "0x0" data bucket to store the tag data of each object identifier under tag a, tag b, and tag c, respectively. For example, array 1 stores the tag data of each object identifier under tag a, array 2 stores the tag data of each object identifier under tag b, and array 3 stores the tag data of each object identifier under tag c.

[0169] It is understandable that arrays 1 through 3 can all be viewed as the fourth array under the ArrayContainer data storage structure in C3. Therefore, under the ArrayContainer data storage structure, arrays 1 through 3 actually store the fourth bitmap data that satisfies the requirements for non-numeric labels. Based on this, bitmap data corresponding to label a, label b, and label c can be constructed respectively using the array elements in arrays 1 through 3.

[0170] As shown in Figure 5a, the bitmap data corresponding to label a is "1, 0, 0, 0, 0, 0, 1, 1", the bitmap data corresponding to label b is "0, 0, 1, 0, 1, 1, 0, 0", and the bitmap data corresponding to label c is "0, 1, 0, 1, 0, 1, 0, 1".

[0171] The first selection condition in the selection requirement is: objects from objects 1-8 that satisfy both label a and label c, and objects that satisfy both label b and label c. Therefore, we can first perform a logical OR operation (i.e., find the union) on the bitmap data corresponding to label a and label b, obtaining the bitmap data "1, 0, 1, 0, 1, 1, 1, 1". Based on this bitmap data, we can select objects that satisfy either label a or label b. That is, the object corresponding to the object identifier with a binary bit of 1 in this bitmap data is the object that satisfies either label a or label b. Then, we perform a logical AND operation (i.e., find the intersection) on this bitmap data and the bitmap data corresponding to label c, obtaining the bitmap data "0, 0, 0, 0, 0, 1, 0, 1". Based on this bitmap data, we can obtain the objects that satisfy the first selection condition, that is, the object corresponding to the object identifier with a binary bit of 1 in this bitmap data is the object that satisfies the first selection condition (i.e., the object that satisfies both label a and label c, and the object that satisfies both label b and label c).

[0172] Referring to Figure 5b, which is a schematic diagram of another application scenario provided by an embodiment of this application, the numerical labels include two numerical labels: "the value of 2022-11-10" and "the value of 2022-11-11". The values ​​"100, 98, 10, 200, 32, 23, 6, 7" corresponding to "the value of 2022-11-10" shown in Figure 5b are array elements stored in array 4, and the values ​​"500, 40, 32, 302, 23, 42, 20, 94" corresponding to "the value of 2022-11-11" are array elements stored in array 5. Arrays 4 and 5 can both be considered as the second array under the NumBitmap data storage structure (the third array is similar).

[0173] The second selection criterion in the selection requirement is: objects in objects 1-8 whose sum of the values ​​from 2022-11-10 and 2022-11-11 is greater than or equal to 100. Therefore, we first sum the values ​​under the two numeric labels "values ​​from 2022-11-10" and "values ​​from 2022-11-11", obtaining the sum of the values ​​corresponding to objects 1-8 as "600, 138, 42, 502, 55, 65, 26, 101". The array storing "600, 138, 42, 502, 55, 65, 26, 101" can also be considered a second array, thus "600, 138, 42, 502, 55, 65, 26, 101" can also be considered array elements of the second array corresponding to the numeric labels in D2. Then, the sum of the eight values ​​greater than or equal to 100 is obtained, namely 600, 138, 502, and 101. If the sum is greater than or equal to 100, the binary bit corresponding to the object identifier is set to 1; otherwise, it is set to 0. This yields the bitmap data for objects 1-8 as "1, 1, 0, 1, 0, 0, 0, 1". The object corresponding to the object identifier with a binary bit set to 1 is the object that satisfies the second selection condition. The second selection condition can be seen as the numerical selection requirement in D2, and the bitmap data "1, 1, 0, 1, 0, 0, 0, 1" can be considered as the bitmap data corresponding to the numerical label.

[0174] The third selection condition in the selection requirement is: objects in objects 1-8 that satisfy either the first selection condition or the second selection condition. Refer to Figure 5c, which is a schematic diagram of another application scenario provided by an embodiment of this application. Thus, as shown in Figure 5c, the bitmap data "0, 0, 0, 0, 0, 1, 0, 1" obtained from the first selection condition can be logically ORed with the bitmap data "1, 1, 0, 1, 0, 0, 0, 1" obtained from the second selection condition to obtain the bitmap data "1, 1, 0, 1, 0, 1, 0, 1". This achieves step D3 above, and "1, 1, 0, 1, 0, 1, 0, 1" is the target bitmap. The final selected objects are those in objects 1-8 that satisfy both label a and label c, those that satisfy both label b and label c, and those whose sum of values ​​for the two days 2022-11-10 and 2022-11-11 is greater than 100.

[0175] Depending on the specific selection requirements, the following steps can be included in addition to D1:

[0176] E1: Perform a logical AND operation between the array elements of the second array corresponding to the numeric label and the bitmap data corresponding to the non-numeric label to obtain the numeric result corresponding to each object identifier.

[0177] E2: Based on the numerical calculation result and numerical calculation requirements corresponding to each object identifier, perform numerical calculations on the numerical calculation result corresponding to the object identifier to obtain the final numerical calculation result.

[0178] The numerical calculation requirement refers to the requirement to perform calculations on the numerical results corresponding to each object identifier.

[0179] E1-E2 are illustrated by examples. See Figure 5d, which is a schematic diagram of another application scenario provided by an embodiment of this application.

[0180] If the selection requirement is the sum of the values ​​of objects 1-8 that satisfy both label a and label c on November 10, 2022 and November 11, 2022, and the sum of the values ​​of objects that satisfy both label b and label c on November 10, 2022 and November 11, 2022, then as shown in Figure 5d, we can first perform a logical AND operation between the numerical sums of objects 1-8 respectively, “600, 138, 42, 502, 55, 65, 26, 101”, and the bitmap data “0, 0, 0, 0, 0, 1, 0, 1” obtained from the first selection condition, to obtain the numerical operation result corresponding to each object identifier, namely “0, 0, 0, 0, 0, 65, 0, 101”. It is understandable that “600, 138, 42, 502, 55, 65, 26, 101” can be regarded as array elements of the second array corresponding to the numeric labels in E1, and “0, 0, 0, 0, 0, 1, 0, 1” can be regarded as bitmap data corresponding to the non-numeric labels in E1.

[0181] The numerical calculation result "0, 0, 0, 0, 0, 65, 0, 101" indicates that objects 6 and 8 are objects from objects 1-8 that satisfy both label a and label c, and objects that satisfy both label b and label c. Furthermore, the sum of the values ​​for object 6 on November 10th and 11th, 2022 is 65, and the sum of the values ​​for object 8 on the same days is 101. Further, the numerical calculation requires finding the sum of the numerical values, so the sum is 65 + 101 = 166, and 166 is the final numerical calculation result in E2.

[0182] It is understood that the above embodiments are all based on the premise that the bitmap data corresponding to the object identifier is 32-bit data. If the number of object identifiers exceeds the maximum value of 32-bit data, the data can be stored in a sharding manner, with each shard containing 0xFFFF data buckets. Each shard has a corresponding shard number, such as shard 1, shard 2, shard 3, etc. Each shard can store a maximum value that can be represented by 32 bits of binary, denoted as aa, thus ensuring that the value in each shard can be represented by 32 bits of binary. Object identifiers are stored starting from shard 1. After shard 1 is full, shard 2 is used to store object identifiers, and so on. Therefore, the object identifier actually represented by the value in a shard is equal to the number of shards preceding that shard multiplied by aa, plus the value in that shard. For example, if the value in shard 2 is 10, and the only shard preceding shard 2 is shard 1, then the number of shards preceding shard 2 is 1. Therefore, the object identifier actually represented by the value 10 in shard 2 is 1 multiplied by aa plus 10, i.e., (aa+10). Thus, (aa+10) is the object identifier actually represented by the value 10 in shard 2. It can be understood that each shard can be a partition within memory or a memory segment; this is not limited and can be determined based on the actual situation.

[0183] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0184] Based on the data processing method provided in the above embodiments, this application also provides a data processing apparatus, which will be described below with reference to the accompanying drawings. Since the principle by which the apparatus in this disclosure solves the problem is similar to the data processing method described above in this application, the implementation of the apparatus can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0185] Referring to Figure 6, this figure is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. As shown in Figure 6, the data processing device includes:

[0186] The first acquisition unit 601 is used to acquire at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier; the bitmap data includes first bitmap data and second bitmap data, the first bitmap data corresponding to the address of a first data bucket in the database; the object identifier starts from 0 and increases sequentially;

[0187] The second acquisition unit 602 is used to determine the numerical label and acquire the label data under the numerical label for each object identifier; the label data under the numerical label is a numerical value;

[0188] The first construction unit 603 is used to construct the first array and the second array corresponding to the numerical labels in the first data bucket;

[0189] The first storage unit 604 is used to store the second bitmap data corresponding to each object identifier as an array element in the first array when the number of objects of the at least one object is less than or equal to a first quantity, and to store the tag data under the numerical tag of each object identifier as an array element in the second array; the array elements with the same array index in the first array and the second array correspond to the same object identifier.

[0190] In one possible implementation, the device further includes:

[0191] The second construction unit is used to construct a third array corresponding to the numerical labels in the first data bucket;

[0192] The second storage unit is used to store the tag data of each object identifier under the numerical tag as an array element in the third array when the number of objects is greater than the first quantity and less than or equal to the second quantity; the array subscript of the third array corresponds to the second bitmap data, and the second quantity is the maximum number of array elements stored in the third array.

[0193] In one possible implementation, the device further includes:

[0194] The third acquisition unit is used to acquire the first quantity based on the target storage capacity and the space storage capacity corresponding to a single group of elements when the space storage capacity of the third array is a fixed target storage capacity.

[0195] Wherein, the space storage capacity corresponding to the single group of elements is the sum of the space storage capacity corresponding to the data type of the array elements of the first array and the space storage capacity corresponding to the data type of the array elements of the second array.

[0196] In one possible implementation, when the data type of the array elements of the first array is short integer and the data type of the array elements of the second array is integer, the space storage capacity corresponding to the single group of elements is the sum of the space storage capacity of the short integer data type and the space storage capacity of the integer data type.

[0197] Wherein, when the data type of the array elements of the third array is integer, the target storage capacity is the product of the target power of 2 and the space storage capacity of the integer data type; the target power is the number of bits of the second bitmap data.

[0198] In one possible implementation, the bitmap data corresponding to each object identifier includes third bitmap data and fourth bitmap data; the third bitmap data corresponds to the address of the second data bucket in the database; the device further includes:

[0199] A determining unit is configured to determine non-numeric tags and obtain tag data for each object identifier under the non-numeric tags; the tag data under the non-numeric tags is non-numeric.

[0200] The third construction unit is used to construct the fourth and fifth arrays corresponding to the non-numeric labels in the second data bucket;

[0201] The third storage unit is used to store the fourth bitmap data corresponding to the object identifier of the non-numerical label as an array element in the fourth array when the number of objects whose tag data under the non-numerical label matches the object of the non-numerical label is less than or equal to the third quantity.

[0202] The fourth storage unit is used to store the tag data of each object identifier under the non-numerical tag as an array element in the fifth array when the number of objects matching the tag data under the non-numerical tag is greater than the third quantity and the number of objects of the at least one object is less than or equal to the fourth quantity; the fourth quantity is the maximum number of array elements stored in the fifth array; the fourth bitmap data corresponding to each object identifier corresponds to the corresponding array element.

[0203] In one possible implementation, when the data bits of the first bitmap data and the data bits of the third bitmap data are the same, the apparatus further includes:

[0204] The fourth construction unit is used to construct bitmap data corresponding to the non-numeric label based on the array elements of the fourth array or the fifth array; each bit in the bitmap data corresponding to the non-numeric label is used to indicate whether the object identifier satisfies the non-numeric label.

[0205] The fourth acquisition unit is used to acquire the bitmap data corresponding to the numerical label according to the array elements of the second array corresponding to the numerical label, based on the numerical selection requirements.

[0206] The first processing unit is used to perform logical operations on the bitmap data corresponding to the numerical label and the bitmap data corresponding to the non-numerical label to obtain the target bitmap.

[0207] In one possible implementation, the device further includes:

[0208] The second operation unit is used to perform a logical AND operation between the array elements of the second array corresponding to the numerical label and the bitmap data corresponding to the non-numerical label to obtain the numerical operation result corresponding to each object identifier.

[0209] The third calculation unit is used to perform numerical calculations on the numerical calculation results corresponding to each object identifier according to the numerical calculation results and numerical calculation requirements, so as to obtain the final numerical calculation result.

[0210] In one possible implementation, the device further includes:

[0211] A compression serialization unit is used to compress and serialize the array elements in the first array and the second array based on Varint encoding.

[0212] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0213] It should be noted that the specific implementation of each unit in this embodiment can be found in the relevant descriptions in the above method embodiments. The division of units in this application embodiment is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. The functional units in this application embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiments, the processing unit and the sending unit can be the same unit or different units. The integrated unit can be implemented in hardware or as a software functional unit.

[0214] Based on the data processing method provided in the above embodiments, this application also provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method described in any of the above embodiments.

[0215] Referring now to FIG7, a schematic diagram of the structure of an electronic device 700 suitable for implementing embodiments of this application is shown. The terminal device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Android Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (televisions), desktop computers, etc. The electronic device shown in FIG7 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0216] As shown in Figure 7, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0217] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 shows electronic device 700 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0218] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this application.

[0219] The electronic device provided in this application embodiment and the data processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0220] Based on the data processing method provided in the above embodiments, this application provides a computer-readable medium storing a computer program thereon, wherein the program, when executed by a processor, implements the data processing method as described in any of the above embodiments.

[0221] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0222] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0223] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0224] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned data processing method.

[0225] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0226] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0227] The units described in the embodiments of this application can be implemented in software or in hardware. The name of the unit / module does not necessarily limit the unit itself; for example, a voice data acquisition module can also be described as a "data acquisition module".

[0228] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0229] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0230] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0231] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0232] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0233] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0234] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, The method includes: acquiring at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier; the bitmap data includes first bitmap data and second bitmap data, the first bitmap data corresponding to the address of a first data bucket in a database; the object identifiers start from 0 and increment sequentially; determining numeric labels and acquiring label data under the numeric labels for each object identifier; the label data under the numeric labels is a value; constructing a first array and a second array corresponding to the numeric labels in the first data bucket; when the number of objects of the at least one object is less than or equal to a first number, storing the second bitmap data corresponding to each object identifier as an array element in the first array, and storing the label data under the numeric labels for each object identifier as an array element in the second array; the first array and the second array contain the same... An array of subscripts corresponds to the same object identifier; the method further includes: constructing a third array corresponding to the numeric labels in the first data bucket; when the number of objects is greater than the first number and less than or equal to the second number, storing the label data of each object identifier under the numeric label as an array element in the third array; the array subscript of the third array corresponds to the second bitmap data, and the second number is the maximum number of array elements stored in the third array; the method further includes: when the space storage capacity of the third array is a fixed target storage capacity, obtaining the first number according to the target storage capacity and the space storage capacity corresponding to a single group of elements; wherein, the space storage capacity corresponding to a single group of elements is the sum of the space storage capacity corresponding to the data type of the array elements of the first array and the space storage capacity corresponding to the data type of the array elements of the second array.

2. The method according to claim 1, characterized in that, When the data type of the array elements of the first array is short integer and the data type of the array elements of the second array is integer, the space storage capacity corresponding to the single array element is the sum of the space storage capacity of the short integer data type and the space storage capacity of the integer data type; wherein, when the data type of the array elements of the third array is integer, the target storage capacity is the product of the target power of 2 and the space storage capacity of the integer data type; the target power is the number of bits of the second bitmap data.

3. The method according to claim 1, characterized in that, The bitmap data corresponding to each object identifier includes third bitmap data and fourth bitmap data; the third bitmap data corresponds to the address of the second data bucket in the database; the method further includes: determining non-numeric tags and obtaining tag data under the non-numeric tags for each object identifier; the tag data under the non-numeric tags is non-numeric; constructing a fourth array and a fifth array corresponding to the non-numeric tags in the second data bucket; when the number of objects whose tag data under the non-numeric tags matches the non-numeric tags is less than or equal to a third quantity, storing the fourth bitmap data corresponding to the object identifiers of the objects whose tag data under the non-numeric tags matches the non-numeric tags as array elements in the fourth array; when the number of objects whose tag data under the non-numeric tags matches the non-numeric tags is greater than the third quantity and the number of objects of the at least one object is less than or equal to the fourth quantity, storing the tag data under the non-numeric tags for each object identifier as array elements in the fifth array; the fourth quantity is the maximum number of array elements stored in the fifth array; the fourth bitmap data corresponding to each object identifier corresponds to the corresponding array element.

4. The method according to claim 3, characterized in that, When the data bits of the first bitmap data and the data bits of the third bitmap data are the same, the method further includes: constructing bitmap data corresponding to the non-numeric label based on the array elements of the fourth array or the fifth array; each bit in the bitmap data corresponding to the non-numeric label is used to indicate whether the object identifier satisfies the non-numeric label; obtaining the bitmap data corresponding to the numeric label based on the array elements of the second array corresponding to the numeric label according to the numeric selection requirements; and performing logical operations on the bitmap data corresponding to the numeric label and the bitmap data corresponding to the non-numeric label to obtain the target bitmap.

5. The method according to claim 4, characterized in that, The method further includes: performing a logical AND operation between the array elements of the second array corresponding to the numerical label and the bitmap data corresponding to the non-numerical label to obtain a numerical operation result corresponding to each object identifier; and performing numerical operations on the numerical operation results corresponding to each object identifier and the numerical operation requirements to obtain a final numerical operation result.

6. The method according to claim 1, characterized in that, The method further includes: compressing and serializing the array elements in the first array and the second array based on Varint encoding.

7. A data processing apparatus, characterized in that, The apparatus includes: a first acquisition unit, configured to acquire at least one object, an object identifier for each object, and bitmap data corresponding to each object identifier; the bitmap data includes first bitmap data and second bitmap data, the first bitmap data corresponding to the address of a first data bucket in a database; the object identifiers start from 0 and increment sequentially; a second acquisition unit, configured to determine numerical tags and acquire tag data under the numerical tags for each object identifier; the tag data under the numerical tags is a numerical value; a first construction unit, configured to construct a first array and a second array corresponding to the numerical tags in the first data bucket; a first storage unit, configured to, when the number of objects of the at least one object is less than or equal to a first number, store the second bitmap data corresponding to each object identifier as an array element in the first array, and store the tag data under the numerical tags for each object identifier as an array element in the second array; the first array and the second... The array elements with the same array index in the array correspond to the same object identifier; the device further includes: a second construction unit, used to construct a third array corresponding to the numerical label in the first data bucket; a second storage unit, used to store the label data of each object identifier under the numerical label as an array element in the third array when the number of objects is greater than the first quantity and less than or equal to the second quantity; the array index of the third array corresponds to the second bitmap data, and the second quantity is the maximum number of array elements stored in the third array; the device further includes: a third acquisition unit, used to acquire the first quantity according to the target storage capacity and the space storage capacity corresponding to a single group of elements when the space storage capacity of the third array is a fixed target storage capacity; wherein, the space storage capacity corresponding to a single group of elements is the sum of the space storage capacity corresponding to the data type of the array elements of the first array and the space storage capacity corresponding to the data type of the array elements of the second array.

8. An electronic device, characterized in that, include: One or more processors; A storage device having stored one or more programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-6.

Images