A data processing method, device and electronic equipment

Abstract

The present disclosure provides a data processing method, device and electronic equipment, a specific embodiment of the method includes: obtaining a plurality of filtering conditions for filtering data in a target data table and a logical relationship between the plurality of filtering conditions; determining a first value corresponding to each filtering condition in the plurality of filtering conditions, the first value corresponding to any filtering condition being determined by at least one first record obtained according to the filtering condition; performing a logical operation based on the first value and the logical relationship between the plurality of filtering conditions to determine a target record to generate a pivot table. This embodiment can simplify the process of generating a pivot table and improve the speed of generating a pivot table.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, apparatus and electronic device. Background Technology

[0002] Pivot tables are tabular analysis tools used to categorize and summarize complex data. Besides quickly performing calculations such as summation, counting, and averaging, they can also display key data as needed. The generation of pivot tables requires filtering the data in the source table; for example, finding the union or intersection of records corresponding to one or more filter criteria to obtain the filtered records. However, in application scenarios with large source data tables, this typically involves a significant amount of computation and high complexity. Therefore, an efficient data processing method is needed. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, and electronic device.

[0004] According to a first aspect, a data processing method is provided, the method comprising:

[0005] Obtain multiple filtering conditions used to filter data in a target data table and the logical relationships between the multiple filtering conditions;

[0006] Determine a first value corresponding to each of the plurality of filtering conditions; the first value corresponding to any filtering condition is obtained by converting a first binary number, the first binary number being determined by at least one first record in the target data table, the at least one first record being determined by the filtering condition.

[0007] Logical operations are performed based on the first value and the logical relationship between the multiple filtering conditions;

[0008] Based on the result of the logical operation, the target record is determined; the target record is used to generate a pivot table.

[0009] According to a second aspect, a data processing apparatus is provided, the apparatus comprising:

[0010] The acquisition module is used to acquire multiple filtering conditions for filtering data in a target data table and the logical relationships between the multiple filtering conditions.

[0011] The processing module is used to determine a first value corresponding to each of the plurality of filtering conditions; the first value corresponding to any filtering condition is obtained by converting a first binary number, the first binary number being determined by at least one first record in the target data table, and the at least one first record being determined by the filtering condition.

[0012] The processing module is further configured to perform logical operations based on the first value and the logical relationship between the multiple filtering conditions;

[0013] The processing module is further configured to determine a target record based on the result of the logical operation; the target record is used to generate a pivot table.

[0014] According to a third aspect, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed in a computer, implements the method described in any one of the first aspects.

[0015] According to a fourth aspect, an electronic device is provided, including a memory, a processor, and executable code stored in the memory, wherein the processor executes the executable code to implement the method described in any one of the first aspects.

[0016] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0017] This disclosure provides a data processing method and apparatus. First, it acquires multiple filtering conditions and their logical relationships in a target data table. Then, based on the binary number corresponding to the record determined by each filtering condition, it determines the numerical value corresponding to each filtering condition. Next, it performs logical operations on the numerical values ​​corresponding to the filtering conditions according to the logical relationships between the filtering conditions to determine the target record. This simplifies the process of further filtering records determined by the filtering conditions to generate a pivot table. In application scenarios with large source data tables, this solution can improve the speed of generating pivot tables and has higher data processing efficiency.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a data processing method in the related art disclosed herein;

[0021] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this disclosure;

[0022] Figure 3 This is a flowchart of an algorithm for determining the binary number corresponding to a numerical range, provided in an embodiment of this disclosure.

[0023] Figure 4 This is a flowchart of the algorithm for decompressing the target value provided in this embodiment of the disclosure;

[0024] Figure 5 This is a block diagram of a data processing apparatus provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic block diagram of an electronic device provided in some embodiments of this disclosure;

[0026] Figure 7 This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure;

[0027] Figure 8 This is a schematic diagram of a storage medium provided in some embodiments of this disclosure. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0029] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0032] With the continuous development of computer technology, more and more spreadsheet applications support the generation of pivot tables. To allow pivot tables to be integrated with different business processes, they can be extracted as an independent module on the product side. Typically, the product side uses an ascending array of sequence numbers to store the sequence numbers of the filtered records in the source data table. The pivot table generation engine then calculates the sequence number array to obtain the formatted data. Each business process can then integrate this formatted data into its respective view layer, facilitating further data processing by users.

[0033] In practical applications, the source data table is typically a two-dimensional table. For example, the fields of the source data table can be used as one dimension, and the records containing the field values ​​can be used as another dimension. Depending on the user's needs, a record can contain the values ​​of one or more fields from the source data table, and the sequence number of a record can be its row number or column number in the source data table. For ease of explanation, the solution proposed in this disclosure will be illustrated below using the fields of the source data table as the column dimension and the records containing the field values ​​as the row dimension. In this case, the record number is the row number of the record in the source data table. Correspondingly, when the fields are used as the row dimension and the records containing the field values ​​are used as the column dimension, a similar operation can be performed using the solution proposed in this disclosure to solve the same technical problem and achieve the same technical effect.

[0034] Figure 1 The diagram illustrates a data processing method in related technologies.

[0035] like Figure 1 As shown, the source data table undergoes two preliminary filters to obtain two sequence number arrays: Array A and Array B. Array A and Array B contain different sequence numbers. These sequence numbers can be the sequence numbers of records obtained by filtering the same field (e.g., city) of the source data table according to two different filtering conditions (e.g., city = shenzhen, city = hangzhou).

[0036] Array A stores the serial numbers 1, 3, 5, 6, 7, 8, 11, 12, and 25. Array A represents 9 records selected from the source data table based on the first filter condition (e.g., city = shenzhen), with row numbers 1, 3, 5, 6, 7, 8, 11, 12, and 25 respectively. Similarly, array B stores the serial numbers 2, 3, 5, 7, 10, 12, 14, 16, and 25. Array B represents 9 records selected from the source data table based on the second filter condition (e.g., city = hangzhou), with row numbers 2, 3, 5, 7, 10, 12, 14, 16, and 25 respectively. Each selected record includes field values ​​for multiple fields (e.g., city, country, etc.).

[0037] To generate a pivot table, two preliminary filters are performed in the source data table to obtain the required two sets of records. After storing the corresponding serial numbers of the records in a serial number array, the pivot table generation engine needs to perform calculations on the serial number array. This calculation may include iterating through serial number arrays A and B, finding the serial numbers contained within them, and then taking the union of the serial numbers contained in serial number arrays A and B to obtain serial number array C. In other words, taking the union of the rows containing the filter field values ​​(e.g., city) yields the filter rows for that field.

[0038] The sequence number array C, obtained by calculating the union of the sets, stores the sequence numbers 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 16, and 25. Based on these 13 sequence numbers in array C, the corresponding record can be determined. Then, based on the field values ​​contained in the record, the user's target pivot table is generated.

[0039] As mentioned above, because the indexes are stored in ascending order using index arrays, to perform a union operation on two index arrays containing different indices, it's necessary to compare index array A and index array B, traversing both arrays to find the different indices. If the process of generating the pivot table also requires combining index array D (the generation process of index array D is similar to that of index arrays A and B), the above process also requires pairwise comparisons of index arrays A, B, and D. Therefore, under this storage method, the time complexity of generating the pivot table is O(2^mn) = O(mn), and the space complexity is O(mn), where m is the number of indices in the index array, and n is the number of index arrays involved in the calculation.

[0040] It is not difficult to see that in application scenarios with large source data tables, such computational tasks often have extremely high complexity, making the generation of pivot tables slow and inefficient.

[0041] In view of this, the present disclosure provides a data processing scheme. First, it obtains multiple filtering conditions for filtering data in a target data table and the logical relationships between these conditions. Then, based on the binary number corresponding to the record determined by each filtering condition, it determines the value corresponding to each filtering condition. Next, it performs logical operations on the values ​​corresponding to the filtering conditions according to the logical relationships between them to determine the target record. This simplifies the process of further filtering the records determined by the filtering conditions to generate a pivot table. In application scenarios with large source data tables, this scheme can improve the speed of generating pivot tables and has higher data processing efficiency.

[0042] The following will provide a detailed description of a data processing method, apparatus, and electronic device provided in the embodiments of this disclosure, in conjunction with the above content.

[0043] Figure 2 The diagram illustrates a flowchart of a data processing method provided by an embodiment of this disclosure. The entity implementing this method can be any device, platform, server, or device cluster with computing and processing capabilities. Figure 2 As shown, the method mainly includes the following steps:

[0044] Step S201: Obtain multiple filtering conditions and logical relationships between the multiple filtering conditions used to filter data in the target data table.

[0045] The target data table can be a table created by manually adding data or downloading data from a database. For example, users can set different filter conditions to filter the data in the target data table. Specifically, different conditions and logical relationships between these conditions can be used as filter criteria to filter any field in the target data table.

[0046] For example, for a field whose value is numeric, the filter criteria could be the magnitude of the value or the range of the value. For a field whose value is text, the filter criteria could be the text content, font, color, or a combination of these individual criteria based on a set logical relationship. Depending on the user's different needs, different filter criteria can be used to filter the target data table based on its content, resulting in different filter results.

[0047] In one example, to generate a pivot table, one or more fields in the target data table that need to be filtered can be identified, as well as the various filter conditions for filtering a field multiple times or filtering each field separately.

[0048] Additionally, it's necessary to obtain the logical relationships between the various filter conditions so that the pivot table generation engine can perform one or more calculations on the corresponding sequence numbers of the records determined by each filter condition to obtain the target records. Specifically, this can be achieved by performing a bitwise AND operation on the sequence numbers to find the intersection of filter rows for different fields, or by performing a bitwise OR operation on the sequence numbers to find the union of filter rows for the same field.

[0049] Step S202: Determine the first value corresponding to each of the multiple filtering conditions. The first value corresponding to any filtering condition is obtained by converting a first binary number, which is determined by at least one first record in the target data table, and the at least one first record is determined by the filtering condition.

[0050] The first record is the record obtained by filtering the target data table according to any one of multiple filtering conditions. The first record has a corresponding sequence number in the target data table. A first identifier can be determined for at least one first record based on any filtering condition, and the corresponding sequence number of the first record in the target data table can be determined based on this first identifier. For example, the first identifier can be a sequence number or a marker associated with a sequence number (English letters, Roman numerals, etc.). It is necessary to save the sequence numbers corresponding to the records obtained after filtering according to any filtering condition.

[0051] In this embodiment, the user side stores the sequence number corresponding to each of the at least one first records by assigning each first record a digit to a binary number. Specifically, for each first record in the at least one first record, the digits in the binary number corresponding to the first identifier of each first record are determined. Based on the digits in the binary number corresponding to each first identifier, a first binary number is determined. The first binary number is then converted to obtain a first value. The first value is stored on the user side, and the first value contains the sequence number corresponding to the record obtained after filtering according to any filtering condition.

[0052] For example, depending on the user's different needs, each first identifier can correspond to one or more binary bits in the corresponding binary number. Based on the above, the sequence number of the corresponding first record in the target data table can be determined through each first identifier. If the user also needs to carry other information through each first identifier, then each first identifier can be set to correspond to multiple binary bits in the corresponding binary number, with each binary bit carrying different information. In this embodiment, only one binary bit is used as an example, where each first identifier corresponds to one binary bit in the corresponding binary number, and this binary bit corresponds to the sequence number of the first record corresponding to each first identifier. It can be understood that when each first identifier corresponds to multiple binary bits in the corresponding binary number, a similar operation can be performed to achieve the same technical effect.

[0053] For a given programming language, the first numerical value stored on the user side can be an integer or other data type. The binary number obtained by converting the first numerical value has multiple binary bits, and each binary number can correspond to at least one first identifier.

[0054] Therefore, on the user side, by saving a single value, at least one first record's sequence number can be saved. The sequence number is compressed and stored, which improves the data storage efficiency on the user side and reduces memory overhead.

[0055] For a given programming language, the number of bits that can be stored in the supported storage types is fixed. For example, in JavaScript, an integer can contain 32 bits. When the target data table has a large number of cells (e.g., 5 million cells), the number of first records determined by filtering the target data table according to any filter condition may also be very large, far exceeding the number of bits that the storage types supported by the programming language can contain. Therefore, it is necessary to store the multiple sequence numbers corresponding to the multiple first records in segments, with each segment containing a sequence number corresponding to a binary number.

[0056] In one implementation, for at least one first record determined according to any filtering condition, a first identifier corresponding to the at least one first record is determined, and the bits of the binary number corresponding to the first identifier are represented in segmented binary numbers. Each segment of binary numbers may have the same or different number of bits, obtained by data conversion from an integer or other data type in a programming language.

[0057] Specifically, the first value corresponding to any filtering condition includes at least one target value, and the first binary number obtained by converting the first value contains at least one segment of binary numbers. Each target value is obtained by converting the segment of binary numbers included in the first binary number, and each segment of binary numbers corresponds to a value range, which represents the value range of the sequence number corresponding to each first record determined by a filtering condition.

[0058] Therefore, by representing the bits of the binary number corresponding to the first identifier in the segmented binary number, the user side can save the sequence number information corresponding to any number of first records, thereby supporting more application scenarios.

[0059] Step S203: Perform logical operations based on the first value and the logical relationship between multiple filtering conditions.

[0060] In this embodiment, at least one first record is determined from the target data table based on each of multiple filtering conditions. This first record may be an initial screening result, not the final data to be displayed in the pivot table. The pivot table generation engine needs to perform one or more calculations based on this initial screening result to determine the target record. For example, the filter row for each field may be obtained by taking the union of the rows containing the filter field values, or the filter rows for all fields may be obtained by taking the intersection of the filter rows. A pivot table is then generated based on the target data for users to perform data analysis and processing.

[0061] Specifically, the process involves determining at least one target value included in the first value corresponding to each of the multiple filtering conditions, as well as determining the value range corresponding to each target value and the corresponding binary number segment. Based on the logical relationship between the multiple filtering conditions, bitwise logical operations are performed on the binary numbers corresponding to the same value range among the multiple filtering conditions to obtain a second value. The second value corresponds to a second binary number, and each target record in at least one target record corresponds to a digit in the second binary number.

[0062] Therefore, by compressing and storing the sequence numbers corresponding to the records obtained according to the filtering conditions in numerical values, and then performing bitwise logical operations on the binary numbers corresponding to different numerical values, the speed of generating pivot tables and the efficiency of data processing are improved.

[0063] Step S204: Determine the target record based on the result of the logical operation. The target record is used to generate the pivot table.

[0064] In this embodiment, the second value is the result of a logical operation. By decompressing the second value, the target record can be determined, and a pivot table can be generated based on the target record. Since the target record is obtained by further filtering based on at least one of the first records, the target record also corresponds to a digit in the second binary number.

[0065] In one approach, the sequence number of the target record can be determined directly by decompressing the second value. In another approach, the second binary number corresponding to the second value can be determined first, and then the second binary number can be decompressed to determine the sequence number of the target record.

[0066] The process of decompressing the second value to obtain the sequence number of the included record is the reverse of the process described above, which obtains the first value from the sequence number of the record determined by the filtering conditions.

[0067] Therefore, by compressing and storing the sequence numbers corresponding to the records obtained according to the filtering conditions in numerical values, performing bitwise logical operations on the binary numbers corresponding to different numerical values, and finally decompressing the results of the logical operations, the target record can be determined, simplifying the process of generating pivot tables.

[0068] The following is a schematic illustration of the solution proposed in this disclosure, using a complete application example.

[0069] One application scenario is to use the TavaScript programming language (TypeScript is an upgraded version of JavaScript) to write applications that implement the solution proposed in this disclosure in a computer or human-computer interaction device. The TavaScript programming language supports the storage method of bit enumerations, which refers to an enumeration class implemented using bit operations, with the enumeration items increasing sequentially by powers of 2.

[0070] In the TavaScript programming language, when performing bitwise operations on numbers, the value is converted from a double type to a signed int32 type. The portion of the original value exceeding 31 bits (the highest bit being the sign bit) is discarded. Therefore, an integer can only represent a maximum of 31 binary bits.

[0071] Table 1 shows a target data table provided in an embodiment of this disclosure. As shown in Table 1, Table 1 contains at least the fields id, country, and city. Specifically, the field id contains at least the value ABCD, the field country contains at least the value China, and the field city contains at least the value shanghai. Table 1 indicates the sequence number corresponding to the record containing these field values.

[0072] Table 1

[0073] Serial Number id country city … … … 0 ABCD China Shanghai … … … 1 … … Shanghai … … … 2 ABCD … … … … … 3 … China Shanghai … … … 4 ABCD China … … … … … … … … … … … 31 ABCD … … … … … 32 ABCD … … … … … … … … … … … … 34 ABCD … … … … … … … … … … … … 62 … China Shanghai … … … … … … … … … … 64 … China … … … … 65 … … Shanghai … … … 66 … China Shanghai … … … … … … … … … …

[0074] As per user requirements, for the three fields id, country, and city in the target data table, the three filter conditions are id=ABCD, country=China, and city=shanghai, respectively. These three filter conditions are related by a bitwise AND operation. This embodiment calculates the intersection of the rows containing the filter field values ​​obtained from filtering according to each of the three conditions, based on the numerical values ​​corresponding to the three filter conditions and the bitwise AND operation between them.

[0075] According to step S202 above, the next step is to determine the first value corresponding to each of the multiple filtering conditions. The first value includes at least one target value, and the first binary number obtained by converting the first value includes at least one segment of binary numbers. Each target value is obtained by converting the segment of binary numbers included in the first binary number, and each segment of binary numbers corresponds to a value range, which represents the value range of the sequence number corresponding to each first record determined by the filtering condition.

[0076] As shown in Table 1, the records determined by the three filtering criteria have the following serial numbers: 0, 2, 4, 31, 32, 34; 0, 3, 4, 62, 64, 66; and 0, 1, 3, 62, 65, 66. It is necessary to determine the numerical range of these serial numbers based on their values, identify the corresponding binary number range, and then convert the binary number into the target value.

[0077] Figure 3 The diagram shows a flowchart of an algorithm for determining the binary number corresponding to a numerical range, provided in an embodiment of this disclosure. Figure 3 As shown, this algorithm is implemented using the TavaScript programming language. It converts the indices within a numerical range into binary numbers using a left shift (<<) operation, and then performs a bitwise OR operation on these binary numbers to obtain the target binary number. The main steps include:

[0078] Step S301: Record the serial numbers that need to be saved within the numerical range.

[0079] For example, let's denote a sequence number that needs to be saved within a numerical range as row.

[0080] Step S302: Determine the range of values ​​to be saved for the serial number.

[0081] In one example, a modulo operation can be performed on the serial number to determine the numerical range where the serial number is to be stored. Taking TypeScript as an example, it can be expressed as: let offset = row % 31, startRow = row - offset, where offset is the value obtained from the modulo operation and startRow is the starting serial number of the numerical range where the serial number is to be stored.

[0082] Step S303: Determine whether any other serial numbers have been stored in this numerical range.

[0083] In one example, the target value corresponding to the numerical range where the serial number is located can be set as fieldIndexMapValue.

[0084] If the judgment result is no, go to step S304, initialize the target value fieldIndexMapValue to 0, and then execute step S305.

[0085] If the judgment result is yes, execute step S305 and save the digit corresponding to this serial number into the target value.

[0086] In one example, by performing a bitwise OR operation on 1 << offset and fieldIndexMapValue[startRow], the digit corresponding to this serial number is saved into the target value.

[0087] Taking the serial numbers 0, 2, 4, 31, 32, 34 corresponding to the records determined by filtering according to the filtering condition id = ABCD as an example, Figure 3 the algorithm shown is described.

[0088] According to the magnitude of the serial number values of each of the serial numbers 0, 2, 4, 31, 32, 34, determine the corresponding numerical ranges. Save the serial numbers 0, 2, 4 in the first numerical range (serial numbers with serial number values ∈ (0 - 31) correspond to one numerical range), and save the serial numbers 31, 32, 34 in the second numerical range (serial numbers with serial number values ∈ (32 - 63) correspond to one numerical range). If there are other serial numbers, and so on.

[0089] For each numerical range, the corresponding binary number segment can be determined according to the following algorithm.

[0090] Convert the serial numbers 0, 2, 4 into three binary numbers respectively through a left shift (<<) operation. The binary numbers can be obtained by shifting the binary number of decimal 1 by 0, 2, 4 bits respectively to get binary numbers of a certain length (the length is 32 bits, always positive by default, and the highest sign bit is 0) to represent the included serial numbers. The specific representation is as follows:

[0091] 1 << 0 = 0000 0000 0000 0000 0000 0000 0000 0001,

[0092] 1<<2=0000 0000 0000 0000 0000 0000 0100,

[0093] 1<<4=0000 0000 0000 0000 0000 0000 0001 0000,

[0094] Performing a bitwise OR operation on these three binary numbers yields a binary number representing their summation:

[0095] 1<<0|1<<2|1<<4=0000 0000 0000 0000 0000 0000 0001 0101,

[0096] As shown above, the first numerical interval corresponds to a binary number segment of 0000 0000 0000 0000 00000000 00010101. Converting this binary number yields a decimal number 21, which is then used as the target value corresponding to the binary number segment.

[0097] Similarly, for serial numbers 31, 32, and 34, according to... Figure 3 By operating the algorithm shown, the corresponding binary number for the second numerical range can be determined as 0000 0000 0000 0000 0000 0000 0000 1011, and the target value determined based on this binary number is 31.

[0098] On the user side, the above-mentioned multiple numerical ranges and the target values ​​corresponding to the numerical ranges are saved.

[0099] In one implementation, the multiple target values ​​generated above can be stored using an array. However, in some application scenarios where the distribution of the sequence numbers generated by the filtering criteria is sparse, there may be a large number of empty value intervals, meaning the target value determined by the sequence number corresponding to that interval is 0. To carry the information of the value intervals corresponding to multiple target values, it is necessary to store a large number of target values ​​for these empty intervals. For example, if the filtered sequence numbers are 0, 30999, and 61999, storing these target values ​​using an array would result in an array of length 2000 (calculating 61999 / 31≈1999.97, rounded up). Compared to the previous array with only 3 elements [0, 30999, 61999], this would actually degrade memory storage.

[0100] In another implementation, a map is used to store the multiple target values ​​generated above. The map structure is set as key: value. Taking the sequence numbers 0, 2, 4, 31, 32, and 34 determined by the filtering condition id = ABCD as an example, in the TypeScript programming language's bit enumeration structure, the map stored is: 0:21, 31:11. Here, 0 and 31 are the map keys, representing the starting sequence numbers of the value range, and 21 and 11 are the map values, representing the target values ​​obtained by converting the segmented binary numbers.

[0101] In application scenarios where the sequence number distribution is relatively sparse, although a map needs to store not only multiple target values ​​but also the starting sequence number key of the value range, using a map results in more stable memory usage performance compared to using an array.

[0102] Taking serial numbers 0, 2, 4, 31, 32, and 34 as examples, in Figure 1 The array of indices shown requires 6 integer array elements. In the bit enumeration structure using map mapping, 4 integer data points are used to store the starting index of the value range and the corresponding target value of the range. Generally speaking, using a bit enumeration structure with map mapping can optimize memory usage to the original level. In this example, The denominator 6 indicates that the sequence array contains 6 integer array elements, and the numerator 4 indicates that 4 integer data are needed to store the starting sequence number of the numerical range and the target value corresponding to the numerical range.

[0103] The target numerical set determined based on the above can be represented as:

[0104]

[0105] In the aforementioned set of target values, taking the filter condition id = ABCD as an example, the first value of this filter condition includes two target values: 21 and 11. Each target value corresponds to a value range.

[0106] Logical operations are performed based on the first value corresponding to the three filtering conditions contained in the above target value set, and the logical relationship between the three filtering conditions.

[0107] First, iterate through the target value sets with the two filtering conditions id=ABCD and country=China. Then, find the target values ​​with the same value range (0) and perform a bitwise AND operation to find their intersection. This results in a new set of target values, which can be represented as:

[0108]

[0109] Then, iterate through the filtering condition city=shanghai and the new set of target values, and take the intersection of the target values ​​with the same value interval (0) by bitwise AND to obtain the final set of target values, which can be represented as:

[0110]

[0111] The target value in the above-generated set of target values ​​is the second value. Within this set of target values, there is only one value interval (0), and the target value corresponding to this interval is 1. The second value contains only this one target value.

[0112] In this embodiment, the records obtained by filtering according to the three filtering conditions are numbered 0, 2, 4, 31, 32, 34; 0, 3, 4, 62, 64, 66; and 0, 1, 3, 62, 65, 66. Figure 1 The array of indices shown requires 18 integer array elements. In the bit enumeration structure using map mapping, 12 integer data elements are needed to store the numerical range and the corresponding target value for each range. If the bit enumeration structure using map mapping is compressed, the compression efficiency is... The performance of logical operations based on the logical relationships between multiple filtering conditions can be optimized to the original level. (k represents the compression efficiency of the sequence number), meaning the time complexity is optimized to... For the original

[0113] Because the filtered serial numbers are stored in compressed form, the calculation algorithm executed by the pivot table generation engine also needs to be adjusted. The second value needs to be decompressed to determine the records corresponding to the serial numbers contained within it.

[0114] In one implementation, the decompression operation of the target value is achieved by repeatedly executing the following steps until the target value is 0. For a specific target value, the most complex case involves repeating this process 31 times to determine the sequence number of each bit in a binary segment corresponding to the target value.

[0115] Figure 4 The diagram shows a flowchart of the algorithm for decompressing target values ​​provided in an embodiment of this disclosure. Figure 4 As shown, the main steps include:

[0116] Step S401: Determine the target value.

[0117] Step S402: Determine whether the target value is 0.

[0118] If not, proceed to step S403, where for the current i-th operation, perform a bitwise AND operation between the target value and 1.

[0119] Step S404: Determine the sequence number based on the calculation result of step S403. If the result is 1, it means that the target value includes the i-th row. Otherwise, it does not include the i-th row.

[0120] Step S405: Shift the target value one bit to the right (>>1) or divide it by 2. Then jump to step S402.

[0121] If so, the program ends.

[0122] Taking the TypeScript programming language as an example, the coding implementation process can be represented as follows:

[0123]

[0124]

[0125] In another implementation, the target value can be first converted into a binary string, and then the binary string can be iterated. The underlying principle is similar to the first implementation described above. If the i-th character is 1, it means that the target value contains the i-th row.

[0126] Taking the TypeScript programming language as an example, the coding process can be represented as follows:

[0127]

[0128] In both of the above implementations, the time complexity is O(i), where i is the largest index contained in the target value.

[0129] It should be noted that although the operations of the methods of this disclosure embodiment are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0130] Corresponding to the aforementioned data processing method embodiments, this disclosure also provides embodiments of a data processing apparatus.

[0131] Figure 5 A block diagram of a data processing apparatus provided in an embodiment of the present disclosure is shown. Figure 5 As shown, the device 500 may include an acquisition module 501 and a processing module 502.

[0132] The acquisition module 501 is used to acquire multiple filtering conditions for filtering data in the target data table and the logical relationships between the multiple filtering conditions.

[0133] Processing module 502 is used to determine a first value corresponding to each of the multiple filtering conditions. The first value corresponding to any filtering condition is obtained by converting a first binary number. The first binary number is determined by at least one first record in the target data table, and the at least one first record is determined by the filtering condition.

[0134] The processing module 502 is also used to perform logical operations based on the first value and the logical relationship between multiple filtering conditions.

[0135] The processing module 502 is also used to determine the target record based on the result of the logical operation; the target record is used to generate a pivot table.

[0136] In some implementations, the above filtering criteria are filtering criteria for any field of the target data table.

[0137] In some implementations, each of the at least one first record corresponds to a digit in a first binary number.

[0138] In some implementations, for each of the multiple filtering conditions, the processing module 502 determines the first value corresponding to that filtering condition in the following manner:

[0139] Determine the first identifier corresponding to at least one first record in the target data table that satisfies the filtering condition;

[0140] The first binary number is determined based on the first identifier corresponding to at least one first record;

[0141] The first binary number is converted to obtain the first numerical value.

[0142] In some implementations, when processing module 502 determines the first binary number based on the first identifier corresponding to at least one first record, it is used to:

[0143] Determine the digits in the binary number corresponding to each first identifier;

[0144] The first binary number is determined based on the digits in the binary number corresponding to each first identifier.

[0145] In some implementations, for any filtering condition, the first value corresponding to the filtering condition includes at least one target value, and the first binary number obtained by converting the first value includes at least one segment of binary number.

[0146] Each target value is obtained by converting a segment of binary numbers included in the first binary number. Each segment of binary numbers corresponds to a value range, which represents the value range of the sequence number corresponding to each first record determined by the filtering condition.

[0147] In some implementations, when the processing module 502 performs logical operations based on the first numerical value and the logical relationship between multiple filtering conditions, it is used to:

[0148] Determine at least one target value included in the first value corresponding to each of the multiple filtering conditions;

[0149] Determine the numerical range corresponding to each target value and the binary number corresponding to each target value;

[0150] Based on the logical relationship between multiple filtering conditions, bitwise logical operations are performed on binary numbers corresponding to the same numerical range among the multiple filtering conditions.

[0151] In some implementations, when the processing module 502 determines the target record based on the result of logical operations, it is used to:

[0152] Determine the second binary number corresponding to the result of the logical operation; the second binary number corresponds to at least one target record in the target data table;

[0153] The target record is determined based on the second binary number.

[0154] In some implementations, each target record in at least one target record corresponds to a digit in a second binary number.

[0155] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0156] Figure 6 This is a schematic block diagram of an electronic device provided for some embodiments of this disclosure. For example... Figure 6As shown, the electronic device 910 includes a processor 911 and a memory 912, and can be used to implement a client or server. The memory 912 stores computer-executable instructions (e.g., one or more computer program modules) non-transitoryly. The processor 911 executes the computer-executable instructions, which, when run by the processor 911, can perform one or more steps of the data processing method described above, thereby implementing the data processing method described above. The memory 912 and the processor 911 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0157] For example, processor 911 can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For example, the central processing unit (CPU) can be an x86 or ARM architecture. Processor 911 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 910 to perform desired functions.

[0158] For example, memory 912 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and processor 911 may run one or more computer program modules to implement various functions of electronic device 910. Various application programs and various data, as well as various data used and / or generated by the application programs, may also be stored in the computer-readable storage medium.

[0159] It should be noted that, in the embodiments of this disclosure, the specific functions and technical effects of the electronic device 910 can be referred to the description of the data processing method above, and will not be repeated here.

[0160] Figure 7This is a schematic block diagram of another electronic device provided in some embodiments of this disclosure. The electronic device 920 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. The electronic device 920 can be a terminal device, etc., and can be used to implement a client or server. The electronic device 920 can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that... Figure 7 The illustrated electronic device 920 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0161] like Figure 7 As shown, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 also stores various programs and data required for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are interconnected via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0162] Typically, the following devices can be connected to I / O interface 925: input devices 926 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 927 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 928 including, for example, magnetic tapes, hard disks, etc.; and communication devices 929. Communication device 929 allows electronic device 920 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 7 An electronic device 920 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices.

[0163] For example, according to embodiments of this disclosure, the above-described data processing method can be implemented as a computer software program. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the above-described data processing method. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by the processing device 921, the functions defined in the data processing method provided by embodiments of this disclosure can be implemented.

[0164] Figure 8 This is a schematic diagram of a storage medium provided for some embodiments of this disclosure. For example, such as... Figure 8 As shown, the storage medium 930 can be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 931. When the non-transitory computer-executable instructions 931 are executed by a processor, the data processing method described in the embodiments of this disclosure can be implemented. For example, when the non-transitory computer-executable instructions 931 are executed by a processor, one or more steps in the data processing method described above can be performed.

[0165] For example, the storage medium 930 can be used in the aforementioned electronic device, such as the storage medium 930 may include the memory in the electronic device.

[0166] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0167] For example, the description of storage medium 930 can be found in the description of the memory in the embodiments of the electronic device, and will not be repeated here. The specific functions and technical effects of storage medium 930 can be found in the description of the data processing method above, and will not be repeated here.

[0168] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, the method comprising: Obtain multiple filtering conditions used to filter data in a target data table and the logical relationships between the multiple filtering conditions; Determine the first value corresponding to each of the plurality of filtering conditions; The first value corresponding to any filtering condition is obtained by converting a first binary number, which is determined by at least one first record in the target data table, and the at least one first record is determined by the filtering condition. For any filtering condition, the first value corresponding to the filtering condition includes at least one target value, and the first binary number obtained by converting the first value includes at least one segment of binary numbers. Each target value is obtained by converting a segment of binary numbers included in the first binary number, and each segment of binary numbers corresponds to a value range, which represents the value range of the sequence number corresponding to each first record determined by the filtering condition. The at least one target value is stored using a map mapping method. Logical operations are performed based on the first value and the logical relationship between the multiple filtering conditions; Determine the second binary number corresponding to the result of the logical operation; the second binary number corresponds to at least one target record in the target data table; The target record is determined based on the second binary number; the target record is used to generate a pivot table. The filtering condition is a filtering condition for any field of the target data table; each of the at least one first record corresponds to a digit of the first binary number; each of the at least one target record corresponds to a digit of the second binary number.

2. The method according to claim 1, wherein, For each of the multiple filtering conditions, the first value corresponding to that filtering condition is determined as follows: Determine the first identifier corresponding to at least one first record in the target data table that satisfies the filtering condition; The first binary number is determined according to the first identifier corresponding to each of the at least one first record; The first binary number is converted to obtain the first numerical value.

3. The method according to claim 2, wherein, Determining the first binary number based on the first identifier corresponding to each of the at least one first record includes: Determine the digits in the binary number corresponding to each of the first identifiers; The first binary number is determined based on the number of bits in the binary number corresponding to each of the first identifiers.

4. The method according to claim 1, wherein, The logical operation based on the first numerical value and the logical relationship between the multiple filtering conditions includes: Determine at least one target value included in the first value corresponding to each of the plurality of filtering conditions; Determine the numerical range corresponding to each target value and the binary number corresponding to each target value; Based on the logical relationship between the multiple filtering conditions, bitwise logical operations are performed on the binary numbers corresponding to the same numerical range among the multiple filtering conditions.

5. A data processing apparatus, the apparatus comprising: The acquisition module is used to acquire multiple filtering conditions for filtering data in a target data table and the logical relationships between the multiple filtering conditions. The processing module is used to determine the first value corresponding to each of the plurality of filtering conditions; The first value corresponding to any filtering condition is obtained by converting a first binary number, which is determined by at least one first record in the target data table, and the at least one first record is determined by the filtering condition. For any filtering condition, the first value corresponding to the filtering condition includes at least one target value, and the first binary number obtained by converting the first value includes at least one segment of binary numbers. Each target value is obtained by converting a segment of binary numbers included in the first binary number, and each segment of binary numbers corresponds to a value range, which represents the value range of the sequence number corresponding to each first record determined by the filtering condition. The at least one target value is stored using a map mapping method. The processing module is further configured to perform logical operations based on the first value and the logical relationship between the multiple filtering conditions; The processing module is further configured to determine a second binary number corresponding to the result of the logical operation; the second binary number corresponds to at least one target record in the target data table; determine the target record based on the second binary number; the target record is used to generate a pivot table; The filtering condition is a filtering condition for any field of the target data table; each of the at least one first record corresponds to a digit of the first binary number; each of the at least one target record corresponds to a digit of the second binary number.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-4.

7. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-4.

Images