Data processing method, device, equipment and storage medium

Abstract

The present invention discloses a data processing method, apparatus, device and storage medium, relating to the field of data processing technology. The method comprises: obtaining data to be processed and a preset mark, the preset mark being used to indicate a first data structure, the first data structure comprising any one of an array, a set and a bitmap group, the bitmap group being created based on the memory occupied by the array and the memory occupied by the set; performing deduplication processing on the data to be processed and the data in the first data structure; and inserting the data to be processed into the first data structure when the data to be processed and the data in the first data structure are not repeated. The present invention solves the problems of low deduplication performance and small storage space in the data processing methods in the prior art. For data to be processed of different sizes, the data deduplication and insertion are performed accordingly in combination with the inherent characteristics of the data structure, thereby improving deduplication efficiency and storage performance.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In some application scenarios, data processing involves deduplicating received data before storage. This is currently primarily accomplished through HashSets and Bitmaps. HashSets are collections that do not allow duplicate elements, while Bitmaps are data structures. The problem with these two data processing methods is that while deduplication performance is guaranteed, deduplication efficiency and memory usage cannot be guaranteed. Conversely, while memory usage is guaranteed, deduplication performance and storage performance cannot be guaranteed. This results in an inability to achieve an effective balance in overall data processing performance, leading to low data processing performance. Summary of the Invention

[0003] The main purpose of the present invention is to provide a data processing method, device, equipment and storage medium, aiming to solve the technical problem of low data processing performance in the data processing method in the prior art.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present invention provides a data processing method, comprising:

[0006] Obtaining data to be processed and a preset tag, where the preset tag is used to indicate a first data structure, where the first data structure includes any one of an array, a set, and a bitmap group, where the bitmap group is created based on the occupied memory of the array and the occupied memory of the set;

[0007] performing deduplication processing on the data to be processed and the data in the first data structure;

[0008] When the data to be processed does not overlap with the data in the first data structure, the data to be processed is inserted into the first data structure.

[0009] Optionally, in the above data processing method, before the step of obtaining the preset mark, the method further includes:

[0010] Create arrays and collections;

[0011] Determine a first memory and a second memory, wherein the first memory is the memory occupied by the array, and the second memory is the memory occupied by the set;

[0012] determining a bitmap capacity according to the first memory and the second memory;

[0013] Determining the number of bitmaps according to the bitmap capacity;

[0014] A bitmap group is created according to the bitmap capacity and the number of bitmaps.

[0015] Optionally, in the above data processing method, the bitmap capacity is the sum of a first multiple of the first memory and a second multiple of the second memory, and the first multiple and the second multiple are in a linear relationship.

[0016] Optionally, in the above data processing method, the step of determining the bitmap capacity based on the first memory and the second memory includes:

[0017] The bitmap capacity is determined using a first calculation formula based on the first memory and the second memory; wherein the first calculation formula is:

[0018] C=8(βm+n),

[0019] Wherein, C represents the bitmap capacity, β represents the preset coefficient, m represents the first memory, and n represents the second memory.

[0020] Optionally, in the above data processing method, the step of obtaining a preset mark includes:

[0021] A preset mark is determined according to the preset parameters of the data to be processed.

[0022] Optionally, in the above data processing method, when the first data structure is an array or a set, after the step of inserting the data to be processed into the first data structure, the method further includes:

[0023] The preset mark is updated according to the amount of data in the first data structure.

[0024] Optionally, in the above data processing method, the step of updating the preset mark according to the amount of data in the first data structure includes:

[0025] When the amount of data in the first data structure reaches a first threshold, determining a second data structure according to a preset position sequence; wherein the preset position sequence includes arrays, sets, and bitmap groups arranged in sequence;

[0026] copying the data in the first data structure to the second data structure;

[0027] The preset mark is updated according to the second data structure.

[0028] Optionally, in the above data processing method, the step of updating the preset mark according to the second data structure includes:

[0029] When the amount of data in the first data structure reaches a second threshold, the preset mark is updated according to the second data structure, wherein the second threshold is greater than the first threshold.

[0030] Optionally, in the above data processing method, when the first data structure is an array, the first threshold is the product of the array capacity and a first preset ratio; when the first data structure is a set, the first threshold is the product of the set capacity and a second preset ratio; wherein the second preset ratio is smaller than the first preset ratio.

[0031] In a second aspect, the present invention provides a data processing device, comprising:

[0032] a data acquisition module, configured to acquire data to be processed and a preset mark, wherein the preset mark is used to indicate a first data structure, wherein the first data structure includes any one of an array, a set, and a bitmap group, wherein the bitmap group is created based on the occupied memory of the array and the occupied memory of the set;

[0033] a data deduplication module, configured to perform deduplication processing on the data to be processed and the data in the first data structure;

[0034] A data insertion module is configured to insert the data to be processed into the first data structure when the data to be processed does not overlap with the data in the first data structure.

[0035] In a third aspect, the present invention provides a data processing device, comprising a processor and a memory, wherein a data processing program is stored in the memory, and when the data processing program is executed by the processor, the data processing method as described above is implemented.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by one or more processors, the data processing method as described above is implemented.

[0037] The above one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:

[0038] The present invention proposes a data processing method, device, equipment and storage medium, which deduplicates the data to be processed and the data in the first data structure by obtaining the data to be processed and a preset mark indicating the first data structure. When the data to be processed and the data in the first data structure are not repeated, the data to be processed is inserted into the first data structure, thereby achieving the purpose of data deduplication and data storage; by adopting a first data structure including an array, a set and a bitmap group, data of unknown size can be processed, the data structure can be automatically determined and adjusted, and the data to be processed can be inserted into a suitable data structure to maximize performance and memory optimization; and by creating a bitmap group based on the memory occupied by the array and the memory occupied by the set, the problem of excessive degradation of deduplication performance and storage performance when the data extremes are too large is solved. The present invention performs data deduplication and insertion correspondingly for data to be processed of different sizes, combined with the inherent characteristics of the first data structure, thereby improving the deduplication efficiency and storage performance of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these provided drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a first embodiment of a data processing method according to the present invention;

[0041] Figure 2 A schematic diagram of the hardware structure of a data processing device according to the present invention;

[0042] Figure 3 This is a flow chart of a second embodiment of a data processing method according to the present invention;

[0043] Figure 4 This is a detailed flowchart of step S600 in the second embodiment of the data processing method of the present invention;

[0044] Figure 5 FIG. 1 is a schematic diagram of functional modules of a data processing device according to a first embodiment of the present invention.

[0045] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0047] It should be noted that, in the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "include..." does not exclude the presence of other identical elements in the process, method, article or system comprising the element. In addition, in the present invention, if there is a description involving "first", "second", etc., the description of "first", "second", etc. is only for descriptive purposes and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0048] In the present invention, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present invention and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used interchangeably. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In addition, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those of ordinary skill in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] An analysis of existing technologies found that in some application scenarios, the data processing process includes deduplicating the received data before storing it. For example, the server of a music playback application needs to count the information of users who use the application within a specified time. However, during the actual use of the application, the user sends user information data to the server every time he listens to a song. In this way, the server will repeatedly receive the user information data of the user within the specified time. Therefore, in order to better count the information of users who actually use the application, the server can deduplicate the data before storing the received user information data.

[0050] Among the existing methods for data processing, the main methods are implemented through sets (HashSet) and bitmaps. Among them, HashSet is a set that does not allow duplicate elements and can only store non-duplicate data. The method of processing data through HashSet specifically determines whether it is duplicate data based on the hash value of the data, and then deduplicates it before storing it. This method has good deduplication performance when processing medium-sized data volumes, but has low deduplication efficiency when facing large cardinality data volumes. At the same time, it consumes a large amount of memory space when facing smaller-scale data volumes. Bitmap is a data structure. The method of processing data through Bitmap can occupy a smaller memory space, but this method has certain errors when calculating the total number of deduplicated data. Moreover, this method is not as efficient and accurate as HashSet when deduplicating medium-sized data volumes.

[0051] At the same time, when performing data deduplication using Bitmap, there is also a problem: when the data range is too large, the deduplication performance of Bitmap will be greatly reduced, which may lead to a decrease in the amount of data that can be stored in the storage space and a decrease in storage performance. For example, for a number: 10000000, a traditional Bitmap requires at least a bit array of length 1000W, and then sets the position of the array index 1000W to 1 (where 1 represents the existence of this number and 0 represents its non-existence). Although only one number is stored, a bitmap of length 1000W needs to be initialized. This not only wastes storage space but also leads to a decrease in deduplication performance.

[0052] To address this issue, existing technologies typically use an efficient Roaring Bitmap (RBM) method. This method divides stored int numbers into high-order and low-order bits, each of which consists of 16 bits. A 65,536-bit array is stored in the high-order bits and sorted as the key, while a normal bitmap is stored in the low-order bits. However, while this method can reduce the amount of memory required for data storage, searching for a number first requires a binary search in the high-order array. This results in a worst-case scenario of 16 queries, which can reduce query performance.

[0053] Therefore, the existing data processing method of deduplicating data and then storing it, if the deduplication performance is guaranteed, cannot guarantee the deduplication efficiency and memory occupancy. If the memory occupancy is guaranteed, it cannot guarantee the deduplication performance and storage performance, resulting in the overall performance of data processing always being unable to achieve an effective balance, and there is a problem of low data processing performance.

[0054] In view of the technical problem of low data processing performance in the data processing methods in the prior art, the present invention provides a data processing method, the overall idea of ​​which is as follows:

[0055] Obtain data to be processed and a preset mark, where the preset mark is used to indicate a first data structure, where the first data structure includes any one of an array, a set, and a bitmap group, and the bitmap group is created based on the occupied memory of the array and the occupied memory of the set; deduplicate the data to be processed and the data in the first data structure; and insert the data to be processed into the first data structure when the data to be processed and the data in the first data structure do not duplicate each other.

[0056] Through the above technical solution, the purpose of data deduplication and data storage is achieved; by adopting a first data structure including arrays, sets and bitmap groups, data of unknown size can be processed, the data structure can be automatically determined and adjusted, and the data to be processed can be inserted into the appropriate data structure to maximize performance and memory optimization; and by creating a bitmap group based on the memory occupied by the array and the memory occupied by the set, the problem of excessive degradation of deduplication performance and storage performance when the data range is too large is solved. The present invention performs data deduplication and insertion correspondingly for data to be processed of different sizes, combined with the inherent characteristics of the first data structure, thereby improving the deduplication efficiency and storage performance of data processing.

[0057] The data processing method, apparatus, device and storage medium provided by the present invention are described in detail below through specific embodiments and implementation modes in conjunction with the accompanying drawings.

[0058] Example 1

[0059] Reference Figure 1 A flow chart of the present invention is provided, which proposes a first embodiment of the data processing method of the present invention, and the data processing method is applied to a data processing device.

[0060] The data processing device refers to a terminal device or network device that can achieve network connection. The device can be a terminal device such as a mobile phone, computer, tablet computer, embedded industrial computer, or a network device such as a server or cloud platform.

[0061] like Figure 2 1 is a schematic diagram of the hardware structure of a data processing device, which may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005.

[0062] Those skilled in the art will understand that Figure 2The hardware structure shown in the figure does not constitute a limitation on the data processing device of the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0063] Specifically, the communication bus 1002 is used to implement connection and communication between these components;

[0064] The user interface 1003 is used to connect to the client and perform data communication with the client. The user interface 1003 may include an output unit, such as a display screen, an input unit, such as a keyboard, and optionally, the user interface 1003 may also include other input / output interfaces, such as a standard wired interface or a wireless interface.

[0065] The network interface 1004 is used to connect to the backend server and perform data communication with the backend server. The network interface 1004 may include an input / output interface, such as a standard wired interface or a wireless interface, such as a Wi-Fi interface.

[0066] The memory 1005 is used to store various types of data. These data may include, for example, instructions for any application or method in the data processing device, as well as application-related data. The memory 1005 may be a high-speed RAM memory or a stable memory such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the processor 1001.

[0067] For details, please refer to Figure 2 , the memory 1005 may include an operating system, a network communication module, a user interface module and a data processing program, wherein the network communication module is mainly used to connect to the server and perform data communication with the server;

[0068] The processor 1001 is configured to call the data processing program stored in the memory 1005 and perform the following operations:

[0069] Obtaining data to be processed and a preset tag, where the preset tag is used to indicate a first data structure, where the first data structure includes any one of an array, a set, and a bitmap group, where the bitmap group is created based on the occupied memory of the array and the occupied memory of the set;

[0070] performing deduplication processing on the data to be processed and the data in the first data structure;

[0071] When the data to be processed does not overlap with the data in the first data structure, the data to be processed is inserted into the first data structure.

[0072] Based on the above data processing equipment, the following Figure 1The flowchart shown in the figure describes the data processing method of this embodiment in detail. This method can be applied to a hardware server or a cloud server. For example, it can be used in an operation analysis platform, such as a music player application operation analysis platform, to perform funnel analysis and retention analysis, and other data processing processes to deduplicate and store the received data. The method may include the following steps:

[0073] Step S100: Obtain data to be processed and preset marks.

[0074] The preset mark is used to indicate a first data structure, which includes any one of an array (Array), a set (HashSet) and a bitmap group (Bitmap group), and the bitmap group is created based on the occupied memory of the array and the occupied memory of the set.

[0075] Specifically, the data to be processed can be numbers or strings, such as user usage data or user ID (Identity Document) data. The total amount of data to be processed can be of any size. Compared to existing technologies, there is no need to limit the amount of data to be processed, nor is it necessary to know the amount of data to be processed.

[0076] Specifically, after establishing an array and a set, the capacity of each bitmap in the bitmap group is determined based on the memory occupied by the array and the memory occupied by the set. Then, based on the hardware configuration information of the device, the maximum total amount of data that can be processed by the device is determined, that is, the maximum value of the to-be-processed data that can be obtained. The number of bitmaps in the bitmap group is determined based on the capacity of each bitmap. Finally, a bitmap group is established based on the number of bitmaps and the bitmap capacity, wherein the total capacity of the array, set, and bitmap group increases in sequence.

[0077] The preset mark can be set. For example, when the data to be processed is just obtained and needs to be stored for the first time, the first data structure indicated by the preset mark is directly specified as a set or a bitmap group; the preset mark can also be obtained directly through initialization. For example, when the data to be processed is just obtained and needs to be stored for the first time, the preset mark is directly initialized. At this time, the first data structure indicated by the preset mark is an array with the minimum capacity; the preset mark can also be obtained by judging certain information. For example, the first data structure indicated by the corresponding preset mark can be determined as an array, a set or a bitmap group based on the specific type of the data to be processed.

[0078] The preset mark can determine whether the first data structure indicated by the corresponding preset mark is an array, a set or a bitmap group according to the data volume of the data to be processed, which can be specifically divided into two cases.

[0079] The first case is that if the data volume scale of the data to be processed is known, the specific size does not need to be known. In this case, the preset mark can be determined according to the data volume scale of the data to be processed. For example, when the data volume is small, the first data structure indicated by the corresponding preset mark is an array. When the data volume is medium, the first data structure indicated by the corresponding preset mark is a set. When the data volume is large, the first data structure indicated by the corresponding preset mark is a bitmap group. In specific implementation, different data volume ranges and their corresponding preset marks can also be set according to actual conditions. The above situation is for the case where the data volume scale of the data to be processed is known, that is, it is known whether the data volume of the data to be processed is small, medium or large.

[0080] The second situation is that when faced with a situation where the amount of data to be processed is unknown, since the amount of data to be processed is unclear, the array can be set to the first data structure first, and the steps of this method can be repeated until the array can no longer be inserted, and then the data that has been inserted into the array is copied to the set, and the steps of this method are repeated to insert the remaining data to be processed into the set, until the set can no longer be inserted, and then the data that has been inserted into the set is copied to the bitmap group. The data to be processed must be completely inserted into the bitmap group because the total amount of data in the bitmap group is equal to the total amount of data to be processed that can be received. First insert the data to be processed into the array. If the amount of data to be processed is small, it can be directly inserted into the array. There is no need to copy it to other data structures later. Compared with directly inserting data into a set or bitmap group, the data structure of the array itself does not take up too much space, and the deduplication efficiency is high, which is especially suitable for data to be processed with a smaller data size. Correspondingly, when the array cannot accommodate the data to be processed, it means that the data size of the data to be processed is larger than the total capacity of the array. At this time, the data already inserted into the array is copied to the set, and the subsequent uninserted data is continued to be inserted into the set. If all the data to be processed can be directly inserted into the set, there is no need to copy the data to the bitmap group. Compared with the method of directly inserting the data into the bitmap group, it does not take up too much storage space. However, in reality, the total amount of data to be processed may be larger than the total capacity of the set. Therefore, when the set cannot accommodate the data to be processed, the data already inserted into the set is continued to be copied to the bitmap group, and the subsequent uninserted data is also inserted into the bitmap group until all the data to be processed has been inserted into the bitmap group, completing the processing of the unknown size and large amount of data to be processed.

[0081] Step S300: Deduplication is performed on the data to be processed and the data in the first data structure.

[0082] Before inserting the data to be processed into the first data structure indicated by the preset marker, the data to be processed must first be deduplicated. Specifically, the data to be processed must be compared with the data in the first data structure. If they are consistent, processing of the current data to be processed can be terminated and the next data to be processed can be acquired. If they are inconsistent, the next step can be performed. It should be noted that when performing the data comparison in this step, if the data to be processed is acquired for the first time, there is no data to be inserted in the array at this time, and this step can be skipped and the next step can be performed.

[0083] Specifically, deduplication can be performed according to a binary search method. When the data to be processed is not just one set of data, but multiple sets of data, such as multiple user IDs, the data in the first data structure can be searched to see whether there is duplicate data in any set of data in the multiple sets of data. If so, the corresponding data in the data to be processed can be deleted, that is, not stored in the first data structure, and the remaining non-duplicate data, that is, the deduplicated data to be processed, can be stored in the first data structure. If not, all the data of the data to be processed can be saved and stored in the first data structure.

[0084] Regardless of whether the first data structure into which the data to be processed is to be inserted is an array, a set, or a bitmap group, this step requires deduplication of all the data to be processed. For example, for data with a small amount of data, it is known that the first data structure indicated by the preset data is an array, and all the data can be directly obtained at once as the data to be processed. By executing this method only once, all the deduplicated data to be processed can be stored in the array; for data with a medium amount of data, it is known that the first data structure indicated by the preset data is a set, and a portion of the data can be obtained each time as the data to be processed until all the data are stored in the set; for data with a large amount of data, it is known that the first data structure indicated by the preset data is a bitmap group, and similarly, a portion of the data can be obtained each time as the data to be processed until all the data are stored in the bitmap group. Therefore, this embodiment can make full use of the characteristics of each data structure itself. For example, when the data amount is small, the array has higher efficiency and accuracy; when the data amount is medium, the set has higher efficiency and accuracy; when the data amount is large and there is an extreme difference, the bitmap group can improve storage performance and bit operation performance.

[0085] The data processing method provided by this embodiment deduplicates the data to be processed and the data in the first data structure by obtaining the data to be processed and a preset mark indicating the first data structure. When the data to be processed and the data in the first data structure are not repeated, the data to be processed is inserted into the first data structure, thereby achieving the purpose of data deduplication and data storage. By adopting a first data structure including an array, a set, and a bitmap group, data of unknown size can be processed, the data structure can be automatically determined and adjusted, and the data to be processed can be inserted into a suitable data structure to maximize performance and memory optimization. The problem of excessive degradation of deduplication performance and storage performance when the data extremes are too large is also solved by creating a bitmap group based on the occupied memory of the array and the occupied memory of the set. The present invention performs data deduplication and insertion for data to be processed of different sizes, combined with the inherent characteristics of the first data structure, thereby improving the deduplication efficiency and storage performance of data processing.

[0086] Example 2

[0087] Based on the same invention concept, Figures 3 and 4 , a second embodiment of the data processing method of the present invention is proposed, which is applied to a data processing device.

[0088] The following combination Figure 3 The data processing method of this embodiment is described in detail in the flowchart shown in FIG. The method may include the following steps:

[0089] Step S100: Obtain data to be processed and preset marks.

[0090] Furthermore, step S100 may include:

[0091] Step S110: Obtain data to be processed.

[0092] Specifically, the data to be processed can be numbers or strings, such as user usage data or user ID (Identity Document) data. The total amount of data to be processed can be of any size, and there is no need to limit or know the size of the acquired data to be processed.

[0093] It should be noted that after the step of obtaining the data to be processed, the following step S120 can be executed in sequence to achieve the purpose of creating multiple data structures based on the obtained data to be processed and then processing the data to be processed; step S120 can also be an independently executed step branch, so as to achieve the pre-establishment of multiple general data structures, and after obtaining the data to be processed, these data structures are directly called to achieve the purpose of processing the data to be processed.

[0094] Step S120: creating arrays, sets and bitmap groups;

[0095] The bitmap group is created based on the memory occupied by the array and the memory occupied by the set.

[0096] Specifically, the bitmap group may be a group containing multiple bitmaps, all of which have the same capacity. At the same time, the bitmaps do not have any intersection, that is, all the deduplicated data to be processed may be obtained based on the union of each bitmap.

[0097] Step S120 may include:

[0098] Step S121: Create arrays and collections;

[0099] Step S122: determining a first memory and a second memory, wherein the first memory is the memory occupied by the array, and the second memory is the memory occupied by the set;

[0100] Step S123: determining a bitmap capacity according to the first memory and the second memory;

[0101] Specifically, the bitmap capacity is the sum of a first multiple of the first memory and a second multiple of the second memory, and the first multiple and the second multiple are in a linear relationship.

[0102] In a specific implementation, the bitmap capacity may be determined using a first calculation formula based on the first memory and the second memory; wherein the first calculation formula is:

[0103] C=8(βm+n),

[0104] Wherein, C represents the bitmap capacity, β represents the preset coefficient, m represents the first memory, and the unit is Byte, and n represents the second memory, and the unit is Byte.

[0105] The bitmap capacity C is the initial capacity of each bitmap in the bitmap group, and the preset coefficient β can be set manually according to actual needs; the memory occupied by the array, i.e., the first memory m, is consistent with the memory size occupied by the program or virtual machine that performs data deduplication in the device, and can be adjusted according to actual needs. It should be noted that if, in the actual implementation process, it is impossible to predetermine the memory occupied by the array and the memory occupied by the collection, 1 / 8192 of the JVM heap memory can be determined as the memory occupied by the collection, i.e., the second memory n, and the memory occupied by the array, i.e., the first memory m, can be set to 512Bytes. In this way, since 1 / 8192 of the memory is small, it will not occupy too many resources, thereby saving resources; on the other hand, since 512Bytes can store 64 numbers, only 6 searches are required in the subsequent deduplication binary search process, which is 16 times less than the existing technology, and can reduce the number of searches by more than half, thereby saving search resources.

[0106] In this embodiment, the user ID is used as an example of the data to be processed. In this step, the bitmap capacity C is initialized. For example, assuming that the number in the data is 64 bits, one number is 8 bytes (8Byte), and the memory of the data processing device is set to 2G, the preset coefficient β can be set to 1 / 8192 by default, and the preset coefficient β can also be adjusted appropriately according to the JVM heap memory. Here, the preset coefficient β is set to 1, and the memory occupied by the array, that is, the first memory m, is 512. Then the memory occupied by the set, that is, the second memory n=1024*1024 / 4, can be calculated to obtain the capacity of each bitmap as follows:

[0107] Step S124: Determine the number of bitmaps according to the bitmap capacity.

[0108] Specifically, the calculation formula for the number of bitmaps is:

[0109]

[0110] Where N represents the number of bitmaps in the bitmap group, U max Indicates the maximum value of the available data set to be processed.

[0111] In this embodiment, after determining the bitmap capacity C, the number of bitmaps N in the bitmap group can be further determined based on the capacity of each bitmap. It should be noted that the "data set to be processed" refers to the set consisting of all data to be processed. For example, if the data set to be processed is {1000000000, 1000000001}, where the maximum value is 1000000001, we obtain C = 2101248. Based on the above formula for calculating the number of bitmaps, we can calculate N = 1000000001 / 2101248 = 476.

[0112] Step S125: creating a bitmap group according to the bitmap capacity and the number of bitmaps.

[0113] Create a bitmap group based on the bitmap capacity C and the number of bitmaps N. After creating the bitmap group, each bitmap in it will record a starting value. At the same time, each bitmap is lazy loaded. If a value in a certain range does not exist, the bitmap will not be initialized, so as not to waste memory. Assuming that the calculated bitmap capacity C = 100W, the starting value of the bitmap can be recorded. For example, the starting value of bitmap1 can be considered to be 0, and the starting value of bitmap10 can be considered to be 900W.

[0114] In this embodiment, after determining the bitmap capacity C = 2101248 and the number of bitmaps N = 476, the data to be processed can be continued to be traversed. Taking the case of storing the data to be processed {10000000000, 1000000001} in the bitmap group as an example, 1000000000 is represented as, the 1907200th bit of bitmap476 is set to 1 (1 represents the existence of this number, 0 represents non-existence), and 1000000001 is represented as, the 1907201st bit of bitmap476 is set to 1 (1 represents the existence of this number, 0 represents non-existence). Since the data is only in bitmap476, only bitmap476 needs to be initialized, and other bitmaps do not need to be initialized. This can save a lot of memory, thereby improving bitmap performance. Through the data structure of the bitmap group, the storage and bit operation performance can be greatly improved.

[0115] Step S130: Obtain a preset tag, wherein the preset tag is used to indicate a first data structure, and the first data structure includes any one of an array, a set, and a bitmap group.

[0116] The number of preset marks corresponds to the number of data structures. Here, there are three data structures, and three different numerical values ​​or symbols can be set as preset marks. For example, the preset mark flag can be an integer value starting from 0 and adding 1 in sequence, or it can be a user-defined numerical value or character representing multiple different levels, which can be set according to the number or category of the data structure. For example, when the data structure includes three: array, set and bitmap group, the preset mark flag can be set to three different numerical values ​​0, 1, and 2. Then, based on the three preset mark flags and the three data structures, a correspondence between the preset mark and the data structure is established, so that different preset marks indicate different data structures. For example, when the preset mark flag is the first variable value 0, the first data structure indicated is an array; when the preset mark flag is the second variable value 1, the first data structure indicated is a set; when the preset mark flag is the third variable value 2, the first data structure indicated is a bitmap group.

[0117] Specifically, corresponding to the process of obtaining the data to be processed for the first time, if the data to be processed is of unknown data size and unknown preset parameters, the preset mark can be directly initialized, and an initial first data structure can be determined first, and the data to be processed can be tried to be inserted into the first data structure. For example, the initialized preset mark can directly indicate an array, and an initial array is determined first. Then, based on the amount of inserted data, it is determined whether a first data structure with a larger storage capacity needs to be re-determined. If necessary, after executing the subsequent steps of this process, the preset mark can be updated to change the first data structure to be inserted, so that when the data to be processed of the next process is obtained, the data to be processed can be inserted into the re-determined first data structure, until a final data structure for storing data is determined, so that all the deduplicated data to be processed can be inserted into the final data structure for storage.

[0118] Taking the example of this embodiment, in the case where the data size of the data to be processed is unknown, the preset mark can be started from 0 after initialization, that is, the data to be processed is first set to be inserted into the array; if after subsequent steps and repeated processes, the amount of data stored in the array is full, the preset mark flag can be adjusted to 1, and the data to be processed can be reinserted into the collection; if the collection is also full, the preset mark flag can be adjusted to 2, and the data to be processed can be reinserted into the bitmap group, so that the data to be processed of unknown size can be inserted into the most appropriate data structure.

[0119] By presetting tags, we can distinguish the data of different sizes to be processed, and the corresponding first data structure to be inserted. By traversing each data structure, the data to be processed can be inserted into the most appropriate data structure, thereby improving the deduplication efficiency.

[0120] In one embodiment, step S130 may include:

[0121] Step S131: Determine a preset mark according to preset parameters of the data to be processed.

[0122] Among them, the preset mark can be determined according to the data volume of the data to be processed, and the preset mark can also be determined according to the preset parameters of the data to be processed. The preset parameters are data types or data types, etc. For example, according to the data types, they are strings, numbers, and texts, and the corresponding preset marks are determined to indicate arrays, sets, and bitmap groups respectively. For example, according to the data types, they are text data, audio data, and video data, and the corresponding preset marks are determined to indicate arrays, sets, and bitmap groups respectively.

[0123] The preset mark flag can start with a certain user-defined value. For example, when the user already knows that the amount of data to be processed is definitely greater than the capacity of the array, the preset mark flag can be directly determined to be 1. Correspondingly, the data to be processed can be directly inserted into the set without having to test whether the data to be processed is greater than the capacity of the array, which can save processes. For example, when the data to be processed is a user ID, the amount of data to be processed may generally be only 20M. Under normal circumstances, it is sufficient to store it in a set. Then the preset mark can be directly determined to be 1, indicating that the first data structure is a set. Subsequent deduplication and storage are only performed on this set. If the set cannot store it, the preset mark is changed to 2 to store the data to be processed in the subsequent process in the bitmap group.

[0124] Step S300: Deduplication is performed on the data to be processed and the data in the first data structure.

[0125] When the preset mark is obtained in step S130, it means that it is known which data structure the current data to be processed is about to be inserted into. Correspondingly, it can be queried whether data has been stored in the data structure. If so, the stored data can be called and compared with the current data to be processed. If the data to be processed exists in the stored data, the data to be processed is removed and not inserted into the data structure. If the data to be processed does not exist in the stored data, the data to be processed can be inserted into the data structure, that is, the data to be processed is retained and waits for insertion. If no data is stored in the data structure, it means that the current data to be processed may be the first process of the entire processing process, then this step can be skipped and the next step can be executed directly, that is, the data to be processed is directly inserted into the data structure.

[0126] Deduplication of the data to be processed and the data in the first data structure can fully combine the characteristics of each data structure itself. For example, array deduplication is more efficient and accurate when the data volume is small; set deduplication through hash value is more efficient and accurate when the data volume is medium; bitmap group deduplication can improve storage performance when the data volume is large and there are extreme differences.

[0127] Step S500: When the data to be processed does not overlap with the data in the first data structure, insert the data to be processed into the first data structure.

[0128] Specifically, in the process of deduplicating the data to be processed and the data in the first data structure, the data to be processed that is duplicated with the data in the first data structure will be deleted, while the data to be processed that is not duplicated with the data in the first data structure will be retained. At this time, step S500 can be executed to insert the data to be processed into the first data structure.

[0129] In this embodiment, assuming that the data to be processed is multiple user IDs, the preset flag is initialized to 0. Then, the first user ID is directly inserted into the array by default. Therefore, there is no data in the array at this time, and there is no need to deduplicate the first user ID. Then, the second user ID is obtained. At this time, the preset flag is still 0. According to the preset flag, it is determined that the array is currently used to store the second user ID. Then, the second user ID and the data stored in the array are deduplicated. If there is no duplication, the second user ID is inserted into the array. If there is a duplication, the second user ID is directly deleted. This cycle is repeated for each user ID obtained, and the storage is completed. The data stored in the data structure is thus deduplicated data.

[0130] Since the data to be processed is duplicated, it may not be possible to determine at the beginning how large the data to be stored is and which data structure is most suitable. Therefore, you can try them one by one in the order of arrays, sets, and bitmap groups. For example, when you first use array storage, you find that the array is almost full, but there is still data to be processed. In this case, you can switch to set storage. When you find that the set is also almost full, you can switch to bitmap groups. In this way, when the data to be processed is a relatively small amount of data, there is no need to use sets or bitmap groups to achieve data deduplication and storage, and it does not take up a lot of memory space, and the deduplication efficiency is faster. Correspondingly, when the data to be processed is a relatively large amount of data, the final deduplicated data is stored in the most suitable data structure in sequence. There is no need to worry about the situation where the set can hold all the deduplicated data to be processed, but it is still extravagant to use bitmap groups to store the data. This saves storage space of the data structure and improves storage performance.

[0131] Furthermore, since the total amount of the data to be processed is unclear, when inserting the data to be processed into the first data structure, it can be inserted in the order of an array with a small capacity and a small memory footprint, a set with a medium capacity and a medium memory footprint, and a bitmap group with a large capacity and a large memory footprint. Furthermore, since the total amount of the data to be processed may exceed the first data structure determined at the beginning, it is necessary to tentatively insert the data to be processed into the array, set, and bitmap group in sequence. In other words, the final data structure will change according to the specific amount of data to be processed after deduplication. Correspondingly, for the case where the first data structure is an array or a set, a trial data insertion can be performed. That is, when the first data structure is an array or a set, after the step of inserting the data to be processed into the first data structure, the method can further include:

[0132] Step S600: updating the preset mark according to the amount of data in the first data structure.

[0133] In one embodiment, Figure 4 In the detailed flow diagram shown, step S600 may include:

[0134] Step S610: determining whether the data volume of the first data structure reaches a first threshold;

[0135] Specifically, when the first data structure is an array, the first threshold is the product of the array capacity and a first preset ratio; when the first data structure is a set, the first threshold is the product of the set capacity and a second preset ratio; wherein the second preset ratio is smaller than the first preset ratio.

[0136] To ensure the integrity of the next data structure when data needs to be inserted into it, the data of the first data structure that is currently full or almost full can be copied to the next data structure, and the remaining uninserted data to be processed can be inserted into the next data structure. To ensure sufficient time for the data of the first data structure to be copied to the next data structure, backup can be initiated when the data inserted into the first data structure does not reach the total capacity of the first data structure, that is, when a first threshold is reached. Therefore, the first threshold can be set as a judgment condition for starting data backup.

[0137] Since the total capacity of the array is smaller than the total capacity of the set, the amount of data inserted into the array will also be smaller than the amount of data inserted into the set. Therefore, the time to copy the data of the array to the set will be smaller than the time to copy the data of the set to the bitmap group. Therefore, the second preset ratio is smaller than the first preset ratio.

[0138] In this embodiment, the array capacity is the same as the memory occupied by the array. Correspondingly, when the first data structure is an array, the first threshold T1 can be set to n*90%. The set capacity is the same as the memory occupied by the set. Correspondingly, when the first data structure is a set, the first threshold T2 can be set to m*70%.

[0139] Step S620: When the amount of data in the first data structure reaches a first threshold, determining a second data structure according to a preset position sequence; wherein the preset position sequence includes arrays, sets, and bitmap groups arranged in sequence;

[0140] When the amount of data inserted into the first data structure does not reach the first threshold, the process may directly return to the step of obtaining data to be processed to continue inserting the remaining data to be processed that has not been inserted into the first data structure into the first data structure.

[0141] Specifically, when the first data structure is an array, if the amount of data inserted into the array reaches the first threshold T1, the second data structure is determined to be a set; when the first data structure is a set, if the amount of data inserted into the set reaches the first threshold T2, the second data structure is determined to be a bitmap group.

[0142] Step S630: copying the data of the first data structure to the second data structure;

[0143] In order to ensure the integrity of the second data structure, during the process of copying data, the data inserted into the first data structure by the next process will continue to be backed up to the second data structure until the first data structure is full and no more data can be inserted. It is determined that the first data structure is not suitable for the amount of data to be processed. At this time, the data in the first data structure that is full and has been backed up to the second data structure can be directly discarded. For example, the first data structure can be directly initialized and the data therein can be cleared.

[0144] Specifically, when the first data structure is an array, the data inserted into the array is copied to the set, and the process returns to step S110, where the remaining data to be processed that has not been inserted into the array is further deduplicated and then inserted into the array. When the first data structure is a set, the data to be inserted into the set is copied to the bitmap group, and the process returns to step S110, where the remaining data to be processed that has not been inserted into the set is further deduplicated and then inserted into the set.

[0145] Step S640: Update the preset mark according to the second data structure.

[0146] In one embodiment, the preset mark can be directly updated according to the second data structure, and subsequent data to be processed can be inserted into the second data structure, which can ensure that there will be no other factors affecting the copying of data from the first data structure to the second data structure.

[0147] In another embodiment, step S640 may include:

[0148] Step S641: determining whether the amount of data in the first data structure reaches a second threshold;

[0149] The second threshold is greater than the first threshold.

[0150] Specifically, when the first data structure is an array, the second threshold is the total capacity of the array, that is, the memory occupied by the array m; when the first data structure is a set, the second threshold is the total capacity of the set, that is, the memory occupied by the set n, where m<n.

[0151] When the amount of data inserted into the first data structure reaches a first threshold, the remaining data to be processed can continue to be inserted into the first data structure, which can avoid the situation where there is not much remaining data and all of it can be inserted into the first data structure, and the second data structure that occupies more memory is used.

[0152] However, if the first data structure is full when the remaining data to be processed is continuously inserted, the second data structure still needs to be used for data deduplication and data storage to ensure data integrity. Therefore, the second threshold can be directly set to the total capacity of the first data structure.

[0153] Step S642: When the amount of data in the first data structure reaches a second threshold, updating the preset flag according to the second data structure;

[0154] When the amount of data inserted into the first data structure reaches a second threshold, the preset mark can be updated to indicate the second data structure, so as to determine the second data structure as the first data structure of the new process. Correspondingly, repeating this method can insert the remaining uninserted data to be processed into the new first data structure, that is, the second data structure.

[0155] When the amount of data inserted into the first data structure does not reach the second threshold, the process returns to the step of obtaining the data to be processed to insert the remaining data to be processed that has not been inserted into the first data structure into the first data structure.

[0156] Specifically, when the first data structure is an array and the second data structure is a set, if the amount of data inserted into the array reaches m, at this time, the preset mark can be updated to 1, and the set will be used as the first data structure in the new round of data processing flow, and steps S300 and S500 will be executed to insert the remaining data to be processed into the set; this situation is similar to the first data structure determined to be a set according to the data volume of the data to be processed, such as when the preset mark is the second variable value 1, the data deduplication and insertion are performed after determining that the first data structure is a set. The only difference is that one is that part of the data has been backed up from the array, and only the remaining data needs to be inserted into the set, and the other is that the deduplicated data to be processed needs to be directly inserted into the set; when the first data When the first data structure is a set and the second data structure is a bitmap group, if the amount of data inserted into the set reaches n, the preset mark can be updated to 2, and the bitmap group will be used as the first data structure in the new round of data processing flow, and steps S300 and S500 will be executed to insert the remaining data to be processed into the bitmap group; this situation is similar to the first data structure determined as a bitmap group based on the data volume of the data to be processed, for example, when the preset mark is the third variable value 2, the first data structure is determined to be a bitmap group and then data deduplication and insertion are performed. The only difference is that one is that part of the data has been backed up from the set, and only the remaining data needs to be inserted into the bitmap group, and the other is that the deduplicated data to be processed needs to be directly inserted into the bitmap group.

[0157] Specifically, after step S500, the method may further include:

[0158] Step S710: determining whether all the data to be processed are inserted into the first data structure;

[0159] Step S720: If the data to be processed is not all inserted into the first data structure, execute step S620;

[0160] Specifically, if all the data to be processed are inserted into the first data structure, the current process is terminated. If the data to be processed is a group of data and is processed in multiple processes, after any group of data is inserted into the first data structure, it can be determined whether all the groups of data to be processed have been inserted; if so, the entire process can be terminated; if not, the remaining data to be processed can be further grouped and processed for deduplication and storage, or subsequent steps can be performed, such as step S620. If the data to be processed is multiple groups of data, it can be determined whether all the data to be processed have been inserted in the process of inserting the data to be processed into the first data structure; if so, the current process or the entire process can be terminated; if not, the remaining data to be processed can be continued to be inserted into the first data structure.

[0161] The following describes specific implementation methods in combination with specific situations.

[0162] In one embodiment, when the first data structure indicated by the preset mark is an array, the method may specifically include:

[0163] Step A1: obtaining data to be processed and a preset mark, where the preset mark is 0, indicating that the first data structure is an array;

[0164] Step A2: When the data to be processed does not overlap with the data in the array, insert the data to be processed into the array;

[0165] Step A3: Determine whether the amount of data inserted into the array reaches a first threshold value T1, where the first threshold value T1 is the product of the array capacity and a first preset ratio;

[0166] Step A4: If the amount of data inserted into the array does not reach the first threshold value T1, return to step A1 and continue to insert data that does not duplicate the data in the array into the array;

[0167] Step A5: If the amount of data inserted into the array reaches a first threshold value T1, determining that the second data structure is a set;

[0168] Step A6: Copy the data inserted into the array to the collection, and return to step A1 to continue inserting the remaining data that does not duplicate the data in the array into the array;

[0169] Step A7: Determine whether the amount of data inserted into the array reaches a second threshold value T3, where the second threshold value T3 is the array capacity;

[0170] Step A8: If the amount of data inserted into the array does not reach the second threshold T3, return to step A1;

[0171] Step A9: If the amount of data inserted into the array reaches the second threshold value T3, the preset flag is set to 1 according to the set update, and the remaining data to be processed that does not overlap with the data in the set is inserted into the set;

[0172] Step A10: Determine whether the amount of data inserted into the collection reaches a first threshold value T2, where the first threshold value T2 is the product of the collection capacity and a second preset ratio;

[0173] Step A11: If the amount of data inserted into the set does not reach the first threshold T2, return to step A9 and continue inserting data that does not duplicate data in the set into the set;

[0174] Step A12: If the amount of data inserted into the set reaches the first threshold T2, determining that the third data structure is a bitmap group;

[0175] Step A13: copy the data to be inserted into the set to the bitmap group, and return to step A9 to continue inserting the remaining data that does not duplicate the data in the set into the set;

[0176] Step A14: Determine whether the amount of data inserted into the collection reaches a second threshold value T4, where the second threshold value T4 is the collection capacity;

[0177] Step A15: If the amount of data inserted into the set does not reach the second threshold T4, return to step A9;

[0178] Step A16: If the amount of data inserted into the set reaches the second threshold T4, the bitmap group is updated with a preset flag of 2, and the remaining data to be processed that does not overlap with the data in the bitmap group is inserted into the bitmap group; the process ends when all the data to be processed are inserted into the bitmap group.

[0179] In another embodiment, when the first data structure indicated by the preset mark is a set, the method may specifically include:

[0180] Step B1: obtaining data to be processed and a preset mark, where the preset mark is 1, indicating that the first data structure is a set;

[0181] Step B2: When the data to be processed does not overlap with the data in the collection, insert the data to be processed into the collection;

[0182] Step B3: Determine whether the amount of data inserted into the collection reaches a first threshold value T2, where the first threshold value T2 is the product of the collection capacity and a second preset ratio;

[0183] Step B4: If the amount of data inserted into the set does not reach the first threshold value T2, return to step B1 and continue to insert data that does not duplicate the data in the set into the set;

[0184] Step B5: If the amount of data inserted into the set reaches the first threshold T2, determining that the second data structure is a bitmap group;

[0185] Step B6: copy the data to be inserted into the set to the bitmap group, and return to step B1 to continue inserting the remaining data that does not duplicate the data in the set into the set;

[0186] Step B7: Determine whether the amount of data inserted into the collection reaches a second threshold value T4, where the second threshold value T4 is the collection capacity;

[0187] Step B8: If the amount of data inserted into the set does not reach the second threshold T4, return to step B1;

[0188] Step B9: If the amount of data inserted into the set reaches the second threshold T4, the bitmap group is updated with a preset flag of 2, and the remaining data to be processed that does not overlap with the data in the bitmap group is inserted into the bitmap group; this process ends when all the data to be processed are inserted into the bitmap group.

[0189] In another embodiment, when the first data structure indicated by the preset mark is a bitmap group, the method may specifically include:

[0190] Step C1: Obtaining data to be processed and a preset flag, where the preset flag is 2, indicating that the first data structure is a bitmap group;

[0191] Step C2: When the data to be processed does not overlap with the data in the bitmap group, insert the data to be processed into the bitmap group; and the process ends when all the data to be processed are inserted into the bitmap group.

[0192] For more implementation details of the specific implementation of the above method steps, please refer to the description of the specific implementation in Example 1. For the sake of brevity of the description, they will not be repeated here.

[0193] The data processing method provided in this embodiment proposes a high-performance data processing method for data of varying sizes. This method eliminates the need to predetermine the size of the data to be processed or set a specific data structure. Instead, the method automatically adjusts the data structure based on the data volume, maximizing performance and memory usage. Furthermore, the proposed novel bitmap group structure addresses the significant degradation in bitmap storage and bitwise operations performance when data is sparse.

[0194] Example 3

[0195] Based on the same invention concept, Figure 5 , a first embodiment of the data processing device of the present invention is proposed, which can be a virtual device applied to a data processing device.

[0196] The following combination Figure 5 The functional module diagram shown in FIG. 1 is a detailed description of the data processing device provided in this embodiment, and the device may include:

[0197] a data acquisition module, configured to acquire data to be processed and a preset mark, wherein the preset mark is used to indicate a first data structure, wherein the first data structure includes any one of an array, a set, and a bitmap group, wherein the bitmap group is created based on the occupied memory of the array and the occupied memory of the set;

[0198] a data deduplication module, configured to perform deduplication processing on the data to be processed and the data in the first data structure;

[0199] A data insertion module is configured to insert the data to be processed into the first data structure when the data to be processed does not overlap with the data in the first data structure.

[0200] Furthermore, the device may further include:

[0201] Data structure creation module for:

[0202] Create arrays and collections;

[0203] Determine a first memory and a second memory, wherein the first memory is the memory occupied by the array, and the second memory is the memory occupied by the set;

[0204] Determining a bitmap capacity based on the first memory and the second memory; wherein the bitmap capacity is the sum of a first multiple of the first memory and a second multiple of the second memory, and the first multiple and the second multiple are in a linear relationship;

[0205] Determining the number of bitmaps according to the bitmap capacity;

[0206] A bitmap group is created according to the bitmap capacity and the number of bitmaps.

[0207] Furthermore, the data structure creation module can be specifically used to:

[0208] The bitmap capacity is determined using a first calculation formula based on the first memory and the second memory; wherein the first calculation formula is:

[0209] C=8(βm+n),

[0210] Wherein, C represents the bitmap capacity, β represents the preset coefficient, m represents the first memory, and n represents the second memory.

[0211] Furthermore, the data acquisition module may include:

[0212] The preset mark determination submodule is used to determine the preset mark according to the preset parameters of the data to be processed.

[0213] Furthermore, when the first data structure is an array or a set, the apparatus may further include:

[0214] A preset mark updating module is used to update the preset mark according to the amount of data in the first data structure.

[0215] Furthermore, the preset mark updating module may include:

[0216] a first judging unit, configured to judge whether the amount of data inserted into the first data structure reaches a first threshold;

[0217] a first determination unit configured to determine a second data structure according to a preset position sequence when the amount of data in the first data structure reaches a first threshold; wherein the preset position sequence includes arrays, sets, and bitmap groups arranged in sequence;

[0218] a data copying unit, configured to copy the data in the first data structure to the second data structure;

[0219] A preset mark updating unit is configured to update the preset mark according to the second data structure.

[0220] Specifically, the preset mark updating module may further include:

[0221] a second determining unit, configured to determine whether the amount of data inserted into the first data structure reaches a second threshold;

[0222] The second determination unit is configured to update the preset mark according to the second data structure when the amount of data in the first data structure reaches a second threshold; wherein the second threshold is greater than the first threshold.

[0223] Specifically, when the first data structure is an array, the first threshold is the product of the array capacity and a first preset ratio; when the first data structure is a set, the first threshold is the product of the set capacity and a second preset ratio; wherein the second preset ratio is smaller than the first preset ratio.

[0224] It should be noted that the functions that can be implemented by each module in the data processing device provided in this embodiment and the corresponding technical effects achieved can refer to the description of the specific implementation methods in each embodiment of the data processing method of the present invention. For the sake of brevity of the description, they will not be repeated here.

[0225] Example 4

[0226] Based on the same invention concept, Figure 2 , which is a schematic diagram of the hardware structure of a data processing device involved in various embodiments of the present invention. This embodiment provides a data processing device, which may include a processor and a memory. The memory stores a data processing program. When executed by the processor, the data processing program implements all or part of the steps of various embodiments of the data processing method of the present invention.

[0227] Specifically, the data processing device refers to a terminal device or network device that can achieve network connection, which can be a terminal device such as a mobile phone, computer, tablet computer, portable computer, or a network device such as a server or cloud platform.

[0228] It can be understood that the data processing device may further include a communication bus, a user interface and a network interface.

[0229] The communication bus is used to realize the connection and communication between these components.

[0230] The user interface is used to connect to the client and communicate data with the client. The user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard. Optionally, the user interface may also include other input / output interfaces, such as a standard wired interface and a wireless interface.

[0231] The network interface is used to connect to the backend server and perform data communication with the backend server. The network interface may include an input / output interface, such as a standard wired interface, or a wireless interface, such as a Wi-Fi interface.

[0232] The memory is used to store various types of data, which may include, for example, instructions for any application or method in the data processing device, as well as data related to the application. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Optionally, the memory can also be a storage device independent of the processor.

[0233] The processor is used to call the data processing program stored in the memory and execute the data processing method as described above. The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute all or part of the steps of each embodiment of the data processing method as described above.

[0234] Example 5

[0235] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, a server, etc., wherein a computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, it can implement all or part of the steps of each embodiment of the data processing method of the present invention.

[0236] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0237] The above descriptions are merely optional embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by utilizing the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining data to be processed and a preset tag, where the preset tag is used to indicate a first data structure, the first data structure including any one of an array, a set, and a bitmap group, the bitmap group being created based on the memory occupied by the array and the memory occupied by the set, with the total capacity of the array, the set, and the bitmap group increasing in sequence; performing deduplication processing on the data to be processed and the data in the first data structure; When the data to be processed does not overlap with the data in the first data structure, inserting the data to be processed into the first data structure; Before the step of obtaining the preset mark, the method further includes: Determining a bitmap capacity according to the memory occupied by the array and the memory occupied by the collection; Determining the number of bitmaps according to a quotient of a maximum value of the data to be processed and a capacity of the bitmap; A bitmap group is created according to the bitmap capacity and the number of bitmaps.

2. The data processing method according to claim 1, wherein: The determining the bitmap capacity according to the memory occupied by the array and the memory occupied by the set includes: Create arrays and collections; Determine a first memory and a second memory, wherein the first memory is the memory occupied by the array, and the second memory is the memory occupied by the set; A bitmap capacity is determined according to the first memory and the second memory.

3. The data processing method according to claim 2, wherein: The bitmap capacity is the sum of a first multiple of the first memory and a second multiple of the second memory, and the first multiple and the second multiple are in a linear relationship.

4. The data processing method according to claim 1, wherein: The step of obtaining a preset mark includes: A preset mark is determined according to the preset parameters of the data to be processed.

5. The data processing method according to claim 1, wherein: When the first data structure is an array or a set, after the step of inserting the to-be-processed data into the first data structure, the method further includes: The preset mark is updated according to the amount of data in the first data structure.

6. The data processing method according to claim 5, wherein: The step of updating the preset mark according to the amount of data in the first data structure includes: When the amount of data in the first data structure reaches a first threshold, determining a second data structure according to a preset position sequence, wherein the preset position sequence includes arrays, sets, and bitmap groups arranged in sequence; copying the data in the first data structure to the second data structure; The preset mark is updated according to the second data structure.

7. The data processing method according to claim 6, wherein: The step of updating the preset mark according to the second data structure includes: When the amount of data in the first data structure reaches a second threshold, the preset mark is updated according to the second data structure, wherein the second threshold is greater than the first threshold.

8. A data processing device, characterized in that: The device comprises: a data acquisition module, configured to acquire data to be processed and a preset mark, wherein the preset mark is used to indicate a first data structure, wherein the first data structure includes any one of an array, a set, and a bitmap group, wherein the bitmap group is created based on the memory occupied by the array and the memory occupied by the set, and the total capacity of the array, the set, and the bitmap group increases in sequence; a data deduplication module, configured to perform deduplication processing on the data to be processed and the data in the first data structure; a data insertion module, configured to insert the data to be processed into the first data structure when the data to be processed does not overlap with the data in the first data structure; The device further comprises: The data structure creation module is used to determine the bitmap capacity according to the memory occupied by the array and the memory occupied by the collection; determine the number of bitmaps according to the quotient of the maximum value of the data to be processed and the bitmap capacity; and create a bitmap group according to the bitmap capacity and the number of bitmaps.

9. A data processing device, characterized in that: The device includes a processor and a memory, wherein a data processing program is stored in the memory, and when the data processing program is executed by the processor, the data processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by one or more processors, the data processing method according to any one of claims 1 to 7 is implemented.

Images