Data batch processing method and device

Abstract

The invention discloses a data batch processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of performing initialization processing on a Bloom filter based on a data size of data to be processed, grouping the data to be processed, performing duplicate checking processing on each grouped data through the Bloom filter, and performing fragmentation operation on the data after duplicate checking to realize parallel processing of different fragmented data. According to the embodiment, the data duplicate checking efficiency is improved, and meanwhile, the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for batch processing of data. Background Art

[0002] With the development of Internet technology, the amount of data processed online is increasing. In the current scenario of large-scale ladder data import, a centralized relational database management system is usually used. When importing data, in order to avoid data errors, it is necessary to ensure the uniqueness of the imported data and deduplicate the data through a unique identifier at the database level. The entire data import process is carried out in a single-threaded environment.

[0003] When the above method is used to import large amounts of data, the data import efficiency mainly depends on the processing capacity of the database. The writing of large amounts of data often brings a heavy burden to the database, affecting the import efficiency. At the same time, the deduplication mechanism based on unique identifiers further increases the burden on the database, affecting the efficiency of data processing. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for data batch processing, which can improve the efficiency of data batch processing and reduce the burden on the database.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for batch processing of data is provided, comprising:

[0006] Grouping the data to be processed in the current batch to obtain at least one data group, and determining a data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field;

[0007] Combined with the data in the data storage shard corresponding to each of the data groups, each of the data groups is deduplicated by using a Bloom filter to obtain target data corresponding to each of the data groups, and the target data corresponding to each of the data groups is stored in the corresponding data storage shard;

[0008] Based on the sorting field, data sorting processing is performed on each of the data storage slices through parallel tasks to complete the batch processing operation.

[0009] Optionally, the method further comprises:

[0010] Calculate hash function parameters and bit array size based on the amount of data to be processed in each batch and a preset false positive rate;

[0011] The Bloom filter is initialized based on the hash function parameters and the bit array size.

[0012] Optionally, the data to be processed in the current batch is grouped to obtain at least one data group, including:

[0013] Encode the dimensions and dimension values ​​of the current batch of data to be processed;

[0014] The data to be processed are grouped according to the encoding result to obtain no less than one data group.

[0015] Optionally, encode the dimensions and dimension values ​​of the current batch of data to be processed, including:

[0016] Determine the encoding length corresponding to each dimension according to the number of dimension values ​​corresponding to each dimension of the data to be processed;

[0017] Based on the encoding length corresponding to each dimension, the dimensions and dimension values ​​of the current batch of data to be processed are encoded.

[0018] Optionally, combining the data in the data storage shard corresponding to each of the data groups, performing data deduplication processing on each of the data groups through a Bloom filter to obtain target data corresponding to each of the data groups, including:

[0019] For each of the data groups, determining, by a Bloom filter, whether each data in the data group already exists in a data storage slice corresponding to the data group;

[0020] Adding data determined not to exist in the data storage slice corresponding to the data group to the target data corresponding to the data group;

[0021] For data that is determined to already exist in the data storage slice corresponding to the data group, the accuracy of the determination result is determined by searching the data in the data storage slice corresponding to the data group, and if the determination result is inaccurate, the data is added to the target data corresponding to the data group.

[0022] Optionally, based on the sorting field, performing data sorting processing on each of the data storage shards through parallel tasks includes:

[0023] Based on the sorting field, through parallel tasks, an insertion sort algorithm is used in combination with a head and tail check of the sorting field value to perform data sorting processing on each data storage fragment.

[0024] According to another aspect of an embodiment of the present invention, a device for batch processing of data is provided, comprising:

[0025] A grouping module, used to group the data to be processed in the current batch to obtain at least one data group, and determine the data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field;

[0026] A deduplication module is used to combine the data in the data storage shard corresponding to each of the data groups, perform data deduplication processing on each of the data groups through a Bloom filter, obtain the target data corresponding to each of the data groups, and store the target data corresponding to each of the data groups in the corresponding data storage shards respectively;

[0027] The processing module is used to perform data sorting processing on each of the data storage slices through parallel tasks based on the sorting field to complete the batch processing operation.

[0028] Optionally, the device further comprises:

[0029] A calculation module, used for calculating hash function parameters and bit array size based on the data volume of each batch of data to be processed and a preset false positive rate;

[0030] An initialization module is used to initialize the Bloom filter based on the hash function parameters and the bit array size.

[0031] Optionally, the grouping module is further used to:

[0032] Encode the dimensions and dimension values ​​of the current batch of data to be processed;

[0033] The data to be processed are grouped according to the encoding result to obtain no less than one data group.

[0034] Optionally, the grouping module is further used to:

[0035] Determine the encoding length corresponding to each dimension according to the number of dimension values ​​corresponding to each dimension of the data to be processed;

[0036] Based on the encoding length corresponding to each dimension, the dimensions and dimension values ​​of the current batch of data to be processed are encoded.

[0037] Optionally, the deduplication module is further used to:

[0038] For each of the data groups, determining, by a Bloom filter, whether each data in the data group already exists in a data storage slice corresponding to the data group;

[0039] Adding data determined not to exist in the data storage slice corresponding to the data group to the target data corresponding to the data group;

[0040] For data that is determined to already exist in the data storage slice corresponding to the data group, the accuracy of the determination result is determined by searching the data in the data storage slice corresponding to the data group, and if the determination result is inaccurate, the data is added to the target data corresponding to the data group.

[0041] Optionally, the processing module is further configured to:

[0042] Based on the sorting field, through parallel tasks, an insertion sort algorithm is used in combination with a head and tail check of the sorting field value to perform data sorting processing on each data storage fragment.

[0043] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for data batch processing provided by an embodiment of the present invention.

[0044] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for batch processing of data provided by an embodiment of the present invention is implemented.

[0045] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data batch processing method provided by an embodiment of the present invention is implemented.

[0046] An embodiment of the above invention has the following advantages or beneficial effects: filtering and deduplicating the grouped data of the data to be processed through a Bloom filter to obtain the grouped data of the target data, and storing the grouped data in the corresponding shards of the data respectively, and performing sorting operations on the data in each shard to realize the batch processing flow of the data, thereby optimizing the data deduplication process, avoiding data duplication checking in the database, reducing the pressure on the database, and improving the efficiency of data import. At the same time, data sorting by sharding realizes flexible sorting of small-range data, thereby improving the efficiency of data batch processing.

[0047] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0049] Figure 1 is a schematic diagram of main steps of a method for batch processing of data according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of main modules of a device for batch processing of data according to an embodiment of the present invention;

[0051] Figure 3 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0052] Figure 4 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0054] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0055] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0056] Figure 1 Schematic diagram of the main process of the method for batch processing of data according to an embodiment of the present invention. Figure 1 As shown, the data batch processing method of the embodiment of the present invention mainly includes the following steps S101 to S103.

[0057] Step S101, grouping the data to be processed in the current batch to obtain no less than one data group, and determining the data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field.

[0058] The data that needs to be processed for warehousing is obtained. For example, the data obtained may be logistics quotation data of a large amount of data, including dimension data and ladder data of the logistics quotation. Specifically, taking a logistics quotation method as an example, the dimensions of the logistics quotation include piece type, category, weight / volume, the values ​​corresponding to the piece type are large piece type / medium piece type / small piece type, the values ​​corresponding to the category are grade A / grade B / grade C, the values ​​corresponding to the weight include: 0-2kg, 2-3kg, and the values ​​corresponding to the volume include 0-2m³, 2-4m³. Among them, weight and volume belong to ladder data, and the dimensions and ladder data can be flexibly set according to different scenarios, and the corresponding dimension values ​​and ladder values ​​can also be flexibly set, without limitation. For example, the dimensions under scenario A include piece type and category, and the dimensions under scenario B include piece type, category, shape, etc.; the ladder data under scenario A includes weight 0-2kg, 2-3kg, and the ladder data under scenario B includes weight 0-5kg, 5-10kg. Each piece of data corresponds to a quotation information. For example, large-sized grade A (0-2kg) goods correspond to quotation information A. Quotation information A can be a specific value or quotation calculation rule information (10+4 / kg).

[0059] According to the dimensions and step data of the acquired data, data with the same dimensions or data with the same partial dimensions and partial dimension values ​​within the preset range are divided into the same group. For example, the logistics data of large-sized A-class and weighing 0-5kg are divided into one group to obtain the data of each group. Among them, the sorting field of the data to be processed is the field of the step data; each group corresponds to a storage shard.

[0060] Optionally, when performing grouping, data grouping may also be performed only based on dimensional data, such as grouping data of the same dimension into one group, such as grouping all large-sized logistics data into one group.

[0061] Step S102, combining the data in the data storage shard corresponding to each of the data groups, performing data deduplication processing on each of the data groups through a Bloom filter, obtaining the target data corresponding to each of the data groups, and storing the target data corresponding to each of the data groups in the corresponding data storage shards respectively.

[0062] According to the grouping results, the data of each group is checked for duplicates in turn through the Bloom filter, and the target data that needs to be stored is determined. Specifically, a hash calculation is performed based on the dimensional data and ladder data of each data to obtain the value of the corresponding bit array, and it is determined whether it is the same as the value of the bit array of the stored data. If not, it means that there is no duplicate data and the data needs to be stored; if so, it means that there is duplicate data and the data needs to be deleted. Among them, the stored data is the data in the data storage shard corresponding to each data group; when the data is stored, the data is stored in the data storage shard corresponding to the group to which it belongs.

[0063] Step S103: Based on the sorting field, data sorting processing is performed on each of the data storage slices through parallel tasks to complete a batch processing operation.

[0064] Sort the data in the shard according to the sorting field, that is, the field of the ladder data, and sort each shard separately through parallel tasks. When sorting, you can sort in ascending or descending order according to the values ​​corresponding to the field of the ladder data, for example, sort in the order of (0, 2], (2, 3], ..., to achieve sorting operations on small-range data.

[0065] The data batch processing method provided in the embodiment of the present invention can filter and deduplicate the grouped data of the data to be processed through a Bloom filter, obtain the grouped data of the target data, and store the grouped data in the corresponding shards in the data, and perform sorting operations on the data in each shard to realize the batch processing flow of the data, thereby optimizing the data deduplication process, avoiding data duplication checking in the database, reducing the pressure on the database, and improving the efficiency of data import. At the same time, data sorting by sharding realizes flexible sorting of small-range data, thereby improving the efficiency of data batch processing.

[0066] Optionally, after obtaining the data to be processed, it is necessary to preprocess the data to be processed, including checking the validity of the data. Specifically, it includes verifying the empty fields of the data and identifying illegal fields. When verifying the fields of the data, if the data contains empty fields or the fields contained are not in the preset legal field table, the data is marked as illegal and deleted, and will not participate in subsequent duplicate detection. While performing the inspection, the dimension fields and ladder field sets of each data, as well as the total number of rows of the data, are obtained.

[0067] Optionally, the method further includes: calculating hash function parameters and bit array size based on the data volume of each batch of data to be processed and a preset false positive rate; initializing the Bloom filter based on the hash function parameters and bit array size. Initializing the Bloom filter according to the amount of acquired data includes calculating the number of hash functions and the size of the bit array required by the Bloom filter according to the total number of rows of data. Initializing the Bloom filter according to the calculated number of hash functions and bit array to ensure that each data corresponds to a unique bit array value for subsequent data duplication checking.

[0068] Optionally, the initialization operation of the Bloom filter only needs to be performed once. Since the amount of data acquired each time subsequently is the same, the calculation of the number of hash functions and the number of bit arrays only needs to be performed once.

[0069] Optionally, when the Bloom filter is initialized, the calculation is performed in combination with the preset false positive rate, that is, the hash function parameters, including the number of hash functions and the size of the bit array, are calculated based on the data volume and the preset false positive rate to ensure that the value of the bit array of each data is different under the condition that the false positive rate is met. Among them, the false positive rate is generally 0.001, at which time the resource usage is relatively balanced, and when the data volume is within 10 million, the memory usage is about 20M.

[0070] The data batch processing method provided in an embodiment of the present invention can calculate hash function parameters and bit array size according to the data volume, and initialize the Bloom filter according to the calculation result to ensure the uniqueness of the bit array value of each data, so as to realize the deduplication operation of the Bloom filter and improve the efficiency of data batch processing.

[0071] Optionally, the data to be processed in the current batch are grouped to obtain no less than one data group, including: encoding the dimensions and dimension values ​​of the data to be processed in the current batch; grouping the data to be processed according to the encoding results to obtain no less than one data group. When grouping the data to be processed, the dimensions and dimension values ​​of the data to be processed are first encoded. For example, for the piece type, the large piece type is encoded as 00, the medium piece type is encoded as 01, and the small piece type is encoded as 10. For the grade, grade A is 00000, grade B is 00001, etc. The specific encoding value can be set according to the actual situation and is not limited to this. Grouping is performed according to the encoding results, for example, data with the same piece type code are divided into the same group, or data with the same piece type code and grade code are divided into the same group, etc. Each code corresponds to a unique piece of data.

[0072] Optionally, encoding the dimensions and dimension values ​​of the current batch of data to be processed includes: determining the encoding length corresponding to each dimension according to the number of dimension values ​​corresponding to each dimension of the data to be processed; encoding the dimensions and dimension values ​​of the current batch of data to be processed based on the encoding length corresponding to each dimension. One achievable encoding method is to determine the number of bits required for encoding according to the number of dimension values ​​of each dimension in the data to be processed. For example, the item type includes three types: large, medium, and small, and the number of bits required for encoding is 2, and the corresponding codes are 00, 01, and 10 respectively; the category includes 26 categories, and the number of bits required for encoding is 5, and the corresponding codes are 00000-A level, 00001-B level... 11001-X level, etc. The number of codes required for each dimension is summarized, and the number is determined based on a complete BK code. For example, assuming that the code is required: large item B level weight [2-3kg]: item type (large item): 00 category (B level): 00001 weight [2-3kg]: 01. The resulting Binary Key (BK) is: `00 00001 01` By combining these codes together, a 9-bit unique Binary Key (BK) can be generated to represent each combination.

[0073] The data batch processing method provided in an embodiment of the present invention can encode data and realize group processing of data. Dividing similar data into the same group helps to perform efficient sorting operations on small-scale data in the subsequent process. At the same time, it can ensure the continuity and sequentiality of the data processing process based on the grouped data, which is convenient for the subsequent data analysis process.

[0074] Combined with the data in the data storage shard corresponding to each data group, each data group is deduplicated by using a Bloom filter to obtain the target data corresponding to each data group, including: for each data group, using a Bloom filter to determine whether each data in the data group already exists in the data storage shard corresponding to the data group; adding the data that is determined to not exist in the data storage shard corresponding to the data group to the target data corresponding to the data group; for the data that is determined to already exist in the data storage shard corresponding to the data group, searching the data in the data storage shard corresponding to the data group to determine the accuracy of the determination result, and if the determination result is inaccurate, adding the data to the target data corresponding to the data group. When deduplicating the data of each group by using a Bloom filter, determine whether the data has been stored in the shard corresponding to the group, specifically, calculate the value of the bit array of the hash function of the data to determine whether there is the same value, if yes, it means that the data is duplicate data and needs to be deleted and not stored; if not, it means that the data is not duplicate data and needs to be stored. Determine that the data to be stored is the target data. For duplicate data, further judgment is made by checking the database twice, that is, whether the database contains the same data as the duplicate data. If it does, the data is determined to be duplicate data. If not, the data is determined not to be duplicate data and is a false judgment. The data needs to be added to the target data.

[0075] Optionally, based on the sorting field, data sorting processing is performed on each of the data storage shards through parallel tasks, including: based on the sorting field, through parallel tasks, an insertion sort algorithm is used in combination with a head and tail check of the sorting field value to perform data sorting processing on each of the data storage shards. For the data in each shard, the ladder data is sorted by an insertion sort algorithm and stored in a data table to ensure the continuity of the data. Specifically, the continuity of the head and tail connection of the ladder data is judged to determine whether there is a crossover or a gap in the data.

[0076] According to the data batch processing method provided by the embodiment of the present invention, the data to be processed can be grouped, and the efficient deduplication operation of the grouped data can be realized through the Bloom filter, so as to realize data deduplication before the data flows into the database, and the local small-range sorting operation of the data can be realized through the distributed sorting method, thereby improving the flexibility of small-range data processing. At the same time, through the setting of sharding, the speed of data import is improved, the data processing burden of the database is reduced, and the efficiency of data batch processing is improved.

[0077] Figure 2 is a schematic diagram of a data batch processing device 200 provided in an embodiment of the present invention, such as Figure 2As shown, the data batch processing device 200 of the embodiment of the present invention mainly includes: a grouping module 201, a deduplication module 202 and a processing module 203.

[0078] A grouping module 201 is used to group the data to be processed in the current batch to obtain at least one data group and determine the data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field;

[0079] A deduplication module 202 is used to combine the data in the data storage shard corresponding to each data group, perform data deduplication processing on each data group through a Bloom filter, obtain target data corresponding to each data group, and store the target data corresponding to each data group in the corresponding data storage shard;

[0080] The processing module 203 is used to perform data sorting processing on each of the data storage slices through parallel tasks based on the sorting field to complete a batch processing operation.

[0081] The data batch processing device provided according to the embodiment of the present invention can filter and deduplicate the grouped data of the data to be processed through a Bloom filter, obtain the grouped data of the target data, and store the grouped data in the corresponding shards in the data, and perform sorting operations on the data in each shard to realize the batch processing flow of the data, thereby optimizing the data deduplication process, avoiding data duplication checking in the database, reducing the pressure on the database, and improving the efficiency of data import. At the same time, data sorting by sharding realizes flexible sorting of small-range data, thereby improving the efficiency of data batch processing.

[0082] Optionally, the device further comprises:

[0083] A calculation module 204, used to calculate hash function parameters and bit array size based on the data volume of each batch of data to be processed and a preset false positive rate;

[0084] The initialization module 205 is used to initialize the Bloom filter based on the hash function parameters and the bit array size.

[0085] Optionally, the grouping module 201 is further configured to:

[0086] Encode the dimensions and dimension values ​​of the current batch of data to be processed;

[0087] The data to be processed are grouped according to the encoding result to obtain no less than one data group.

[0088] Optionally, the grouping module 201 is further configured to:

[0089] Determine the encoding length corresponding to each dimension according to the number of dimension values ​​corresponding to each dimension of the data to be processed;

[0090] Based on the encoding length corresponding to each dimension, the dimensions and dimension values ​​of the current batch of data to be processed are encoded.

[0091] Optionally, the deduplication module 202 is further configured to:

[0092] For each of the data groups, determining, by a Bloom filter, whether each data in the data group already exists in a data storage slice corresponding to the data group;

[0093] Adding data determined not to exist in the data storage slice corresponding to the data group to the target data corresponding to the data group;

[0094] For data that is determined to already exist in the data storage slice corresponding to the data group, the accuracy of the determination result is determined by searching the data in the data storage slice corresponding to the data group, and if the determination result is inaccurate, the data is added to the target data corresponding to the data group.

[0095] Optionally, the processing module 203 is further configured to:

[0096] Based on the sorting field, through parallel tasks, an insertion sort algorithm is used in combination with a head and tail check of the sorting field value to perform data sorting processing on each data storage fragment.

[0097] The data batch processing device provided by the embodiment of the present invention can group the data to be processed, and implement efficient deduplication operation on the grouped data through Bloom filter, implement data deduplication before the data flows into the database, and implement local small-range sorting operation of data through distributed sorting method, thereby improving the flexibility of small-range data processing. At the same time, through the setting of sharding, the speed of data import is improved, the data processing burden of the database is reduced, and the efficiency of data batch processing is improved.

[0098] Figure 3 An exemplary system architecture 300 is shown to which the method or apparatus for batch processing of data according to an embodiment of the present invention may be applied.

[0099] like Figure 3 As shown, the system architecture 300 may include terminal devices 301, 302, 303, a network 304 and a server 305. The network 304 is used to provide a medium for communication links between the terminal devices 301, 302, 303 and the server 305. The network 304 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0100] Users can use terminal devices 301, 302, 303 to interact with server 305 through network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, 303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0101] The terminal devices 301 , 302 , and 303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0102] The server 305 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 301, 302, and 303. The backend management server may analyze and process the received batch processing requests and other data, and feed back the processing results (such as batch processing data - only an example) to the terminal device.

[0103] It should be noted that the data batch processing method provided in the embodiment of the present invention is generally executed by the server 305 , and accordingly, the data batch processing device is generally set in the server 305 .

[0104] It should be understood that Figure 3 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0105] Reference below Figure 4 , which shows a schematic diagram of the structure of a computer system 400 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 4 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0106] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0107] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0108] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the system of the present invention are executed.

[0109] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be set in a processor. For example, they may be described as: a processor includes a grouping module, a deduplication module, and a blow module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the grouping module may also be described as "a module for grouping the current batch of data to be processed, obtaining no less than one data group, and determining the data storage shard corresponding to each of the data groups".

[0112] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0113] Grouping the data to be processed in the current batch to obtain at least one data group, and determining a data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field;

[0114] Combined with the data in the data storage shard corresponding to each of the data groups, each of the data groups is deduplicated by using a Bloom filter to obtain target data corresponding to each of the data groups, and the target data corresponding to each of the data groups is stored in the corresponding data storage shard;

[0115] Based on the sorting field, data sorting processing is performed on each of the data storage slices through parallel tasks to complete the batch processing operation.

[0116] According to the technical solution of the embodiment of the present invention, the grouped data of the data to be processed is filtered and deduplicated through a Bloom filter to obtain the grouped data of the target data, and the grouped data is stored in the corresponding shards in the data respectively, and the data in each shard is sorted to realize the batch processing flow of the data, thereby optimizing the data deduplication process, avoiding data duplication checking in the database, reducing the pressure on the database, and improving the efficiency of data import. At the same time, data sorting through shards realizes flexible sorting of small-range data, thereby improving the efficiency of data batch processing.

[0117] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data batch processing method provided by an embodiment of the present invention is implemented.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for batch processing of data, characterized in that: include: Grouping the data to be processed in the current batch to obtain at least one data group, and determining a data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field; Combined with the data in the data storage shard corresponding to each of the data groups, each of the data groups is deduplicated by using a Bloom filter to obtain target data corresponding to each of the data groups, and the target data corresponding to each of the data groups is stored in the corresponding data storage shard; Based on the sorting field, data sorting processing is performed on each of the data storage slices through parallel tasks to complete the batch processing operation.

2. The method according to claim 1, characterized in that The method further comprises: Calculate hash function parameters and bit array size based on the amount of data to be processed in each batch and a preset false positive rate; The Bloom filter is initialized based on the hash function parameters and the bit array size.

3. The method according to claim 1 or 2, characterized in that: The current batch of data to be processed is grouped to obtain at least one data group, including: Encode the dimensions and dimension values ​​of the current batch of data to be processed; The data to be processed are grouped according to the encoding result to obtain no less than one data group.

4. The method according to claim 3, characterized in that Encode the dimensions and dimension values ​​of the current batch of data to be processed, including: Determine the encoding length corresponding to each dimension according to the number of dimension values ​​corresponding to each dimension of the data to be processed; Based on the encoding length corresponding to each dimension, the dimensions and dimension values ​​of the current batch of data to be processed are encoded.

5. The method according to claim 1 or 2, characterized in that: Combining the data in the data storage shard corresponding to each of the data groups, performing data deduplication processing on each of the data groups through a Bloom filter to obtain target data corresponding to each of the data groups, includes: For each of the data groups, determining, by a Bloom filter, whether each data in the data group already exists in a data storage slice corresponding to the data group; Adding data determined not to exist in the data storage slice corresponding to the data group to the target data corresponding to the data group; For data that is determined to already exist in the data storage slice corresponding to the data group, the accuracy of the determination result is determined by searching the data in the data storage slice corresponding to the data group, and if the determination result is inaccurate, the data is added to the target data corresponding to the data group.

6. The method according to claim 1, characterized in that The performing data sorting processing on each of the data storage slices based on the sorting field through parallel tasks includes: Based on the sorting field, through parallel tasks, an insertion sort algorithm is used in combination with a head and tail check of the sorting field value to perform data sorting processing on each data storage fragment.

7. A device for batch processing of data, characterized in that: include: A grouping module, used to group the data to be processed in the current batch to obtain at least one data group, and determine the data storage shard corresponding to each data group, wherein the data to be processed includes a sorting field; A deduplication module is used to combine the data in the data storage shard corresponding to each of the data groups, perform data deduplication processing on each of the data groups through a Bloom filter, obtain the target data corresponding to each of the data groups, and store the target data corresponding to each of the data groups in the corresponding data storage shards respectively; The processing module is used to perform data sorting processing on each of the data storage slices through parallel tasks based on the sorting field to complete the batch processing operation.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images