File segmentation method and device

Abstract

The present invention provides a file sharding method and device, which can be used in the financial field or other fields. The method includes: reading records in a data file to be split, screening the records according to the key fields corresponding to the data file to be split; after the record screening passes, determining the hash value corresponding to the record according to the key fields; taking the remainder of the hash value corresponding to the record and the preset number of shards; writing the record into a sharding small file in a sharding folder according to the file structure of the data file to be split; wherein the sharding folder corresponds to the remainder one by one. The present invention uses a hash algorithm to perform efficient file sharding, splits a large file into sharding small files, fully considers factors such as single file hashing and data validity, has high processing efficiency, is not prone to errors, and provides high-quality basic services for modern banking systems to frequently process massive data files.

Description

Technical Field

[0001] The present invention relates to the technical field of file splitting, and in particular to a file segmentation method and device. Background Art

[0002] At present, when processing files with large data volumes in batches, banking systems often read the entire file into the cache at one time, which causes a series of problems such as frequent I\O read and write, high memory usage, slow data processing, and incorrect data formats. In the worst case, it causes a timeout in file processing, affecting the next step of the process; in the worst case, it occupies a large number of online transaction resources and blocks daytime business. Existing file splitting technologies have problems such as low processing efficiency and easy data errors. Summary of the invention

[0003] In view of the problems in the prior art, the main purpose of the embodiments of the present invention is to provide a file fragmentation method and device, so as to realize efficient, asynchronous and effective splitting of large data files into fragmented small files.

[0004] To achieve the above object, an embodiment of the present invention provides a file segmentation method, the method comprising:

[0005] Reading records in the data file to be split, and screening the records according to key fields corresponding to the data file to be split;

[0006] After the record is screened, determining a hash value corresponding to the record according to the key field;

[0007] The remainder is obtained by taking the modulus of the hash value corresponding to the record and the preset number of shards;

[0008] According to the file structure of the data file to be split, the record is written into a fragmented small file in a fragment folder; wherein the fragment folder corresponds to the remainder one by one.

[0009] Optionally, in an embodiment of the present invention, the method further includes: if it is known that the key field is of numeric type, converting the key field into a character type.

[0010] Optionally, in one embodiment of the present invention, determining the hash value corresponding to the record based on the character-type key field includes: taking the byte array of the character-type key field according to the Chinese character internal code extension specification, and writing the byte array into a character buffer; using a preset initial hash value, performing shift operations and logical operations on the characters in the character buffer to obtain a hash value corresponding to the record.

[0011] Optionally, in one embodiment of the present invention, the method further includes: receiving a check file and a data file to be split, and verifying the data file to be split; after the data file to be split is verified, determining a processing mode of the data file to be split according to the correspondence between the check file and the data file to be split; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

[0012] Optionally, in one embodiment of the present invention, the step of writing the record into a shard small file in a shard folder includes: if it is known that the number of records in the shard small file has not reached a preset maximum number of records, writing the record into the shard small file; if it is known that the number of records in the shard small file has reached a preset maximum number of records, creating a new shard small file in the shard folder, and writing the record into the new shard small file.

[0013] The embodiment of the present invention further provides a file segmentation device, the device comprising:

[0014] A record reading module, used for reading records in the data file to be split, and screening the records according to key fields corresponding to the data file to be split;

[0015] A hash value module, used to determine a hash value corresponding to the record according to the key field after the record is screened;

[0016] A remainder determination module, used to obtain a remainder by taking the remainder of the hash value corresponding to the record and the preset number of shards;

[0017] A record writing module is used to write the record into a fragmented small file in a fragment folder according to the file structure of the data file to be split; wherein the fragment folder corresponds to the remainder one by one.

[0018] Optionally, in one embodiment of the present invention, the hash value module is further used to: if it is known that the key field is a numeric type, convert the key field into a character type.

[0019] Optionally, in one embodiment of the present invention, the hash value module includes: a character cache unit, used to obtain the byte array of the character type key field according to the Chinese character internal code extension specification, and write the byte array into the character cache area; a hash value unit, used to use a preset initial hash value to perform shift operations and logical operations on the byte array in the character cache area to obtain a hash value corresponding to the record.

[0020] Optionally, in one embodiment of the present invention, the device also includes: a file verification module, used to receive a check file and a data file to be split, and verify the data file to be split; a processing mode module, used to determine the processing mode of the data file to be split according to the correspondence between the check file and the data file to be split after the data file to be split has passed the verification; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

[0021] Optionally, in one embodiment of the present invention, the record writing module is also used for: if it is known that the number of records in the shard small file has not reached a preset maximum number of records, then writing the record into the shard small file; if it is known that the number of records in the shard small file has reached a preset maximum number of records, then creating a new shard small file in the shard folder and writing the record into the newly created shard small file.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0023] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above method.

[0024] The present invention utilizes a hash algorithm to perform efficient file fragmentation, splitting a large file into fragmented small files, fully considering factors such as single file hashing and data validity, with high processing efficiency and low error rates, providing high-quality basic services for modern banking systems that frequently process massive data files. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flowchart of a file segmentation method according to an embodiment of the present invention;

[0027] Figure 2 A flowchart of determining a hash value in an embodiment of the present invention;

[0028] Figure 3 A flowchart for determining a processing mode in an embodiment of the present invention;

[0029] Figure 4 A flowchart of file segmentation in a specific embodiment of the present invention;

[0030] Figure 5 A schematic diagram of the structure of a file segmentation device according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of a hash value module in an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The embodiments of the present invention provide a file segmentation method and device, which can be used in the financial field or other fields. It should be noted that the file segmentation method and device of the present invention can be used in the financial field, and can also be used in any field other than the financial field. The application field of the file segmentation method and device of the present invention is not limited.

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] At present, the banking system has problems such as slow processing, many erroneous data, and resource occupation when processing large data files. The present invention performs efficient file segmentation based on key field hashing, and asynchronously splits multiple large files into corresponding small segment files. The execution subject of the file segmentation method provided by the embodiment of the present invention includes but is not limited to a computer. Figure 1 FIG. 1 is a flow chart of a file segmentation method according to an embodiment of the present invention. The method shown in the figure includes:

[0036] Step S1, read the records in the data file to be split, and screen the records according to the key fields corresponding to the data file to be split.

[0037] Among them, read the records in the data file to be split one by one, screen the records according to the key fields, and remove the wrong records. For example, if the personal customer file contains the record of the corporate customer number by mistake, it will be removed. According to the type of the data file to be split, determine the key field corresponding to it. For example, if the data file to be split is a personal customer file, its key field is the customer number.

[0038] Step S2, after the record is screened, determine the hash value corresponding to the record based on the key field.

[0039] The specific process of obtaining the hash value according to the key field is as follows: the value of the key field is judged. If it is NULL or an empty string, 0 is directly returned; otherwise, the following judgment is continued: if the key field is a character type, no conversion is performed; if the key field is a numeric type, the leading 0 is removed and converted to a character type. The byte array of the character type key field is obtained according to the GBK (Chinese Character Internal Code Extension Specification) encoding method, and the byte array is transferred to the character buffer. The hash value corresponding to the record is obtained by shifting the characters in the character buffer and performing logical operations.

[0040] Step S3, taking the remainder of the hash value corresponding to the record and the preset number of shards.

[0041] The number of shards is predefined and indicates the number of folders that the data file to be split needs to be split into. For example, based on prior knowledge, the number of shards of a personal customer file is 128, and the hash value of a certain record is 2, then the remainder is 2.

[0042] Step S4, writing the record into a fragmented small file in a fragmented folder according to the file structure of the data file to be split; wherein the fragmented folder corresponds to the remainder one by one.

[0043] Among them, there is a one-to-one correspondence between the shard folder and the remainder, and the correspondence can be achieved by including the remainder in the file name of the shard folder. For example, the file name of the shard folder can be set to File-2, and the shard folder corresponds to the remainder with a value of 2. The shard folder includes at least one small shard file, and the maximum number of records that can be recorded in the small shard file needs to be defined in advance. For example, based on prior knowledge, the number of shards of the personal customer file is 128, and the maximum number of records in the corresponding small shard file is 50,000. It has been verified that this data is the optimal shard data, and the splitting and processing effect is the best.

[0044] Furthermore, when writing a certain record, it can be determined whether the fragmented small file of the current fragmented folder has reached the maximum number of records. If not, the record is directly written into the current fragmented small file according to the file structure of the data file to be split. Specifically, the source file (data file to be split) and the fragmented small file structure are consistent, and the field name, field type, field length, character set, and field order of the source file are maintained in an XML file, which is consistent with the source file. The source file records are read one by one and the key-value pairs {'field name': field value} in the source file record are returned in the form of Map. A List is used to store all the field names of the source file, and the field values ​​of the Map are obtained by looping through the field names. Another List is used to store the retrieved field values, and the character set of the file is matched, and finally the values ​​of the list are written into the fragmented small file according to the field order. If the number of records in the current fragmented small file has reached the maximum number of records, a new fragmented small file is created, and the record is written into the newly created fragmented small file according to the file structure of the data file to be split. Specifically, the file name of the newly created fragmented small file can be named based on the file name of the previous fragmented small file plus 1.

[0045] As an embodiment of the present invention, the method further includes: if it is known that the key field is of numeric type, converting the key field into a character type.

[0046] If the key field is a numeric type, the leading 0 is removed and the key field is converted to a character type.

[0047] In this embodiment, if Figure 2 As shown, according to the key field, determining the hash value corresponding to the record includes:

[0048] Step S21, according to the Chinese character internal code expansion specification, obtain the byte array of the character type key field, and write the byte array into the character buffer area.

[0049] Among them, the byte array of the character type key field is obtained according to the GBK (Chinese Character Internal Code Extension Standard) encoding method, and the byte array is transferred to the character buffer for little-endian storage.

[0050] Step S22, using the preset initial hash value, performing shift operation and logic operation on the byte array in the character buffer area to obtain a hash value corresponding to the record.

[0051] Specifically, take the a priori random numbers seed, m and r, and the initial hash value h is seed m*, where m is the initially defined constant and has no practical meaning. It can be understood as the power of the exponential function in the hash function. Specifically, the value of m can be m=0xc6a4a7935bd1e995L, which can be adjusted according to the actual splitting effect; the character buffer length is raised to the power, and the buffer is processed every 8 bits. The hash value is continuously *m, XOR, and circularly shifted right by r bits. seed, m, and r are all based on empirical values. The biggest advantage of using this hash algorithm is that given a hash function, the file records after splitting are hashed more evenly, the same content of the key fields is distributed relatively concentratedly, and the character buffer technology is used in the value-taking process, and the hashing speed is faster.

[0052] As an embodiment of the present invention, Figure 3 As shown, the method also includes: receiving a check file and a data file to be split, and verifying the data file to be split; after the data file to be split is verified, determining the processing mode of the data file to be split according to the correspondence between the check file and the data file to be split; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

[0053] Among them, the process of determining the processing mode specifically includes: receiving the BIN file (data file, i.e., the data file to be split) and CHK file (check file) from the upstream system; verifying the size, date, arrival and other factors of the BIN file; determining the correspondence between the CHK file and the BIN file, one-to-one or one-to-many; creating a thread in the thread pool according to the correspondence to perform the splitting operation, a single thread processes one file, and multiple files are asynchronously related.

[0054] As an embodiment of the present invention, writing the record into a shard small file in a shard folder includes: if it is known that the number of records in the shard small file has not reached a preset maximum number of records, writing the record into the shard small file; if it is known that the number of records in the shard small file has reached a preset maximum number of records, creating a new shard small file in the shard folder and writing the record into the newly created shard small file.

[0055] Among them, the shard folder includes at least one shard file, and the maximum number of records that can be recorded in the shard file needs to be defined in advance. When writing a record, first determine whether the current shard file has reached the maximum number of records. If not, directly write the record into the current shard file according to the file structure of the data file to be split. If the number of records in the current shard file has reached the maximum number of records, create a new shard file, and write the record into the new shard file according to the file structure of the data file to be split. Specifically, the file name of the newly created shard file can be named based on the file name of the previous shard file plus 1.

[0056] In a specific embodiment of the present invention, Figure 4 As shown in the figure, taking the analysis of individual customer files as an example, the sharding process is as follows:

[0057] 1) Define the number of folder shards, i.e. the number of shards, and the maximum number of records in the sharded small files. For example, based on prior knowledge, the number of shards for individual customer files is 128, and the maximum number of records in the sharded small files is 50,000. It has been verified that this data is the optimal sharded data, and has the best splitting and processing effects.

[0058] 2) Define key fields, for example, the key field of a personal customer file is the customer number.

[0059] 3) Read the records of the large file one by one.

[0060] 4) Screen records based on key fields. For example, if a personal customer file mistakenly contains a corporate customer number, the record will be removed.

[0061] 5) Take the hash value according to the key field. If the key field is a character type, transcoding is required. Among them, the process of taking the hash value according to the key field: take the value of the key field for judgment. If it is NULL or an empty string, return 0 directly. Otherwise, continue the following judgment: if the key field is a character type, no conversion is performed; if the key field is a numeric type, remove the leading 0 and convert it to a character type. Take the byte array of the character type key field in accordance with the GBK encoding method, and transfer the byte array to the character buffer for little-endian storage. Take the a priori random numbers seed, m and r, and the initial hash value h is seed m*character buffer length power. Take the characters in the character buffer in an 8-bit loop and calculate the value k, k*m, then shift right r bits in a loop, and XOR with itself, and then XOR again *m. h is XORed with k, and *m. If the end of the character buffer is less than 8 bits, apply for another 8-bit character buffer, and calculate the number of the buffer, XOR with h, and then XOR *m. After traversing all the character buffers, the hash value is shifted right by r positions again and XORed with itself. After XORing, it is *m, and then it is shifted right by r positions again and XORed with itself to get the final hash value. In summary, the key field is written into the character buffer, an initial hash value is given, and the buffer is processed every 8 bits. The hash value is continuously *m, XORed, and shifted right by r positions. seed, m, and r are all based on experience. The biggest advantage of using this hash algorithm is that given a hash function, the hash of the split file records is more uniform, the same content of the key fields is relatively concentrated, and the character buffer technology is used in the value-taking process, so the hashing speed is faster.

[0062] 6) Take the remainder of the obtained hash value and the number of shards. For example, if the hash value of a record obtained in step 5 is 2 and the number of shards is 128, then the remainder is 2.

[0063] 7) Select the corresponding shard folder according to the remainder value obtained in step 6), and determine whether the corresponding shard file in the shard folder has reached the maximum number of records, that is, the maximum number of records of a single shard file defined in step 1). If the shard file is not full, write it directly according to the structure of the data file to be split; otherwise, create another shard file, add one to the file name, and write the records according to the structure of the data file to be split.

[0064] 8) Repeat the above steps until the data file to be split is read.

[0065] The present invention utilizes a hash algorithm to perform efficient file fragmentation, splitting a large file into fragmented small files, fully considering factors such as multi-file concurrency, single file hashing and data validity, and interspersed with data checking actions. The small files after fragmentation are uniform and hashed, and the data accuracy is high, which effectively improves data validity, has high processing efficiency, and is not prone to errors, providing high-quality basic services for modern banking systems that frequently process massive data files.

[0066] like Figure 5 FIG. 1 is a schematic diagram of the structure of a file segmentation device according to an embodiment of the present invention. The device shown in the figure includes:

[0067] The record reading module 10 is used to read the records in the data file to be split, and screen the records according to the key fields corresponding to the data file to be split.

[0068] Among them, the records in the data file to be split are read one by one, the records are screened according to the key fields, and the erroneous records are eliminated. For example, if the personal customer file mistakenly contains the record of the corporate customer number, it will be eliminated.

[0069] The hash value module 20 is used to determine the hash value corresponding to the record according to the key field after the record passes the screening.

[0070] The specific process of obtaining the hash value according to the key field is as follows: the value of the key field is judged. If it is NULL or an empty string, 0 is directly returned; otherwise, the following judgment is continued: if the key field is a character type, no conversion is performed; if the key field is a numeric type, the leading 0 is removed and converted to a character type. The byte array of the character type key field is obtained according to the GBK (Chinese Character Internal Code Extension Specification) encoding method, and the byte array is transferred to the character buffer. The hash value corresponding to the record is obtained by shifting the characters in the character buffer and performing logical operations.

[0071] The remainder determination module 30 is used to obtain a remainder by taking the remainder of the hash value corresponding to the record and the preset number of shards.

[0072] The number of shards is predefined and indicates the number of folders that the data file to be split needs to be split into. For example, based on prior knowledge, the number of shards of a personal customer file is 128, and the hash value of a certain record is 2, then the remainder is 2.

[0073] The record writing module 40 is used to write the record into the fragmented small files in the fragment folder according to the file structure of the data file to be split; wherein the fragment folder corresponds to the remainder one by one.

[0074] Among them, there is a one-to-one correspondence between the shard folder and the remainder, and the correspondence can be achieved by including the remainder in the file name of the shard folder. For example, the file name of the shard folder can be set to File-2, and the shard folder corresponds to the remainder with a value of 2. The shard folder includes at least one small shard file, and the maximum number of records that can be recorded in the small shard file needs to be defined in advance. For example, based on prior knowledge, the number of shards of the personal customer file is 128, and the maximum number of records in the corresponding small shard file is 50,000. It has been verified that this data is the optimal shard data, and the splitting and processing effect is the best.

[0075] Furthermore, when writing a certain record, it can be determined whether the fragmented small file of the current fragmented folder has reached the maximum number of records. If not, the record is directly written into the current fragmented small file according to the file structure of the data file to be split. If the number of records in the current fragmented small file has reached the maximum number of records, a new fragmented small file is created, and the record is written into the newly created fragmented small file according to the file structure of the data file to be split. Specifically, the file name of the newly created fragmented small file can be named based on the file name of the previous fragmented small file plus 1.

[0076] As an embodiment of the present invention, the hash value module is further used for: if it is known that the key field is of digital type, converting the key field into a character type.

[0077] In this embodiment, if Figure 6 As shown, the hash value module 20 includes:

[0078] A character buffer unit, used for obtaining a byte array of a character key field according to a Chinese character internal code extension specification, and writing the byte array into a character buffer area;

[0079] The hash value unit is used to use a preset initial hash value to perform a shift operation and a logical operation on the byte array in the character buffer area to obtain a hash value corresponding to the record.

[0080] As an embodiment of the present invention, the device further includes:

[0081] A file verification module is used to receive the inspection file and the data file to be split, and verify the data file to be split;

[0082] A processing mode module is used to determine the processing mode of the data file to be split according to the correspondence between the check file and the data file to be split after the data file to be split has passed the verification; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

[0083] As an embodiment of the present invention, the record writing module is also used for: if it is known that the number of records in the shard small file has not reached a preset maximum number of records, then the record is written into the shard small file; if it is known that the number of records in the shard small file has reached a preset maximum number of records, then a new shard small file is created in the shard folder, and the record is written into the new shard small file.

[0084] Based on the same application concept as the above-mentioned file segmentation method, the present invention also provides the above-mentioned file segmentation device. Since the principle of solving the problem by the file segmentation device is similar to that of the file segmentation method, the implementation of the file segmentation device can refer to the implementation of the file segmentation method, and the repeated parts will not be repeated.

[0085] The present invention utilizes a hash algorithm to perform efficient file fragmentation, splitting a large file into fragmented small files, fully considering factors such as single file hashing and data validity, with high processing efficiency and low error rates, providing high-quality basic services for modern banking systems that frequently process massive data files.

[0086] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0087] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above method.

[0088] like Figure 7 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Figure 7 In addition, the electronic device 600 may also include Figure 7 For components not shown, reference may be made to the prior art.

[0089] like Figure 7 As shown, the central processor 100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processor 100 receives inputs and controls the operations of various components of the electronic device 600.

[0090] The memory 140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 100 may execute the program stored in the memory 140 to implement information storage or processing.

[0091] The input unit 120 provides input to the CPU 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0092] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 600 through the central processor 100.

[0093] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0094] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via an antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0095] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide an audio output via the speaker 131 and receive an audio input from the microphone 132, thereby realizing a common telecommunication function. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 130 is also coupled to the central processor 100, so that the sound can be recorded on the local machine through the microphone 132, and the sound stored on the local machine can be played through the speaker 131.

[0096] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0100] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A file segmentation method, characterized in that: The method comprises: Reading records in the data file to be split, and screening the records according to key fields corresponding to the data file to be split; After the record is screened, determining a hash value corresponding to the record according to the key field; The remainder is obtained by taking the modulus of the hash value corresponding to the record and the preset number of shards; According to the file structure of the data file to be split, the record is written into a fragmented small file in a fragment folder; wherein the fragment folder corresponds to the remainder one by one, and the file structure includes: field name, field type, field length, character set, and field order; The step of writing the record into the fragment file in the fragment folder includes: If it is learned that the number of records in the shard file does not reach the preset maximum number of records, the record is written into the shard file; specifically: Through an XML file that is consistent with the file structure of the data file to be split, read the records of the data file to be split one by one, and return the key value in the record of the data file to be split in the form of a Map, wherein the key value includes a field name and a field value; store all the field names in the data file to be split through a first List, and obtain the field values ​​in the Map based on the field names in a loop; store the obtained field values ​​in the Map in a second List, match the character set in the data file to be split, and write the matched field values ​​in the second List into the fragmented small files; If it is known that the number of records in the shard small file has reached a preset maximum number of records, a new shard small file is created in the shard folder, and the record is written into the new shard small file.

2. The method according to claim 1, characterized in that The method further includes: if it is known that the key field is of numeric type, converting the key field into a character type.

3. The method according to claim 2, characterized in that Determining the hash value corresponding to the record according to the key field includes: According to the Chinese character internal code expansion specification, the byte array of the character type key field is obtained, and the byte array is written into the character buffer area; Using the preset initial hash value, a shift operation and a logical operation are performed on the byte array in the character buffer area to obtain a hash value corresponding to the record.

4. The method according to claim 1, characterized in that: The method further comprises: Receive the inspection file and the data file to be split, and verify the data file to be split; After the data file to be split is verified, the processing mode of the data file to be split is determined according to the correspondence between the check file and the data file to be split; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

5. A file segmentation device, characterized in that: The device comprises: A record reading module, used for reading records in the data file to be split, and screening the records according to key fields corresponding to the data file to be split; A hash value module, used to determine a hash value corresponding to the record according to the key field after the record is screened; A remainder determination module, used to obtain a remainder by taking the remainder of the hash value corresponding to the record and the preset number of shards; A record writing module is used to write the record into a fragmented small file in a fragment folder according to the file structure of the data file to be split; wherein the fragment folder corresponds to the remainder one by one, and the file structure includes: field name, field type, field length, character set, and field order; The record writing module is further used to: if it is known that the number of records in the shard small file does not reach the preset maximum number of records, then write the record into the shard small file; specifically: Through an XML file that is consistent with the file structure of the data file to be split, read the records of the data file to be split one by one, and return the key value in the record of the data file to be split in the form of a Map, wherein the key value includes a field name and a field value; store all the field names in the data file to be split through a first List, and obtain the field values ​​in the Map based on the field names in a loop; store the obtained field values ​​in the Map in a second List, match the character set in the data file to be split, and write the matched field values ​​in the second List into the fragmented small files; If it is known that the number of records in the shard small file has reached a preset maximum number of records, a new shard small file is created in the shard folder, and the record is written into the new shard small file.

6. The device according to claim 5, characterized in that The hash value module is also used for: if it is known that the key field is a numeric type, converting the key field into a character type.

7. The device according to claim 6, characterized in that The hash value module includes: A character buffer unit, used for obtaining a byte array of a character key field according to a Chinese character internal code extension specification, and writing the byte array into a character buffer area; The hash value unit is used to use a preset initial hash value to perform a shift operation and a logical operation on the byte array in the character buffer area to obtain a hash value corresponding to the record.

8. The device according to claim 5, characterized in that The device also includes: A file verification module is used to receive the inspection file and the data file to be split, and verify the data file to be split; A processing mode module is used to determine the processing mode of the data file to be split according to the correspondence between the check file and the data file to be split after the data file to be split has passed the verification; wherein the processing mode includes a single-thread processing mode and a multi-thread asynchronous processing mode.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.

Images