Data processing method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence and financial science and technology, and relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed data file, each line of data in the data file comprising a plurality of fields, and each field being isolated by a specific separator; reading the data file in parallel by adopting a memory mapping file mode, segmenting the data file into a plurality of data blocks, and setting a corresponding thread for each data block for reading; positioning the starting position and the ending position of each key field in each row of read data based on a specific separator, and acquiring the key fields by adopting an interception mode; performing hash value calculation on the key field according to a preset hash function, and storing the calculated hash value into a predefined open addressing hash table; and performing statistics on each table item in the open addressing hash table to obtain a data statistics result of the data file. The processing efficiency of the big data file can be improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In banking business systems, data interaction between systems generally relies on file transfer, especially in the regulatory reporting link, which is the final exit of data flow and needs to exchange data frequently with many upstream systems. For scenarios where the amount of data reaches millions or tens of millions, files in txt, dat and other formats are usually used as the medium for data transmission. After receiving notifications from the upstream system, the regulatory application system will automatically pull these files from the specified network attached storage (NAS) disk, then parse the file content and import it into the database. During the file reading process, in order to ensure the accuracy and integrity of the data, a series of verification operations are usually performed, including but not limited to non-empty verification of key fields, and statistical calculations such as summation and average of some amount data. However, when the number of file lines climbs to the billion level, the traditional Java program is unable to read and parse the file content into memory and perform calculations in memory, which significantly increases the time consumption, thereby affecting the efficiency and timeliness of subsequent data reporting.

[0003] Therefore, how to improve the processing speed of big data files at the central processing unit (CPU) level has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, apparatus, computer equipment and storage medium that can improve the processing efficiency of large data files.

[0005] In order to solve the above technical problems, the present application provides a data processing method, which adopts the following technical solution:

[0006] Obtain a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters;

[0007] The data file is read in parallel by using a memory mapping file method, the data file is divided into a plurality of data blocks, and a corresponding thread is set for each data block to read;

[0008] When reading each line of data in the data block, the starting position and the ending position of each key field in each line of data are located based on the specific delimiter, and the key field is obtained according to the starting position and the ending position by interception;

[0009] Obtaining a preset hash function, calculating a hash value for the key field according to the hash function, and storing the calculated hash value in a predefined open addressable hash table;

[0010] The open addressable hash tables corresponding to the data blocks that each thread is responsible for are summarized, and statistics are performed on each table entry in the open addressable hash table to obtain data statistics results of the data file.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides a data processing device, which adopts the following technical solution:

[0012] An acquisition module is used to acquire a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters;

[0013] A reading module is used to read the data file in parallel by using a memory mapping file method, divide the data file into multiple data blocks, and set a corresponding thread for each data block to read;

[0014] A separation module, used for locating the starting position and the ending position of each key field in each line of data based on the specific separator when reading each line of data in the data block, and obtaining the key field according to the starting position and the ending position by interception;

[0015] A calculation module, used to obtain a preset hash function, calculate a hash value for the key field according to the hash function, and store the calculated hash value in a predefined open addressable hash table;

[0016] The statistics module is used to summarize the open addressable hash tables corresponding to the data blocks that each thread is responsible for, and to perform statistics on each table entry in the open addressable hash table to obtain the data statistics result of the data file.

[0017] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, which adopts the following technical solution: the computer device includes a memory and a processor, the memory stores computer-readable instructions, and the processor implements the steps of the data processing method described in any one of the above items when executing the computer-readable instructions.

[0018] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which adopts the following technical solution: the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the data processing method described in any one of the above items are implemented.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0020] The embodiment of the present application adopts a memory-mapped file method to read data files in parallel, which can make full use of the computing power of a multi-core central processing unit, divide a large data file into multiple data blocks, and assign an independent thread to each data block for reading. The parallel processing method greatly improves the speed of data reading, thereby shortening the overall processing time; when reading each row of data in the data block, the starting position and the ending position of each key field are accurately located based on a specific delimiter, and the key field is obtained by interception, thereby avoiding unnecessary string operations and improving the efficiency and accuracy of field extraction; by calculating the hash value of the key field according to a preset hash function, and storing the calculated hash value in a predefined open addressable hash table, the fast search and insertion performance of the open addressable hash table can be utilized to further improve the data processing efficiency while ensuring data integrity; in the summary stage, each thread merges the processing results of the data block it is responsible for (i.e., the table entries in the open addressable hash table), thereby quickly obtaining the data statistical results of the entire data file, significantly accelerating the data statistical process, and improving the processing efficiency of large data files. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0023] Figure 2 is a flow chart of an embodiment of the data processing method of the present application;

[0024] Figure 3 is a structural schematic diagram of an embodiment of a data processing device of the present application;

[0025] Figure 4 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0029] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0030] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0031] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0032] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0033] It should be noted that the data processing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the data processing device is generally arranged in the server / terminal device.

[0034] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0035] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for data processing according to the present application. The data processing method comprises the following steps:

[0036] Step S201, obtaining a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters;

[0037] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can obtain the data file to be processed through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 2G / 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0038] In this embodiment, a data file to be processed is obtained, and the data file is a text file containing multiple lines of data, that is, the data in the data file is stored in lines, and each line of data represents a record, and the record can contain multiple fields. Among them, specific separators are used to separate the fields. The specific separators can include commas, semicolons, tabs, spaces, or custom characters, etc., which are used to separate different fields in each line of data. For example, the name of the data file is "account data file.txt", and the content contained in the data file is: general ledger accounting account name, general ledger accounting account number, beginning debit balance, amount of occurrence, etc.

[0039] Step S202, reading the data file in parallel by using a memory mapping file method, dividing the data file into a plurality of data blocks, and setting a corresponding thread for each data block to read;

[0040] In this embodiment, the memory-mapped file method refers to using the memory-mapped file technology to map the data file into the memory so as to efficiently access and process the data.

[0041] Specifically, a memory-mapped file method is used to map the data file into the memory for fast access; according to the number of cores of the central processing unit, the data file is divided into multiple data blocks, and a corresponding independent thread is assigned to each data block for data reading to achieve parallel processing.

[0042] In some optional implementations, the above-mentioned parallel reading of the data file by using a memory-mapped file method and dividing the data file into a plurality of data blocks may include the following steps:

[0043] Using a memory-mapped file method, mapping the content of the data file into a memory to form a memory-mapped file;

[0044] According to the size of the data file, the memory-mapped file is divided into a plurality of data blocks.

[0045] Specifically, a memory-mapped file method is used to map the content of the data file (i.e., the fields recorded in each row of data) into the memory to form a memory-mapped file for fast access; the size of the data file is obtained, and the memory-mapped file is divided into multiple data blocks according to the size of the data file and the number of cores of the central processing unit, and a corresponding independent thread is assigned to each data block for data reading to achieve parallel processing.

[0046] The embodiment of the present application uses a memory-mapped file method to directly map the content of a data file to the memory to form a memory-mapped file, thereby avoiding the frequent disk read and write of traditional file I / O operations and significantly improving the data access speed; since the memory-mapped file allows the program to access the file content as if it were accessing the memory, it can reduce the number of system calls and reduce the context switching overhead, thereby improving the overall efficiency of data reading. By dividing the memory-mapped file into multiple data blocks according to the size of the data file, and assigning a corresponding independent thread to each data block for data reading, the parallel processing capability of the multi-core processor is fully utilized, and the parallel reading and processing of data is realized, which significantly improves the speed and throughput of data processing.

[0047] Step S203, when reading each line of data in the data block, locate the starting position and the ending position of each key field in each line of data based on the specific delimiter, and obtain the key field according to the starting position and the ending position by interception;

[0048] In this embodiment, in each thread, each row of data in the data block that the thread is responsible for is traversed. Based on a specific separator, the starting position and the ending position of each key field in each row of data are located. The key field is extracted according to the located starting position and the ending position by using an interception method. For example, according to a specific separator (such as a comma or a semicolon), the starting position and the ending position of each key field in each row of data are located, and the key fields such as the general ledger account number, the beginning debit balance and the amount incurred in each row of data are extracted according to the starting position and the ending position.

[0049] In some optional implementations, the above-mentioned positioning of the starting position and the ending position of each key field in each row of data based on the specific delimiter, and obtaining the key field according to the starting position and the ending position by interception may include the following steps:

[0050] Traversing from the beginning of the row of the data block, determining the starting position and the ending position of each key field in each row of data by searching for the specific delimiter;

[0051] According to the starting position and the ending position of the key field, the key field is cut off and extracted from the data block to obtain the key field.

[0052] Specifically, the traversal starts from the beginning of the row of the data block (i.e., the starting position of the first row), that is, starting from the first character or byte of the data block, and checking each character or byte step by step until the end of the data block. During the traversal process, a specific delimiter is continuously searched, and the specific delimiter is used to distinguish different fields in each row of data, such as a comma (,), a tab (\t), a space, or a custom character. When a specific delimiter is found, the position of the specific delimiter is recorded, which can be the end position of the key field or the starting position of the next key field (depending on the specific definition of the delimiter and the format of the data). According to the determined starting position and ending position, the corresponding key field is truncated and extracted from the data block. For example, the key field is intercepted by copying or moving the characters or bytes between the starting position and the ending position to a new memory area.

[0053] The embodiment of the present application can accurately determine the starting position and ending position of each field in each row of data by traversing from the beginning of the row of the data block and searching for a specific delimiter, avoiding complex regular expression matching or parser design, thereby improving the efficiency of data parsing. By accurately locating the starting position and ending position of the field, it is possible to process data files with variable length fields without the need for additional parsing logic. After determining the position of the key field, the field value is obtained through an interception operation, further simplifying the process of data extraction and reducing computing overhead. In addition, the present embodiment allows the processing of data files with different delimiters and field formats, and only the definition of the specific delimiter needs to be changed to adapt to different data format requirements.

[0054] Step S204, obtaining a preset hash function, calculating a hash value for the key field according to the hash function, and storing the calculated hash value in a predefined open addressable hash table;

[0055] In this embodiment, the open address hash table is an array structure for storing hash values ​​and key field values, and the hash conflict is resolved by the open addressing method. In the open address hash table, each table entry contains a hash value, a key field (such as the general ledger account number, the beginning debit balance and the amount of occurrence, etc.) and related statistical data (such as the maximum value, the minimum value, the average value, etc.). The initial size of the open address hash table is pre-set, and a hash function is defined to calculate the hash value of the key field.

[0056] Specifically, for each intercepted key field, a preset hash function is obtained, and the hash function is used to calculate the hash value of the intercepted key field to obtain the calculated hash value. According to the calculated hash value, the corresponding table entry is found in the open addressing hash table. If the table entry is empty, a new table entry is created, and the calculated hash value and key field are stored in the table entry; if a conflict occurs (that is, the hash value is the same but the key field is different), the open addressing method is used to resolve the conflict, and the calculated hash value and key field are stored in the table entry determined by applying the open addressing method.

[0057] In some optional implementations, the above-mentioned performing hash value calculation on the key field according to the hash function may include the following steps:

[0058] Obtaining the data type of the key field;

[0059] A hash function is determined according to the data type, and a hash value is calculated for the key field based on the hash function to obtain a calculated hash value.

[0060] It should be noted that the selection of hash functions needs to take into account factors such as the characteristics of the data (such as the character set of the string, the range of the value), computational efficiency, and the possibility of hash conflicts (i.e., situations where different inputs produce the same output).

[0061] Specifically, the data type of the key field is obtained, and the data type can be an integer (such as int, long), a floating point number (such as float, double), a string (such as char array, string), a date and time, etc. Based on the data type of the key field, the corresponding hash function is matched from a predefined hash function library. Optionally, before the key field is passed to the hash function, it is preprocessed according to the data type of the key field. For example, for a key field of string type, the string is converted to a specific encoding (such as UTF-8); for a key field of numeric type, the numeric value is converted to a binary representation. Afterwards, the preprocessed key field is passed to the selected hash function for calculation, and a hash value (i.e., the calculated hash value) is generated according to the key field through the hash function. The hash value can be an integer or a fixed-length byte sequence.

[0062] The embodiment of the present application can flexibly process different types of data by obtaining the data type of the key field and determining the appropriate hash function according to the data type. Whether it is a numerical value, a string or other complex data type, a suitable hash function can be found for calculation. The appropriate hash function design can minimize the probability of hash conflicts and collisions, which helps to maintain the performance and reliability of the hash table, especially when processing large amounts of data.

[0063] In some optional implementations, the above-mentioned storing the calculated hash value into a predefined open addressable hash table may include the following steps:

[0064] Obtain the subject name of the key field corresponding to the calculated hash value, and search for a corresponding table entry in the open addressable hash table according to the subject name;

[0065] Specifically, for the calculated hash value, determine the key field corresponding to the hash value, obtain the metadata of the key field, and extract the subject name of the key field from the metadata. Obtain a predefined hash function, which is used to receive the subject name as input, and find and return a hash address through open addressing detection. By determining whether the hash address is occupied, it can be determined whether the input subject name has an entry in the open addressing hash table.

[0066] If the entry does not exist, then adding a target entry to the open addressing hash table, and storing the calculated hash value in the target entry;

[0067] Specifically, it is determined whether the hash address found by the open addressing detection is occupied (i.e., whether a valid hash value has been stored) to determine whether the entry exists. If the hash address is not occupied, indicating that the entry does not exist, a target entry is added to the open addressing hash table, and the calculated hash value is stored in the target entry.

[0068] If the entry already exists, the calculated hash value is stored in the entry.

[0069] Specifically, if the hash address is already occupied, indicating that the entry already exists, the calculated hash value is overwritten and stored in the entry.

[0070] The embodiment of the present application searches for the corresponding table entry in the hash table according to the subject name of the key field. This subject name-based search method makes data management more flexible and can perform customized data operations according to different business needs. When a hash conflict occurs in the hash table (i.e., different key fields are mapped to the same hash address), the open addressing method searches for the next available hash address by probing the sequence, thereby effectively solving the conflict problem.

[0071] In some optional implementations, searching for a corresponding entry in the open addressable hash table according to the subject name may include the following steps:

[0072] Obtaining a length value of the subject name, and comparing the length value of the subject name with a preset length threshold;

[0073] Specifically, the length value of the subject name is obtained, and the obtained subject name length is compared with a preset length threshold. This length threshold is preset based on factors such as the characteristics of the hash table, the amount of stored data, and performance requirements. The length threshold can be 8 bytes, 10 bytes, or 16 bytes, etc.

[0074] When the length value of the subject name is less than the preset length threshold, the search strategy of the subject name is determined as a non-loop comparison strategy, and the table entry corresponding to the subject name is matched from the open addressing hash table according to the non-loop comparison strategy;

[0075] Specifically, if the length value of the subject name is less than the preset length threshold, it means that the subject name is relatively short and can be more easily located in the hash table. Therefore, a non-loop comparison strategy is selected as the search strategy. For the non-loop comparison strategy, the address calculated by the hash function is searched in the open addressing hash table. If a table entry matching the subject name is found, the table entry is returned; if a table entry matching the subject name is not found, a limited number of detections are performed according to the characteristics of the hash table (such as linear detection, quadratic detection, etc.) until a matching table entry is found or it is determined that the table entry does not exist.

[0076] When the length value of the subject name is greater than or equal to the preset length threshold, the search strategy for the subject name is determined as a circular comparison strategy, and the table entry corresponding to the subject name is matched from the open addressing hash table according to the circular comparison strategy.

[0077] Specifically, if the length value of the subject name is greater than or equal to the preset length threshold, it means that the subject name is relatively long and multiple comparisons are required in the hash table to find a matching entry. Therefore, a circular comparison strategy is selected as the search strategy. For the circular comparison strategy, multiple circular comparisons are required in the hash table according to the subject name until a matching entry is found or all possible addresses are traversed. During the comparison process, hash conflicts need to be handled according to the detection rules of the hash table.

[0078] The embodiment of the present application can optimize the search performance of the hash table by selecting different search strategies according to the length of the subject name. For shorter subject names, the use of a non-loop comparison strategy can reduce the number of unnecessary comparisons, thereby improving search efficiency. For longer subject names, the use of a loop comparison strategy can ensure that matching items are accurately found in the hash table, even if multiple comparisons are required. At the same time, a reasonable search strategy selection can reduce the occurrence of hash conflicts to a certain extent. For shorter subject names, since the hash function may be more likely to generate evenly distributed hash values, the use of a non-loop comparison strategy can reduce the number of detections caused by hash conflicts. For longer subject names, a loop comparison strategy can search the hash table more comprehensively, thereby reducing the risk of missing matching items due to hash conflicts.

[0079] Step S205 , summarizing the open addressable hash tables corresponding to the data blocks that each thread is responsible for, and performing statistics on each entry in the open addressable hash table to obtain data statistics results of the data file.

[0080] In this embodiment, after all threads have completed reading and processing, the open addressing hash tables corresponding to the data blocks that each thread is responsible for are summarized, and all open addressing hash tables are merged to form a complete hash table, which contains the hash values ​​and statistical data of all key fields. Each table entry in the merged hash table is traversed, and the maximum value, minimum value, average value and other statistical data of each key field are calculated, and the statistical data are stored in a specific data structure to obtain the data statistical results of the data file.

[0081] The embodiment of the present application adopts the above-mentioned scheme, specifically by adopting the memory-mapped file method to read the data file in parallel, it is possible to fully utilize the computing power of a multi-core central processing unit, divide the large data file into multiple data blocks, and assign an independent thread to each data block for reading, thereby greatly improving the speed of data reading through parallel processing, thereby shortening the overall processing time; by accurately locating the starting position and ending position of each key field based on a specific delimiter when reading each row of data in the data block, and obtaining the key field by interception, unnecessary string operations are avoided, and the efficiency and accuracy of field extraction are improved; by calculating the hash value of the key field according to a preset hash function, and storing the calculated hash value in a predefined open addressable hash table, the fast search and insertion performance of the open addressable hash table can be utilized, while ensuring data integrity, further improving data processing efficiency; in the summary stage, each thread merges the processing results of the data block it is responsible for (i.e., the table entries in the open addressable hash table), thereby quickly obtaining the data statistical results of the entire data file, significantly accelerating the data statistical process, and improving the processing efficiency of large data files.

[0082] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned data files, data blocks, open addressable hash tables and other information and other related data to be processed, the above-mentioned data files, data blocks, open addressable hash tables and other information and other related data to be processed can also be stored in a node of a blockchain.

[0083] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0085] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0086] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0087] like Figure 4As shown, the data processing device 400 described in this embodiment includes: an acquisition module 401, a reading module 402, a separation module 403, a calculation module 404 and a statistics module 405. Among them:

[0088] The acquisition module 401 is used to acquire a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters;

[0089] A reading module 402 is used to read the data file in parallel by using a memory mapping file method, divide the data file into a plurality of data blocks, and set a corresponding thread for each data block to read;

[0090] A separation module 403 is used to locate the starting position and the ending position of each key field in each line of data based on the specific separator when reading each line of data in the data block, and to obtain the key field according to the starting position and the ending position by interception;

[0091] The calculation module 404 is used to obtain a preset hash function, calculate the hash value of the key field according to the hash function, and store the calculated hash value in a predefined open addressable hash table;

[0092] The statistics module 405 is used to summarize the open addressable hash table corresponding to the data block responsible for each thread, and to perform statistics on each table entry in the open addressable hash table to obtain the data statistics result of the data file.

[0093] Specifically, the acquisition module 401 is used to obtain a data file to be processed, which is a text file containing multiple lines of data, that is, the data in the data file is stored in rows, each line of data represents a record, and the record can contain multiple fields. Among them, specific delimiters are used to isolate each field. The specific delimiter can include a comma, a semicolon, a tab, a space or a custom character, etc., which are used to separate different fields in each line of data. For example, the name of the data file is "account data file.txt", and the content contained in the data file is: general ledger accounting account name, general ledger accounting account number, beginning debit balance, amount of occurrence, etc.

[0094] Specifically, the memory-mapped file method refers to using memory-mapped file technology to map data files into memory for efficient access and processing of data.

[0095] Specifically, the reading module 402 is used to map the data file into the memory using a memory-mapped file method for fast access; according to the number of cores of the central processing unit, the data file is divided into multiple data blocks, and a corresponding independent thread is assigned to each data block for data reading to achieve parallel processing.

[0096] Specifically, the separation module 403 is used to traverse each row of data in the data block that the thread is responsible for in each thread. Based on a specific separator, the starting position and the ending position of each key field in each row of data are located. The key fields are extracted according to the located starting position and the ending position by using an interception method. For example, according to a specific separator (such as a comma or a semicolon), the starting position and the ending position of each key field in each row of data are located, and the key fields such as the general ledger accounting subject number, the beginning debit balance and the amount of occurrence in each row of data are extracted according to the starting position and the ending position.

[0097] Specifically, an open addressing hash table is an array structure used to store hash values ​​and key field values, and hash conflicts are resolved through open addressing. In the open addressing hash table, each entry contains a hash value, key fields (such as general ledger account number, opening debit balance and amount), and related statistical data (such as maximum value, minimum value, average value, etc.). The initial size of the open addressing hash table is pre-set, and a hash function is defined to calculate the hash value of the key field.

[0098] Specifically, the calculation module 404 is used to obtain a preset hash function for each intercepted key field, and use the hash function to calculate the hash value of the intercepted key field to obtain the calculated hash value. According to the calculated hash value, the corresponding table entry is found in the open addressing hash table. If the table entry is empty, a new table entry is created, and the calculated hash value and key field are stored in the table entry; if a conflict occurs (that is, the hash value is the same but the key field is different), the open addressing method is used to resolve the conflict, and the calculated hash value and key field are stored in the table entry determined by applying the open addressing method.

[0099] Specifically, the statistical module 405 is used to summarize the open addressing hash tables corresponding to the data blocks responsible for each thread after all threads have completed reading and processing, and merge all open addressing hash tables to form a complete hash table, which contains the hash values ​​and statistical data of all key fields. It traverses each table entry in the merged hash table, calculates the maximum value, minimum value, average value and other statistical data of each key field, stores the statistical data in a specific data structure, and obtains the data statistical results of the data file.

[0100] The data processing device 400 of the embodiment of the present application, by adopting a memory-mapped file method to read data files in parallel, can make full use of the computing power of a multi-core central processing unit, divide a large data file into multiple data blocks, and assign an independent thread to each data block for reading, thereby greatly improving the speed of data reading through parallel processing, thereby shortening the overall processing time; by accurately locating the starting position and ending position of each key field based on a specific delimiter when reading each row of data in the data block, and obtaining the key field by interception, unnecessary string operations are avoided, and the efficiency and accuracy of field extraction are improved; by calculating the hash value of the key field according to a preset hash function, and storing the calculated hash value in a predefined open addressable hash table, the fast search and insertion performance of the open addressable hash table can be utilized to further improve the data processing efficiency while ensuring data integrity; in the aggregation stage, each thread merges the processing results of the data block it is responsible for (i.e., the table entries in the open addressable hash table), thereby quickly obtaining the data statistical results of the entire data file, significantly accelerating the data statistical process, and improving the processing efficiency of large data files.

[0101] In some optional implementations of this embodiment, the reading module 402 may include a mapping submodule and a slicing submodule.

[0102] A mapping submodule, used to map the content of the data file into the memory to form a memory mapped file by using a memory mapped file method;

[0103] The sub-slicing module is used to divide the memory-mapped file into multiple data blocks according to the size of the data file.

[0104] Specifically, the mapping submodule is used to map the content of the data file (i.e., the fields recorded in each row of data) into the memory by using a memory-mapped file method to form a memory-mapped file for fast access. The slicing submodule is used to obtain the size of the data file, and according to the size of the data file and the number of cores of the central processing unit, the memory-mapped file is sliced ​​into multiple data blocks, and a corresponding independent thread is assigned to each data block for data reading to achieve parallel processing.

[0105] The reading module 402 of the embodiment of the present application directly maps the content of the data file to the memory by adopting the memory mapping file method to form a memory mapping file, thereby avoiding the frequent disk reading and writing of traditional file I / O operations and significantly improving the data access speed; since the memory mapping file allows the program to access the file content like accessing the memory, it can reduce the number of system calls and reduce the context switching overhead, thereby improving the overall efficiency of data reading. By dividing the memory mapping file into multiple data blocks according to the size of the data file, and assigning a corresponding independent thread to each data block for data reading, the parallel processing capability of the multi-core processor is fully utilized, and the parallel reading and processing of data is realized, which significantly improves the speed and throughput of data processing.

[0106] In some optional implementations of this embodiment, the separation module 403 may include a search submodule and an interception submodule.

[0107] A search submodule, used to traverse from the beginning of the row of the data block and determine the starting position and the ending position of each key field in each row of data by searching for the specific delimiter;

[0108] The interception submodule is used to intercept and extract the key field from the data block according to the starting position and the ending position of the key field to obtain the key field.

[0109] Specifically, the search submodule is used to traverse from the beginning of the row of the data block (i.e., the starting position of the first row), that is, starting from the first character or byte of the data block, and gradually checking each character or byte until the end of the data block. During the traversal process, a specific delimiter is continuously searched, and the specific delimiter is used to distinguish different fields in each row of data, such as a comma (,), a tab (\t), a space, or a custom character. When a specific delimiter is found, the position of the specific delimiter is recorded, which can be the end position of the key field or the starting position of the next key field (depending on the specific definition of the delimiter and the format of the data). The interception submodule is used to truncate and extract the corresponding key field from the data block according to the determined starting position and ending position. For example, the interception of the key field is achieved by copying or moving the characters or bytes between the starting position and the ending position to a new memory area.

[0110] The separation module 403 of the present application embodiment, by traversing from the beginning of the row of the data block, and searching for a specific separator, can accurately determine the starting position and the ending position of each field in each row of data, avoiding complicated regular expression matching or parser design, thereby improving the efficiency of data parsing. By accurately locating the starting position and the ending position of the field, it is possible to process a data file with a variable length field without the need for additional parsing logic. After determining the position of the key field, the field value is obtained by intercepting an operation, further simplifying the process of data extraction, and reducing computational overhead. In addition, the present embodiment allows processing of data files with different separators and field formats, and only the definition of the specific separator needs to be changed to adapt to different data format requirements.

[0111] In some optional implementations of this embodiment, the calculation module 404 may include an acquisition submodule and a calculation submodule.

[0112] An acquisition submodule, used to acquire the data type of the key field;

[0113] The calculation submodule is used to determine a hash function according to the data type, and perform hash value calculation on the key field based on the hash function to obtain a calculated hash value.

[0114] It should be noted that the selection of hash functions needs to take into account factors such as the characteristics of the data (such as the character set of the string, the range of the value), computational efficiency, and the possibility of hash conflicts (i.e., situations where different inputs produce the same output).

[0115] Specifically, the acquisition submodule is used to obtain the data type of the key field, which can be an integer (such as int, long), a floating point number (such as float, double), a string (such as char array, string), a date and time, etc. Based on the data type of the key field, the corresponding hash function is matched from a predefined hash function library. Optionally, before the key field is passed to the hash function, it is preprocessed according to the data type of the key field. For example, for a key field of string type, the string is converted into a specific encoding (such as UTF-8); for a key field of numeric type, the numeric value is converted into a binary representation. Afterwards, the calculation submodule is used to pass the preprocessed key field to the selected hash function for calculation, and a hash value (i.e., the calculated hash value) is generated according to the key field through the hash function. The hash value can be an integer or a fixed-length byte sequence.

[0116] The calculation module 404 of the embodiment of the present application can flexibly process different types of data by obtaining the data type of the key field and determining the appropriate hash function according to the data type. Whether it is a numerical value, a string or other complex data type, a suitable hash function can be found for calculation. The appropriate hash function design can minimize the probability of hash conflicts and collisions, which helps to maintain the performance and reliability of the hash table, especially when processing large amounts of data.

[0117] In some optional implementations of this embodiment, the calculation module 404 includes a table entry search submodule, a table entry new addition submodule, and a table entry storage submodule.

[0118] A table entry search submodule, used to obtain the subject name of the key field corresponding to the calculated hash value, and search the corresponding table entry in the open addressable hash table according to the subject name;

[0119] A table entry adding submodule, used for adding a target table entry to the open addressing hash table if the table entry does not exist, and storing the calculated hash value in the target table entry;

[0120] The table entry storage submodule is used to store the calculated hash value into the table entry if the table entry already exists.

[0121] Specifically, the table entry search submodule is used to determine the key field corresponding to the calculated hash value, obtain the metadata of the key field, and extract the subject name of the key field from the metadata. A predefined hash function is obtained, which is used to receive the subject name as input, and find and return a hash address through open addressing detection. By determining whether the hash address is occupied, it can be determined whether the input subject name has an entry in the open addressing hash table.

[0122] Specifically, the entry adding submodule is used to determine whether the hash address found by the open addressing detection is occupied (i.e., whether a valid hash value has been stored) to determine whether the entry exists. If the hash address is not occupied, indicating that the entry does not exist, a target entry is added to the open addressing hash table, and the calculated hash value is stored in the target entry.

[0123] Specifically, the table entry storage submodule is used to overwrite and store the calculated hash value into the table entry if the hash address is already occupied, indicating that the table entry already exists.

[0124] The calculation module 404 of the embodiment of the present application searches for the corresponding table entry in the hash table according to the subject name of the key field. This search method based on the subject name makes data management more flexible and can perform customized data operations according to different business needs. When a hash conflict occurs in the hash table (i.e., different key fields are mapped to the same hash address), the open addressing method searches for the next available hash address by probing the sequence, thereby effectively solving the conflict problem.

[0125] In some optional implementations of this embodiment, the above-mentioned table entry search submodule may include a numerical comparison unit, a first matching unit and a second matching unit.

[0126] A numerical comparison unit, used to obtain the length value of the subject name, and compare the length value of the subject name with a preset length threshold;

[0127] A first matching unit, configured to determine the search strategy for the subject name as a non-loop comparison strategy when the length value of the subject name is less than the preset length threshold, and match the table entry corresponding to the subject name from the open addressing hash table according to the non-loop comparison strategy;

[0128] The second matching unit is used to determine the search strategy of the subject name as a circular comparison strategy when the length value of the subject name is greater than or equal to the preset length threshold, and match the table entry corresponding to the subject name from the open addressing hash table according to the circular comparison strategy.

[0129] Specifically, the value comparison unit is used to obtain the length value of the subject name, and compare the obtained subject name length with a preset length threshold. This length threshold is preset based on factors such as the characteristics of the hash table, the amount of stored data, and performance requirements. The length threshold can be 8 bytes, 10 bytes, or 16 bytes, etc.

[0130] Specifically, the first matching unit is used to find out that if the length value of the subject name is less than a preset length threshold, it means that the subject name is relatively short and can be more easily and quickly located in the hash table. Therefore, a non-loop comparison strategy is selected as the search strategy. For the non-loop comparison strategy, the address calculated by the hash function is searched in the open addressing hash table. If a table entry matching the subject name is found, the table entry is returned; if a table entry matching the subject name is not found, a limited number of detections are performed according to the characteristics of the hash table (such as linear detection, quadratic detection, etc.) until a matching table entry is found or it is determined that the table entry does not exist.

[0131] Specifically, the second matching unit is used to check if the length value of the subject name is greater than or equal to the preset length threshold, indicating that the subject name is relatively long and multiple comparisons are required in the hash table to find a matching entry. Therefore, a circular comparison strategy is selected as the search strategy. For the circular comparison strategy, multiple circular comparisons need to be performed in the hash table according to the subject name until a matching entry is found or all possible addresses are traversed. During the comparison process, hash conflicts need to be handled according to the detection rules of the hash table.

[0132] The table item search submodule of the embodiment of the present application can optimize the search performance of the hash table by selecting different search strategies according to the length of the subject name. For shorter subject names, the use of a non-loop comparison strategy can reduce the number of unnecessary comparisons, thereby improving search efficiency. For longer subject names, the use of a loop comparison strategy can ensure that matching table items are accurately found in the hash table, even if multiple comparisons are required. At the same time, a reasonable search strategy selection can reduce the occurrence of hash conflicts to a certain extent. For shorter subject names, since the hash function may be more likely to generate evenly distributed hash values, the use of a non-loop comparison strategy can reduce the number of detections caused by hash conflicts. For longer subject names, a loop comparison strategy can search the hash table more comprehensively, thereby reducing the risk of missing matching table items due to hash conflicts.

[0133] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0134] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0135] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0136] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of data processing methods, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0137] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running the data processing method.

[0138] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0139] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data processing method as described above.

[0140] The computer device, computer-readable storage medium and computer-readable instructions thereof provided in the embodiment of the present application can read data files in parallel by using a memory-mapped file method through the processor execution, and can make full use of the computing power of a multi-core central processing unit to divide a large data file into multiple data blocks, and assign an independent thread to each data block for reading, thereby greatly improving the speed of data reading through parallel processing, thereby shortening the overall processing time; by accurately locating the starting position and ending position of each key field based on a specific delimiter when reading each row of data in the data block, and obtaining the key field by interception, unnecessary string operations are avoided, and the efficiency and accuracy of field extraction are improved; by calculating the hash value of the key field according to a preset hash function, and storing the calculated hash value in a predefined open addressable hash table, the fast search and insertion performance of the open addressable hash table can be utilized to further improve the data processing efficiency while ensuring data integrity; in the summary stage, each thread merges the processing results of the data block it is responsible for (i.e., the table entries in the open addressable hash table), thereby quickly obtaining the data statistical results of the entire data file, significantly accelerating the data statistical process, and improving the processing efficiency of large data files.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0142] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0143] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A data processing method, characterized in that: The steps include: Obtain a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters; The data file is read in parallel by using a memory mapping file method, the data file is divided into a plurality of data blocks, and a corresponding thread is set for each data block to read; When reading each line of data in the data block, the starting position and the ending position of each key field in each line of data are located based on the specific delimiter, and the key field is obtained according to the starting position and the ending position by interception; Obtaining a preset hash function, calculating a hash value for the key field according to the hash function, and storing the calculated hash value in a predefined open addressable hash table; The open addressable hash tables corresponding to the data blocks that each thread is responsible for are summarized, and statistics are performed on each table entry in the open addressable hash table to obtain data statistics results of the data file.

2. The data processing method according to claim 1, characterized in that: The step of reading the data file in parallel by using a memory mapping file method and dividing the data file into a plurality of data blocks comprises: Using a memory-mapped file method, the content of the data file is mapped into the memory to form a memory-mapped file; According to the size of the data file, the memory-mapped file is divided into a plurality of data blocks.

3. The data processing method according to claim 1, characterized in that: The step of locating the starting position and the ending position of each key field in each row of data based on the specific delimiter, and obtaining the key field according to the starting position and the ending position by interception includes: Traversing from the beginning of the row of the data block, determining the starting position and the ending position of each key field in each row of data by searching for the specific delimiter; According to the starting position and the ending position of the key field, the key field is cut off and extracted from the data block to obtain the key field.

4. The data processing method according to claim 1, characterized in that: The step of calculating the hash value of the key field according to the hash function includes: Obtaining the data type of the key field; A hash function is determined according to the data type, and a hash value is calculated for the key field based on the hash function to obtain a calculated hash value.

5. The data processing method according to claim 1, characterized in that: The step of storing the calculated hash value into a predefined open addressable hash table comprises: Obtain the subject name of the key field corresponding to the calculated hash value, and search for a corresponding table entry in the open addressable hash table according to the subject name; If the entry does not exist, then adding a target entry to the open addressing hash table, and storing the calculated hash value in the target entry; If the entry already exists, the calculated hash value is stored in the entry.

6. The data processing method according to claim 5, characterized in that: The step of searching for a corresponding entry in the open addressable hash table according to the subject name comprises: Obtaining a length value of the subject name, and comparing the length value of the subject name with a preset length threshold; When the length value of the subject name is less than the preset length threshold, the search strategy of the subject name is determined as a non-loop comparison strategy, and the table entry corresponding to the subject name is matched from the open addressing hash table according to the non-loop comparison strategy; When the length value of the subject name is greater than or equal to the preset length threshold, the search strategy for the subject name is determined as a circular comparison strategy, and the table entry corresponding to the subject name is matched from the open addressing hash table according to the circular comparison strategy.

7. A data processing device, characterized in that: The device comprises: An acquisition module is used to acquire a data file to be processed, wherein the data in the data file is stored in rows, each row of data contains multiple fields, and the fields are separated by specific delimiters; A reading module is used to read the data file in parallel by using a memory mapping file method, divide the data file into multiple data blocks, and set a corresponding thread for each data block to read; A separation module, used for locating the starting position and the ending position of each key field in each line of data based on the specific separator when reading each line of data in the data block, and obtaining the key field according to the starting position and the ending position by interception; A calculation module, used to obtain a preset hash function, calculate a hash value for the key field according to the hash function, and store the calculated hash value in a predefined open addressable hash table; The statistics module is used to summarize the open addressable hash tables corresponding to the data blocks that each thread is responsible for, and to perform statistics on each table entry in the open addressable hash table to obtain the data statistics result of the data file.

8. The data processing device according to claim 7, characterized in that: The calculation module comprises: An acquisition submodule, used to acquire the data type of the key field; The calculation submodule is used to determine a hash function according to the data type, and perform hash value calculation on the key field based on the hash function to obtain a calculated hash value.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data processing method according to any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images