Data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium. The method comprises: performing mapping processing on to-be-processed data by using a mapping unit to obtain mapping output data; in response to the mapping output data reaching a predetermined number, performing sorting on the mapping output data, encapsulating the sorted mapping output data to obtain a mapping data packet; generating, by using a service unit, a mapping data file comprising a predetermined number of mapping data packets in the order in which the mapping data packets are obtained; obtaining, by using a reduction unit, the mapping data file, dividing the mapping data file into a predetermined number of mapping data packets; performing analysis on each mapping data packet to obtain the sorted mapping output data corresponding to each mapping data packet; and performing sorting on the sorted mapping output data to obtain a data processing result. The mapping unit and the reduction unit perform data sorting, and the service unit does not perform sorting, thereby ensuring the performance and response speed of the service unit.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] MapReduce is a batch computing framework primarily used for processing large amounts of static data. It generally includes three processes: Map, Shuffle, and Reduce. Figure 1 As shown, the data processing generally proceeds as follows: the Map unit (map task) converts the input key-value pairs into multiple key-value pairs; the Shuffle unit sorts and merges these pairs; and key-value pairs with the same key are sent to the same Reduce unit (reduce task). The Reduce unit then converts these pairs into a single key-value pair according to certain rules. The Shuffle unit generates a large number of read requests, leading to a decrease in system performance. Using a remote shuffle service framework, since the Shuffle unit does not perform sorting or other processing, can effectively reduce the number of read requests and improve system performance. However, how to implement sorting within the remote shuffle service framework is a problem that needs to be solved. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a data processing method, apparatus, electronic device and storage medium.

[0004] To achieve the above objectives, this application provides a data processing method, comprising:

[0005] The mapping unit performs mapping processing on the data to be processed to obtain mapped output data; in response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is encapsulated to obtain a mapping data packet;

[0006] The service unit generates a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are acquired.

[0007] The mapping data file is obtained using a reduction unit, and the mapping data file is divided into a predetermined number of mapping data packets; each mapping data packet is parsed to obtain sorted mapping output data corresponding to each mapping data packet; and the sorted mapping output data is sorted to obtain the data processing result.

[0008] Optionally, the mapping data packet includes identification information for identifying the mapping unit; parsing each mapping data packet to obtain sorted mapping output data corresponding to each mapping data packet includes:

[0009] Each mapping data packet is parsed to obtain the identification information and the sorted mapping output data; wherein the sorted mapping output data is generated by the mapping unit corresponding to the identification information;

[0010] Before sorting the sorted mapping output data, the process also includes:

[0011] Based on the identification information, the sorted mapping output data is deduplicated to obtain deduplicated mapping output data.

[0012] Sort the sorted mapping output data, including:

[0013] The deduplication-processed mapping output data is sorted.

[0014] Optionally, the identification information includes a mapping unit identifier, a mapping fault tolerance identifier, and a data packet identifier;

[0015] Based on the identification information, the sorted mapping output data is deduplicated to obtain deduplicated mapping output data, including:

[0016] Based on the mapping unit identifier and the mapping fault tolerance identifier, the sorted mapping output data is deduplicated. For mapping data packets with the same mapping unit identifier corresponding to multiple mapping fault tolerance identifiers, the mapping data packet with one mapping fault tolerance identifier is retained to obtain the first deduplicated mapping output data.

[0017] Based on the mapping unit identifier, the mapping fault tolerance identifier, and the data packet identifier, the mapping output data after the first deduplication is deduplicated. For mapping data packets with the same mapping unit identifier and the same mapping fault tolerance identifier corresponding to multiple data packet identifiers, the mapping data packet with one data packet identifier is retained to obtain the mapping output data after the second deduplication.

[0018] Optionally, the mapping data includes compressed length and original length; parsing each mapping data packet yields sorted mapping output data corresponding to each mapping data packet, including:

[0019] Each mapped data packet is parsed to obtain the compressed length and the original length;

[0020] Based on the compression length, the compressed mapping output data is obtained by parsing.

[0021] Before sorting the sorted mapping output data, the process also includes:

[0022] Allocate storage space for the original length;

[0023] The compressed mapping output data is decompressed to obtain decompressed mapping output data;

[0024] The decompressed mapping output data is stored in the storage space;

[0025] Sort the sorted mapping output data, including:

[0026] The mapped output data in the storage space is sorted.

[0027] Optionally, sorting the mapped output data in the storage space includes:

[0028] In response to the parsed compressed mapping output data reaching a predetermined number of sorting values, or the decompressed mapping output data occupying a predetermined amount of storage space, the mapping output data is sorted.

[0029] Optionally, the sorted mapping output data is encapsulated to obtain a mapping data packet, including:

[0030] The sorted mapping output data is compressed to obtain compressed mapping output data;

[0031] Based on the compressed mapping output data, a data header is added to form the mapping data packet; wherein, the data header includes identification information, the original length of the mapping output data, and the compressed length of the compressed mapping output data.

[0032] Optionally, the sorted mapping output data is sorted to obtain data processing results, including:

[0033] The sorted mapping output data is then merged and sorted to obtain the data processing result.

[0034] This application also provides a data processing apparatus, including:

[0035] The mapping unit is used to perform mapping processing on the data to be processed to obtain mapped output data; in response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is encapsulated to obtain a mapping data packet.

[0036] The service unit is used to generate a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are acquired.

[0037] The reduction unit is used to acquire the mapping data file; divide the mapping data file into the predetermined number of mapping data packets; parse each mapping data packet to obtain the sorted mapping output data corresponding to each mapping data packet; and sort the sorted mapping output data to obtain the data processing result.

[0038] Based on the same concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of the above descriptions.

[0039] Based on the same concept, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods described above.

[0040] As can be seen from the above description, the data processing method, apparatus, electronic device, and storage medium provided in this application utilize a mapping unit to map the data to be processed, obtaining mapped output data; in response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is encapsulated to obtain a mapping data packet; a service unit generates a mapping data file containing a predetermined number of mapping data packets according to the order in which the mapping data packets are obtained; a reduction unit obtains the mapping data file and divides the mapping data file into a predetermined number of mapping data packets; each mapping data packet is parsed to obtain the sorted mapping output data corresponding to each mapping data packet; and the sorted mapping output data is sorted to obtain the data processing result. This application utilizes the resources of the mapping unit and the reduction unit themselves to sort the data, without requiring the service unit to participate in the sorting, thus ensuring the performance and response speed of the service unit while implementing the data sorting function. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the MapReduce data processing procedure in some embodiments;

[0043] Figure 2 This is a schematic diagram illustrating an application scenario of an embodiment of this application;

[0044] Figure 3This is a schematic diagram of the method flow of an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the data format of the mapping data packet in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the device structure according to an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0048] The principles and spirit of this application will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0049] According to embodiments of this application, a data processing method, apparatus, electronic device, and storage medium are proposed.

[0050] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0051] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments.

[0052] In related technologies, within the MapReduce batch computing framework, if the number of Map units is M and the number of Reduce units is R, then the shuffle phase will generate M×R read requests. This large number of read requests severely impacts system performance. To reduce the number of read requests, a remote shuffle service is used to aggregate the output data of the Map units, and then the Reduce units read the aggregated data from the remote shuffle service, thereby reducing the number of read requests. However, using a remote shuffle service requires addressing the sorting problem in MapReduce. Some technologies utilize a remote shuffle service to sort the output data of the Map units, and the Reduce units directly read the sorted data. While this achieves sorting, it consumes a significant amount of computational resources from the remote shuffle service. Furthermore, the Reduce units must wait for the remote shuffle service to complete the sorting process before they can obtain the sorted data, reducing the response speed of the remote shuffle service and lowering processing performance.

[0053] To address the aforementioned issues, this application provides a data processing method that sorts and encapsulates the output data of the Map unit, sends the encapsulated data packets to a remote shuffle service, the remote shuffle service organizes the data packets into data files, the Reduce unit obtains the data files from the remote shuffle service, parses the data packets in the data files, and sorts the data based on the parsed data, thereby achieving MapReduce sorting without affecting the processing performance of the remote shuffle service.

[0054] refer to Figure 2This diagram illustrates an application scenario of the data processing method provided in this application. The application scenario includes a Map unit 10 (hereinafter referred to as the mapping unit), a remote shuffle service 20 (hereinafter referred to as the service unit), and a Reduce unit 30 (hereinafter referred to as the reduction unit). The mapping unit processes the input raw data according to a predetermined mapping method to obtain key-value pairs. Key-value pairs with the same key are stored in the same cache. When the cache is full of key-value pairs, the data in the cache is sent to the service unit as mapping output data. The service unit receives the mapping output data from each mapping unit and organizes the key-value pairs with the same key from each mapping unit into a complete mapping data file. The reduction unit, used to process key-value pairs with specific keys, obtains the corresponding mapping data file from the service unit and performs reduction processing on the obtained data according to a predetermined reduction method, thereby realizing batch processing of large amounts of data.

[0055] The following is combined Figure 2 The application scenarios described above illustrate the data processing method according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this regard. Rather, the embodiments of this application can be applied to any applicable scenario.

[0056] First, such as Figure 3 As shown in the figure, this application provides a data processing method, including:

[0057] S301: The mapping unit performs mapping processing on the data to be processed to obtain the mapping output data; in response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is encapsulated to obtain the mapping data packet;

[0058] In this embodiment, the mapping unit generates key-value pairs after mapping the data to be processed according to a predetermined mapping method. Key-value pairs with the same key are stored in the same buffer. When the amount of data in the buffer reaches a predetermined sending quantity, the data in the buffer is used as the mapping output data, sorted, and then encapsulated into a mapping data packet before being sent to the service unit. Optionally, when the data in the buffer is full or the amount of stored data reaches a certain proportion of the data that the buffer can store, sorting and encapsulation into a mapping data packet are performed. By sorting a portion of the data in the mapping unit first, data processing efficiency can be improved.

[0059] S302: The service unit generates a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are obtained;

[0060] In this embodiment, the service unit receives mapping data packets from each mapping unit, organizes key-value pairs with the same key into a complete mapping data file, and generates a mapping data file containing a predetermined number of mapping data packets. The service unit does not need to sort the mapping output data, thus not affecting its performance.

[0061] S303: Use the reduction unit to obtain the mapping data file, divide the mapping data file into a predetermined number of mapping data packets; parse each mapping data packet to obtain the sorted mapping output data corresponding to each mapping data packet; and sort the sorted mapping output data to obtain the data processing result.

[0062] In this embodiment, the reduction unit obtains the mapping data file from the service unit, divides the mapping data file into a predetermined number of mapping data packets, and then parses each mapping data packet according to the encapsulation format of the mapping data packet to obtain the mapping output data in the mapping data packet after being sorted by the mapping unit. After that, the reduction unit sorts the mapping output data in each mapping data packet to obtain the data processing result.

[0063] The data processing method provided in this embodiment involves a mapping unit that maps the data to be processed to obtain mapped output data. When the mapped output data reaches a certain quantity, it sorts the data and encapsulates it into mapped data packets, which are then sent to the service unit. The service unit organizes a certain number of mapped data packets into a mapped data file according to the order in which they are received. The reduction unit obtains the mapped data file from the service unit, divides it into multiple mapped data packets, parses each packet to obtain the mapped output data sorted by the mapping unit, sorts the parsed data, and then performs a reduction process based on the sorted data to obtain the final sorted output data. The desired data processing result is then obtained based on this final sorted output data using a specific reduction method. Each mapping unit and reduction unit utilizes its own computing resources for sorting, which does not affect the performance or response speed of the service unit. The service unit also does not need to configure additional resources for sorting, thus achieving both the sorting function and ensuring system performance.

[0064] In some embodiments, the mapping data packet includes identification information for identifying the mapping unit; parsing each mapping data packet to obtain sorted mapping output data corresponding to each mapping data packet includes:

[0065] Each mapping data packet is parsed to obtain identification information and sorted mapping output data; the sorted mapping output data is generated by the mapping unit corresponding to the identification information.

[0066] Before sorting the sorted mapping output data, the following steps are also included:

[0067] Based on the identification information, the sorted mapping output data is deduplicated to obtain the deduplicated mapping output data.

[0068] Sort the sorted mapping output data, including:

[0069] Sort the deduplicated mapping output data.

[0070] In this embodiment, the mapping data packet encapsulated by the mapping unit includes identification information and mapping output data sorted by the mapping unit. After the reduction unit parses out the identification information and mapping output data, it performs deduplication on the received mapping output data according to the identification information, deletes duplicate data, and retains only one copy of the corresponding mapping output data for the same key value pair of the same mapping unit for subsequent sorting processing, thereby saving the resources required for data processing and improving the efficiency and accuracy of data processing results.

[0071] In some embodiments, the identification information includes a mapping unit identifier, a mapping fault tolerance identifier, and a data packet identifier;

[0072] Based on the identification information, the sorted mapping output data is deduplicated to obtain the deduplicated mapping output data, including:

[0073] Based on the mapping unit identifier and the mapping fault tolerance identifier, the sorted mapping output data is deduplicated. For mapping data packets with the same mapping unit identifier corresponding to multiple mapping fault tolerance identifiers, the mapping data packet with one mapping fault tolerance identifier is retained, and the mapping output data after the first deduplication is obtained.

[0074] Based on the mapping unit identifier, mapping fault tolerance identifier, and data packet identifier, the mapping output data after the first deduplication is deduplicated. For mapping data packets with the same mapping unit identifier and the same mapping fault tolerance identifier corresponding to multiple data packet identifiers, the mapping data packet with one data packet identifier is retained to obtain the mapping output data after the second deduplication.

[0075] In this embodiment, the identification information added to the mapping data packet encapsulated by the mapping unit includes a mapping unit identifier, a mapping fault tolerance identifier, and a data packet identifier. The mapping unit identifier identifies the mapping unit that generates the mapping output data and encapsulates it. The mapping fault tolerance identifier identifies the mapping output data generated by the mapping unit based on the fault tolerance mechanism and the sequence number of the encapsulated mapping output data. MapReduce's fault tolerance mechanism addresses situations where a task fails due to a mapping unit malfunction, process crash, or mapping error, requiring the mapping unit to re-execute the mapping process to generate the mapping output data. To ensure system reliability, based on the fault tolerance mechanism, the same mapping unit may generate one or more sets of mapping output data. The data packet identifier identifies the sequence number of the mapping data packet. Due to the fault tolerance mechanism and network transmission factors, the mapping unit may encapsulate data packets multiple times; the data packet identifier sequence number increments by one each time the same mapping output data is encapsulated.

[0076] The reduction unit parses the identification information and performs a first deduplication process based on the mapping unit identifier and the mapping fault tolerance identifier. For mapping data packets with multiple mapping fault tolerance identifiers under the same mapping unit identifier, only the mapping data packet with one mapping fault tolerance identifier is retained. Then, a second deduplication process is performed based on the mapping unit identifier, the mapping fault tolerance identifier, and the data packet identifier. For mapping data packets with the same mapping unit identifier and the same mapping fault tolerance identifier, only the mapping data packet with one data packet identifier is retained. Through deduplication, for the same set of mapping output data from the same mapping unit, only one set is retained for subsequent processing, eliminating duplicate data caused by repeated generation and retransmission, avoiding wasted computational resources, and improving data processing efficiency.

[0077] In some embodiments, the mapping data includes a compressed length and an original length; parsing each mapping data packet yields sorted mapping output data corresponding to each mapping data packet, including:

[0078] Parse each mapped data packet to obtain the compressed length and the original length;

[0079] Based on the compression length, the compressed mapping output data is obtained through parsing.

[0080] Before sorting the sorted mapping output data, the following steps are also included:

[0081] Allocate storage space for the original length;

[0082] Decompress the compressed mapping output data to obtain the decompressed mapping output data;

[0083] The decompressed mapped output data is stored in the storage space;

[0084] Sort the sorted mapping output data, including:

[0085] Sort the mapped output data in the storage space.

[0086] In this embodiment, to improve data transmission efficiency and resource utilization, the mapping unit compresses the mapping output data to generate compressed mapping output data. Then, it encapsulates mapping data packets based on the compressed mapping output data. The mapping data packets need to have compressed length and original length fields added. The compressed length identifies the length of the compressed mapping output data, and the original length identifies the length of the uncompressed mapping output data. The reduction unit parses the mapping data packets to obtain the compressed length and original length. First, it allocates corresponding storage space based on the original length. Then, it extracts the compressed mapping output data from the mapping data packets based on the compressed length, decompresses this portion of the mapping output data, and saves the decompressed mapping output data in the allocated storage space. Subsequently, the mapping output data in the storage space is directly sorted.

[0087] In some embodiments, sorting the mapped output data in the storage space includes:

[0088] In response to the parsed compressed mapping output data reaching a predetermined number of sorting values, or the decompressed mapping output data occupying a predetermined amount of storage space, the mapping output data is sorted.

[0089] In this embodiment, the reduction unit obtains the mapping data file to be processed from the service unit, divides the mapping data file into multiple mapping data packets, and parses each mapping data packet to obtain the mapping output data corresponding to each mapping data packet. Due to the limitations of the reduction unit's own resources, if it waits for all mapping data packets to be parsed before sorting, it may consume too many computing resources and require the allocation of too many storage resources. To improve resource utilization and processing efficiency, the reduction unit can determine the timing of sorting based on its own resource situation. For example, when the parsed mapping output data reaches a predetermined number of sorting operations, or when the storage space occupied by the decompressed mapping output data has reached a predetermined storage amount, the sorting processing of the mapping output data in the storage space begins.

[0090] In some methods, the reduction unit sorts the mapped output data. This can be done by first sorting the mapped output data in each storage space separately, and then, when the number of mapped output data reaches a predetermined number, or when the allocated storage space reaches a predetermined storage capacity, merging and sorting the mapped output data in all storage spaces to complete the sorting process. After sorting, the corresponding storage space can be released to provide storage resources for subsequent mapped output data.

[0091] Figure 4 The diagram illustrates the data format for encapsulating mapping output data by the mapping unit. In one embodiment, the encapsulated mapping data packet includes a data header and a data portion. The data header includes a mapping unit identifier (Mapper id), a mapping fault tolerance identifier (Mapper attempt id), a data packet identifier (Packet id), a compressed length (Compressed len), and a raw length (Raw len). The data portion (Data) is the compressed mapping output data. The reduction unit parses the mapping data packet to obtain the mapping unit identifier, mapping fault tolerance identifier, data packet identifier, compressed length, and raw length. First, it performs deduplication processing on the mapping data packet based on the mapping unit identifier, mapping fault tolerance identifier, and data packet identifier, removing duplicate data. After deduplication, it allocates storage space based on the raw length, extracts the data portion (compressed mapping output data) from the mapping data packet based on the compressed length, and decompresses the compressed mapping output data. The decompressed mapping output data is then stored in the storage space. Subsequently, the mapping output data in the storage space is sorted. When the occupied storage space reaches a certain amount or the received and parsed mapping output data reaches a certain amount, the mapping output data in all storage spaces is sorted as a whole, thereby achieving the sorting function.

[0092] In some methods, the length of the mapping data packet can be fixed or variable. For fixed-length mapping data packets, the mapping unit encapsulates the mapping data packet according to the fixed length, and the reduction unit divides all mapping data packets from the mapping data file according to the fixed length. For variable-length mapping data packets, the length of the data header is fixed, while the data portion is variable. The mapping unit compresses the mapped output data to determine the compressed length, and the reduction unit parses the mapping data packet, determines the length of the data portion based on the compressed length, and determines the length of the currently parsed mapping data packet, thereby dividing the currently parsed mapping data packet. Optionally, each field in the data header is set to 4 bytes, and the length of the mapping data packet is the 20 bytes of the data header plus the length of the data portion. The specific format of the mapping data packet is not limited, and new functional fields can be added according to functional and business needs. This embodiment is only illustrative and is not intended to limit the scope.

[0093] In some embodiments, the mapping unit encapsulates the sorted mapping output data to obtain a mapping data packet, including:

[0094] The sorted mapping output data is compressed to obtain compressed mapping output data.

[0095] Based on the compressed mapping output data, a data header is added to form a mapping data packet; the data header includes identification information, the original length of the mapping output data, and the compressed length of the compressed mapping output data.

[0096] In this embodiment, to improve data transmission efficiency, the mapping unit first sorts the mapping output data in the same buffer, then compresses the sorted mapping output data, and uses the compressed mapping output data as the data part. A data header is added to this data part to encapsulate it into a mapping data packet. The added data header may include identification information, original length, and compressed length, etc.

[0097] In some embodiments, the reduction unit sorts the sorted mapping output data to obtain data processing results, including: performing a merge sort on the sorted mapping output data to obtain data processing results. That is, after the reduction unit parses out the sorted mapping output data corresponding to each mapping data packet, it performs a merge sort on the sorted mapping output data of each mapping data packet to obtain merge sorted mapping data, which is used as the final sorted data.

[0098] This embodiment provides a data processing method. To ensure the performance and response speed of the service unit while simultaneously implementing sorting functionality, the mapping unit sorts the mapping output data and encapsulates it into mapping data packets, which are then sent to the service unit. The service unit receives the mapping data packets but does not perform any sorting; it simply organizes key-value pairs with the same key into the same mapping data file. The reduction unit obtains the mapping data file from the service unit, divides it into multiple mapping data packets, parses each packet to obtain sorted mapping output data, and then sorts the data based on the sorted mapping output data from the mapping unit. By utilizing their own resources, the mapping and reduction units can achieve data sorting, improving data processing efficiency. Since the service unit does not need to participate in the sorting, it does not affect the service unit.

[0099] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0100] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a data processing apparatus.

[0102] refer to Figure 5 The data processing device includes:

[0103] The mapping unit 501 is used to perform mapping processing on the data to be processed to obtain mapped output data; in response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is encapsulated to obtain a mapping data packet;

[0104] Service unit 502 is used to generate a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are acquired;

[0105] The reduction unit 503 is used to acquire the mapping data file; divide the mapping data file into a predetermined number of mapping data packets; parse each mapping data packet to obtain the sorted mapping output data corresponding to each mapping data packet; and sort the sorted mapping output data to obtain the data processing result.

[0106] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0107] The apparatus of the above embodiments is used to implement the corresponding data processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0108] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method described in any of the above embodiments.

[0109] Figure 6This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0110] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0111] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0112] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0113] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0114] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0115] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0116] The electronic devices described above are used to implement the corresponding data processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0117] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the data processing method as described in any of the above embodiments.

[0118] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0119] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the data processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0120] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word covers the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The article "a" or "an" preceding an element does not exclude the presence of multiple such elements.

[0121] While the spirit and principles of this application have been described with reference to several specific embodiments, it should be understood that this application is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A data processing method, characterized in that, include: The mapping unit is used to perform mapping processing on the data to be processed, and the mapped output data is obtained; In response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is compressed and then encapsulated to obtain a mapping data packet; wherein, the mapping data packet includes a compressed length and an original length; The service unit generates a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are acquired. The process involves: acquiring the mapping data file using a reduction unit; dividing the mapping data file into a predetermined number of mapping data packets; parsing each mapping data packet to obtain sorted mapping output data, including: parsing each mapping data packet to obtain a compressed length and an original length; parsing the compressed mapping output data according to the compressed length; allocating storage space for the original length; decompressing the compressed mapping output data to obtain decompressed mapping output data; storing the decompressed mapping output data in the storage space; and sorting the sorted mapping output data to obtain a data processing result.

2. The method according to claim 1, characterized in that, The mapping data packet includes identification information for identifying the mapping unit; parsing each mapping data packet yields sorted mapping output data corresponding to each mapping data packet, including: Each mapping data packet is parsed to obtain the identification information and the sorted mapping output data; wherein the sorted mapping output data is generated by the mapping unit corresponding to the identification information; Before sorting the sorted mapping output data, the process also includes: Based on the identification information, the sorted mapping output data is deduplicated to obtain deduplicated mapping output data. Sort the sorted mapping output data, including: The deduplication-processed mapping output data is sorted.

3. The method according to claim 2, characterized in that, The identification information includes mapping unit identifier, mapping fault tolerance identifier, and data packet identifier; Based on the identification information, the sorted mapping output data is deduplicated to obtain deduplicated mapping output data, including: Based on the mapping unit identifier and the mapping fault tolerance identifier, the sorted mapping output data is deduplicated. For mapping data packets with the same mapping unit identifier corresponding to multiple mapping fault tolerance identifiers, the mapping data packet with one mapping fault tolerance identifier is retained to obtain the first deduplicated mapping output data. Based on the mapping unit identifier, the mapping fault tolerance identifier, and the data packet identifier, the mapping output data after the first deduplication is deduplicated. For mapping data packets with the same mapping unit identifier and the same mapping fault tolerance identifier corresponding to multiple data packet identifiers, the mapping data packet with one data packet identifier is retained to obtain the mapping output data after the second deduplication.

4. The method according to any one of claims 1-3, characterized in that, Sort the sorted mapping output data, including: The mapped output data in the storage space is sorted.

5. The method according to claim 4, characterized in that, Sort the mapped output data in the storage space, including: In response to the parsed compressed mapping output data reaching a predetermined number of sorting values, or the decompressed mapping output data occupying a predetermined amount of storage space, the mapping output data is sorted.

6. The method according to claim 1, characterized in that, The sorted mapping output data is compressed and then encapsulated to obtain a mapping data packet, which includes: The sorted mapping output data is compressed to obtain compressed mapping output data; Based on the compressed mapping output data, a data header is added to form the mapping data packet; wherein, the data header includes identification information, the original length of the mapping output data, and the compressed length of the compressed mapping output data.

7. The method according to claim 1, characterized in that, The sorted mapping output data is then sorted to obtain the data processing results, including: The sorted mapping output data is then merged and sorted to obtain the data processing result.

8. A data processing apparatus, characterized in that, include: The mapping unit is used to perform mapping processing on the data to be processed, and obtain the mapped output data. In response to the mapping output data reaching a predetermined quantity, the mapping output data is sorted, and the sorted mapping output data is compressed and then encapsulated to obtain a mapping data packet; wherein, the mapping data packet includes a compressed length and an original length; The service unit is used to generate a mapping data file containing a predetermined number of mapping data packets in the order in which the mapping data packets are acquired. A reduction unit is used to acquire the mapping data file; divide the mapping data file into the predetermined number of mapping data packets; parse each mapping data packet to obtain sorted mapping output data corresponding to each mapping data packet, including: parsing each mapping data packet to obtain the compressed length and the original length; parsing the compressed mapping output data according to the compressed length; allocating storage space for the original length; decompressing the compressed mapping output data to obtain decompressed mapping output data; storing the decompressed mapping output data in the storage space; and sorting the sorted mapping output data to obtain a data processing result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1 to 7.

Images