A data processing method and apparatus

Abstract

This application provides a data processing method and apparatus. The method includes: receiving target data in a mixed data format; dynamically acquiring a current cost threshold and acquiring the data processing cost corresponding to the target data; determining whether the data processing cost exceeds the current cost threshold; if not, parsing the target data into point-to-point data; wherein the point-to-point data includes a file descriptor, a starting position, and an offset; and processing the point-to-point data using a point-to-point data processing procedure. It is evident that implementing this method allows file content to be directly sent to acceleration hardware via P2P technology, thereby reducing the time data spends in computer memory and improving data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method and apparatus. Background Technology

[0002] Databases require reading and writing a large number of files when processing data. To achieve this, those skilled in the art have employed hardware acceleration. However, during hardware acceleration, processing data within computer files requires reading the data from the file into computer memory and then sending the data from memory to the acceleration hardware. However, current data transmission efficiency is still not ideal, and those skilled in the art continue to conduct related research and experiments to improve this efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method and apparatus that can directly send file content to acceleration hardware through P2P technology, thereby reducing the process of data passing through computer memory and improving data transmission efficiency.

[0004] The first aspect of this application provides a data processing method, including:

[0005] Receive target data in a mixed data format;

[0006] Dynamically obtain the current cost threshold and obtain the data processing cost corresponding to the target data;

[0007] Determine whether the data processing cost exceeds the current cost threshold;

[0008] If not, the target data is parsed into point-to-point data; wherein the point-to-point data includes a file descriptor, a starting position, and an offset;

[0009] The point-to-point data is processed using a point-to-point data processing procedure.

[0010] In the above implementation process, this method can first receive target data in a mixed data format; then, dynamically obtain the current cost threshold and the data processing cost corresponding to the target data; next, determine whether the data processing cost exceeds the current cost threshold; and if the data processing cost does not exceed the current cost threshold, parse the target data into peer-to-peer data; wherein the peer-to-peer data includes a file descriptor, a starting position, and an offset; finally, process the peer-to-peer data using a peer-to-peer data processing procedure. It is evident that this method can directly send file content to acceleration hardware via P2P technology, thereby reducing the process of data passing through computer memory and thus improving data transmission efficiency.

[0011] Furthermore, the dynamic acquisition of the current cost threshold includes:

[0012] Dynamically monitor the current device status and current point-to-point transmission performance according to preset rules;

[0013] The point-to-point transmission cost is determined based on the current device status and the current point-to-point transmission performance.

[0014] The current cost threshold is determined based on the point-to-point transmission cost.

[0015] Furthermore, the method also includes:

[0016] When it is determined that the data processing cost exceeds the current cost threshold, the target data is parsed into a data pointer;

[0017] The data pointer is processed using a data processing procedure that uses in-memory data.

[0018] Furthermore, the hybrid data format is specifically a quadruple data format, wherein the quadruple data format includes a file descriptor part, a start position part, an offset part, and a data pointer part.

[0019] A second aspect of this application provides a data processing apparatus, the data processing apparatus comprising:

[0020] The receiving unit is used to receive target data in a mixed data format;

[0021] The dynamic acquisition unit is used to dynamically acquire the current cost threshold;

[0022] The acquisition unit is used to acquire the data processing cost corresponding to the target data;

[0023] The judgment unit is used to determine whether the data processing cost exceeds the current cost threshold;

[0024] The parsing unit is used to parse the target data into point-to-point data when it is determined that the current cost threshold is not exceeded; wherein the point-to-point data includes a file descriptor, a starting position, and an offset;

[0025] A data processing unit is used to process the point-to-point data using a point-to-point data processing procedure.

[0026] In the above implementation process, the data processing device can receive target data in mixed data form through a receiving unit; dynamically acquire a current cost threshold through a dynamic acquisition unit; acquire the data processing cost corresponding to the target data through an acquisition unit; determine whether the data processing cost exceeds the current cost threshold through a judgment unit; and, if the determination shows that the cost does not exceed the current cost threshold, parse the target data into peer-to-peer data through a parsing unit. The peer-to-peer data includes a file descriptor, a starting position, and an offset. Finally, the data processing unit processes the peer-to-peer data using a peer-to-peer data processing procedure. Therefore, this method can directly send file content to acceleration hardware using P2P technology, thereby reducing the data's passage through computer memory and improving data transmission efficiency.

[0027] Furthermore, the dynamic acquisition unit includes:

[0028] The monitoring subunit is used to dynamically monitor the current device status and current point-to-point transmission performance according to preset rules.

[0029] A determining subunit is configured to determine the point-to-point transmission cost based on the current device state and the current point-to-point transmission performance; and to determine the current cost threshold based on the point-to-point transmission cost.

[0030] Furthermore, the parsing unit is also used to parse the target data into a data pointer when it is determined that the current cost threshold is exceeded;

[0031] The data processing unit is also used to process the data pointer using a data processing procedure in the form of memory data.

[0032] Furthermore, the hybrid data format is specifically a quadruple data format, wherein the quadruple data format includes a file descriptor part, a start position part, an offset part, and a data pointer part.

[0033] A third aspect of this application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the data processing method described in any one of the first aspects of this application.

[0034] A fourth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the data processing method described in any one of the first aspects of this application. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0038] Figure 3 A comparative diagram of in-memory data and P2P data provided in an embodiment of this application;

[0039] Figure 4 A process diagram for parsing mixed-form data provided in an embodiment of this application;

[0040] Figure 5 This is a schematic diagram illustrating the application of a dynamically adaptable P2P technology provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0042] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0043] Example 1

[0044] Please refer to Figure 1 , Figure 1 This embodiment provides a flowchart illustrating a data processing method. The data processing method includes:

[0045] S101, Receive target data in mixed data format.

[0046] In this embodiment, the hybrid data format is specifically a quadruple data format, which includes a file descriptor part, a start position part, an offset part, and a data pointer part.

[0047] In this embodiment, normally, processing file data involves reading the file into a segment of computer memory. The computer program then organizes this memory into an ordered structure and sends it to the acceleration hardware. The computer primarily processes data in memory through memory addresses.

[0048] However, when using P2P technology, since the data does not pass through computer memory and there are no memory addresses available for computer processing, it needs to be represented in another form. Considering the details of P2P technology in hardware acceleration, this method can represent file data using a (file descriptor, start position, offset) triple, specifically defined as:

[0049] (1) File descriptor: A unique file identifier within a process, provided by the operating system. Generally, it has no unit.

[0050] (2) Starting position: The amount of data between the beginning of the data to be processed and the beginning of the file. The unit is bits, which are generally converted to bytes (B), kilobytes (KB), megabytes (MB), etc.

[0051] (3) Offset: The amount of data between the end and the beginning of the data content to be processed. The unit is bit, which is generally converted to bytes (B), kilobytes (KB), megabytes (MB), etc.

[0052] Please refer to Figure 3 , Figure 3 This illustrates the data format and processing steps when using P2P technology:

[0053] The file F-example has a corresponding file descriptor generated by the operating system's file management module. Without P2P technology, the two required data segments "ABCD…" and "EFGH…" need to be read into memory, and then transmitted to the acceleration hardware by initiating the corresponding memory address operation. With P2P technology, large blocks of data are no longer copied; instead, they are transmitted directly to the acceleration hardware through three attributes. Compared to computer memory processing, this reduces the process of reading data into memory.

[0054] S102. Dynamically monitor the current device status and current point-to-point transmission performance according to preset rules.

[0055] S103. Determine the point-to-point transmission cost based on the current device status and the current point-to-point transmission performance.

[0056] S104. Determine the current cost threshold based on the point-to-point transmission cost.

[0057] S105. Obtain the data processing cost corresponding to the target data.

[0058] S106. Determine whether the data processing cost exceeds the current cost threshold. If yes, proceed to steps S109 to S110; otherwise, proceed to steps S107 to S108.

[0059] S107. Parse the target data into point-to-point data; whereby the point-to-point data includes file descriptor, starting position, and offset.

[0060] S108. Use the point-to-point data processing procedure to process the point-to-point data.

[0061] S109. Parse the target data into a data pointer.

[0062] S110. Data pointers are processed using a data processing procedure that uses memory data.

[0063] In this embodiment, regarding the combination of P2P data format and in-memory data format, the method can select different processing procedures according to different actual situations and program needs. Although P2P technology makes data transmission faster, the P2P data format still has some drawbacks, such as the inability to preprocess data.

[0064] To solve this problem, we can combine P2P data format and memory data format, using a quadruple (file descriptor, starting position, offset, data pointer) to represent the data. The definitions of file descriptor, starting position, and offset are the same as above, while the definition of the data pointer is as follows:

[0065] This transmission is invalid when P2P technology is used; it is the memory address of the actual data content in the file when P2P technology is not used.

[0066] Then, the method parses this mixed-form data through a process to obtain the actual data.

[0067] Please refer to Figure 4 , Figure 4 The process of parsing mixed-form data is illustrated.

[0068] As can be seen, using this method to represent and process data enables transparent processing of both P2P and in-memory data.

[0069] In this embodiment, regarding the process of selecting and matching data formats and dynamically adapting to P2P, after the method can transparently operate on mixed-format data, it is still necessary to select the corresponding processing procedure after parsing the data format. By defining the cost of P2P technology, the database itself can apply different processing methods to different data formats. The specific definition method is as follows:

[0070] (1) The cost is set to infinity when the storage device does not support P2P function or when there is a known problem with the acceleration hardware;

[0071] (2) When CPU must be used to preprocess data, the cost is set to infinity;

[0072] (3) In addition to the above, the query cost of P2P can be set according to the actual situation and the statistical information of the database. For example, the cost can be set according to the time taken to process data using memory and the time taken to process data using P2P technology in the past.

[0073] In this embodiment, the current device status and current P2P performance are dynamically monitored according to the above rules. The P2P cost is adjusted as needed and a threshold is set: when the threshold is exceeded, the mixed data form is parsed into a data pointer and the corresponding memory data form is used for data processing; when the threshold is not exceeded, the mixed data form is parsed into a P2P data form and the corresponding P2P data form is used for processing.

[0074] In this embodiment, since parsing the quadruple data and the selection process are performed separately, if the cost changes between these two steps, it may lead to abnormal situations such as using in-memory processing to handle P2P data. To ensure the correct correspondence between data types and processing procedures, a copy of the cost is made before executing a query. During the query process, the cost copy remains unchanged, and the value of the cost copy is used instead of the cost itself.

[0075] Please refer to Figure 5 , Figure 5 A schematic diagram illustrating the application of this method to dynamically adapt to P2P technology is shown.

[0076] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.

[0077] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.

[0078] As can be seen, implementing the data processing method described in this embodiment can directly send file content to acceleration hardware through P2P technology, thereby reducing the process of data passing through computer memory and thus improving data transmission efficiency.

[0079] Example 2

[0080] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a data processing device provided in this embodiment. Figure 2 As shown, the data processing apparatus includes:

[0081] The receiving unit 210 is used to receive target data in a mixed data format;

[0082] The dynamic acquisition unit 220 is used to dynamically acquire the current cost threshold.

[0083] The acquisition unit 230 is used to acquire the data processing cost corresponding to the target data;

[0084] The judgment unit 240 is used to determine whether the data processing cost exceeds the current cost threshold.

[0085] The parsing unit 250 is used to parse the target data into point-to-point data when it is determined that the current cost threshold is not exceeded; wherein, the point-to-point data includes a file descriptor, a starting position, and an offset;

[0086] The data processing unit 260 is used to process point-to-point data using a point-to-point data processing procedure.

[0087] As an optional implementation, the dynamic acquisition unit 220 includes:

[0088] The monitoring subunit 221 is used to dynamically monitor the current device status and the current point-to-point transmission performance according to preset rules.

[0089] The determination subunit 222 is used to determine the point-to-point transmission cost based on the current device status and the current point-to-point transmission performance; and to determine the current cost threshold based on the point-to-point transmission cost.

[0090] As an optional implementation, the parsing unit 250 is also used to parse the target data into a data pointer when it is determined that the current cost threshold is exceeded;

[0091] The data processing unit 260 is also used to process data pointers using a data processing procedure in the form of memory data.

[0092] In this embodiment, the hybrid data format is specifically a quadruple data format, which includes a file descriptor part, a start position part, an offset part, and a data pointer part.

[0093] In this embodiment, the explanation of the data processing device can be referred to the description in Embodiment 1, and will not be repeated here.

[0094] As can be seen, the data processing apparatus described in this embodiment can directly send file content to acceleration hardware through P2P technology, thereby reducing the process of data passing through computer memory and thus improving data transmission efficiency.

[0095] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the data processing method in embodiment 1 of this application.

[0096] This application provides a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform the data processing method in embodiment 1 of this application.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0098] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0099] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data processing method, characterized in that, include: Receive target data in a mixed data format; Dynamically obtain the current cost threshold and obtain the data processing cost corresponding to the target data; Determine whether the data processing cost exceeds the current cost threshold; If not, the target data is parsed into point-to-point data; wherein the point-to-point data includes a file descriptor, a starting position, and an offset; The point-to-point data is processed using a point-to-point data processing procedure; The dynamic acquisition of the current cost threshold includes: Dynamically monitor the current device status and current point-to-point transmission performance according to preset rules; The point-to-point transmission cost is determined based on the current device status and the current point-to-point transmission performance. The current cost threshold is determined based on the point-to-point transmission cost; The method further includes: When it is determined that the data processing cost exceeds the current cost threshold, the target data is parsed into a data pointer; The data pointer is processed using a data processing procedure in the form of in-memory data; The process of processing the point-to-point data using a point-to-point data processing procedure includes: The file data is represented by the point-to-point data and then directly transmitted to the acceleration hardware.

2. The data processing method according to claim 1, characterized in that, The hybrid data format is specifically a quadruple data format, which includes a file descriptor part, a start position part, an offset part, and a data pointer part.

3. A data processing apparatus, characterized in that, The data processing device includes: The receiving unit is used to receive target data in a mixed data format; The dynamic acquisition unit is used to dynamically acquire the current cost threshold; The acquisition unit is used to acquire the data processing cost corresponding to the target data; The judgment unit is used to determine whether the data processing cost exceeds the current cost threshold; The parsing unit is used to parse the target data into point-to-point data when it is determined that the current cost threshold is not exceeded; wherein the point-to-point data includes a file descriptor, a starting position, and an offset; A data processing unit is used to process the point-to-point data using a point-to-point data processing procedure; The dynamic acquisition unit includes: The monitoring subunit is used to dynamically monitor the current device status and current point-to-point transmission performance according to preset rules. A determining subunit is configured to determine the point-to-point transmission cost based on the current device state and the current point-to-point transmission performance; and to determine a current cost threshold based on the point-to-point transmission cost. The parsing unit is further configured to parse the target data into a data pointer when it is determined that the current cost threshold is exceeded. The data processing unit is also used to process the data pointer using a data processing procedure in the form of memory data; The data processing unit includes: The file data is represented by the point-to-point data and then directly transmitted to the acceleration hardware.

4. The data processing apparatus according to claim 3, characterized in that, The hybrid data format is specifically a quadruple data format, which includes a file descriptor part, a start position part, an offset part, and a data pointer part.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the data processing method according to any one of claims 1 to 2.

6. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the data processing method according to any one of claims 1 to 2.

Images