Data processing method, data processing device and electronic equipment

Abstract

The invention provides a data processing method, a data processing device and electronic equipment, and the data processing method comprises the steps: responding to the operation of a preset writing function, and generating a plurality of order-preserving writing requests comprising order-preserving information for to-be-written data; constructing an order-preserving write request queue based on the plurality of order-preserving write requests; and for each order-preserving write-in request in the order-preserving write-in request queue, sending the order-preserving command and the order-preserving write-in request to a storage device based on the order-preserving information of the order-preserving write-in request, or only sending the order-preserving write-in request to the storage device. Therefore, the data writing sequence can be guaranteed and the state of the order-preserving writing thread can be maintained by calling the preset writing function, so that the overhead caused by sending redundant order-preserving writing commands is reduced, the queue resource and the computing resource of the storage equipment are saved, and the order-preserving writing performance and efficiency under multiple threads are further improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computer storage and computing technology, and more specifically, to a data processing method, a data processing device, and an electronic device. Background Art

[0002] Currently, when performing sequence-preserving writes in the F2FS (flash-friendly file system) file system, it is necessary to wait for the current write command to be executed by the storage device before processing the next write command. The main cost of the waiting time is the DMA transmission of the data, which makes the data transmission process time-consuming.

[0003] In addition, when using the BEIOS (Barrier-Enabled IO Stack) operating system, each thread needs to send a BWC (Barrier Write Command) order-preserving write command separately when performing an order-preserving write operation. When there are many order-preserving write threads executed, a large number of BWC commands will be generated, occupying the queue resources of the storage device, and then consuming the computing resources of the storage device, causing the storage device's order-preserving write capability to decline. Summary of the invention

[0004] The embodiments of the present disclosure provide a data processing method, a data processing device, and an electronic device, which can maintain the state of an order-preserving write thread, reduce the overhead caused by sending redundant order-preserving write commands, save queue resources and computing resources of a storage device, and thereby improve the performance and efficiency of order-preserving write under multiple threads.

[0005] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, the data processing method comprising: in response to the execution of a preset write function, generating multiple order-preserving write requests including order-preserving information for data to be written; constructing an order-preserving write request queue based on the multiple order-preserving write requests; for each order-preserving write request in the order-preserving write request queue, based on the order-preserving information of the order-preserving write request, sending an order-preserving command and the order-preserving write request to a storage device, or only sending the order-preserving write request to the storage device.

[0006] Optionally, the data to be written includes user data and metadata, and the step of generating multiple order-preserving write requests including order-preserving information for the data to be written includes: generating multiple first order-preserving write requests including order-preserving information for the user data, wherein the order-preserving information of each first order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the first order-preserving write requests; generating multiple second order-preserving write requests including order-preserving information for the metadata, wherein the order-preserving information of each second order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the second order-preserving write requests.

[0007] Optionally, for each first order-preserving write request, in response to the first order-preserving write request being the last one generated among the multiple first order-preserving write requests, the order-preserving value of the first order-preserving write request is set to a first value, otherwise the order-preserving value of the first order-preserving write request is set to a second value; for each second order-preserving write request, in response to the second order-preserving write request being the last one generated among the multiple second order-preserving write requests, the order-preserving value of the second order-preserving write request is set to a first value, otherwise the order-preserving value of the second order-preserving write request is set to a second value.

[0008] Optionally, the step of constructing an order-preserving write request queue based on the multiple order-preserving write requests includes: in response to the existence of a preset IO schedule, performing order-preserving scheduling processing based on the multiple order-preserving write requests to construct the order-preserving write request queue; in response to the absence of the preset IO schedule, constructing the order-preserving write request queue according to the generation order of the multiple order-preserving write requests.

[0009] Optionally, in response to the existence of a preset IO schedule, order-preserving scheduling is performed based on the multiple order-preserving write requests, and the step of constructing the order-preserving write request queue includes: inserting the multiple order-preserving write requests into a first-in-first-out queue in the order of generation to obtain an initial order-preserving write request queue; constructing a normal write request queue by applying the preset IO schedule to multiple normal write requests, wherein the normal write request is generated based on other data to be written before the preset write function is run; and merging the initial order-preserving write request queue with the normal write request queue according to the write address of each normal write request and the write address of each order-preserving write request to obtain the order-preserving write request queue.

[0010] Optionally, based on the order-preserving information of the order-preserving write request, the step of sending the order-preserving command and the order-preserving write request to the storage device, or only sending the order-preserving write request to the storage device, includes: obtaining the thread number of the order-preserving information of the order-preserving write request, and performing a hash search based on the thread number to obtain the state of the thread corresponding to the thread number from a preset hash table; in response to the state of the thread being a first state, sending the order-preserving command and the order-preserving write request to the storage device in sequence.

[0011] Optionally, based on the order-preserving information of the order-preserving write request, the step of sending the order-preserving command and the order-preserving write request to the storage device, or only sending the order-preserving write request to the storage device, also includes: after sending the order-preserving command and the order-preserving write request to the storage device in sequence, setting the state of the thread in the preset hash table to a second state, and also setting the states of other threads in the preset hash table that meet preset conditions to the second state, wherein the preset condition indicates that the state of the thread is a first state.

[0012] Optionally, based on the order-preserving information of the order-preserving write request, the step of sending the order-preserving command and the order-preserving write request to the storage device, or only sending the order-preserving write request to the storage device, also includes: in response to not obtaining the state of the thread corresponding to the thread number from the preset hash table, setting the state of the thread to a second state, and adding the thread number and the state of the thread to the preset hash table.

[0013] Optionally, based on the order-preserving information of the order-preserving write request, the step of sending the order-preserving command and the order-preserving write request to the storage device, or only sending the order-preserving write request to the storage device, also includes: in response to setting the state of the thread to the second state, or the state of the thread obtained from the preset hash table is not the first state, determining the order-preserving value in the order-preserving information of the order-preserving write request; based on the order-preserving value, sending the order-preserving write request to the storage device, and resetting the state of the thread.

[0014] Optionally, the step of resetting the state of the thread includes: in response to the order preservation value being a second value, setting the state of the thread to a third state; in response to the order preservation value being a first value, setting the state of the thread to a first state.

[0015] According to a second aspect of an embodiment of the present disclosure, a data processing device is provided, comprising: an order-preserving write request generating unit, configured to generate multiple order-preserving write requests including order-preserving information for data to be written in response to the execution of a preset write function; an order-preserving write request queue building unit, configured to build an order-preserving write request queue based on the multiple order-preserving write requests; and a sending unit, configured to send an order-preserving command and the order-preserving write request to a storage device for each order-preserving write request in the order-preserving write request queue based on the order-preserving information of the order-preserving write request, or to send only the order-preserving write request to the storage device.

[0016] Optionally, the data to be written includes user data and metadata, and the order-preserving write request generation unit is further configured to: generate multiple first order-preserving write requests including order-preserving information for the user data, wherein the order-preserving information of each first order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the first order-preserving write requests; generate multiple second order-preserving write requests including order-preserving information for the metadata, wherein the order-preserving information of each second order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the second order-preserving write requests.

[0017] Optionally, the order-preserving write request generation unit is further configured to: for each first order-preserving write request, in response to the first order-preserving write request being the last one generated among the multiple first order-preserving write requests, set the order-preserving value of the first order-preserving write request to a first value, otherwise set the order-preserving value of the first order-preserving write request to a second value; for each second order-preserving write request, in response to the second order-preserving write request being the last one generated among the multiple second order-preserving write requests, set the order-preserving value of the second order-preserving write request to a first value, otherwise set the order-preserving value of the second order-preserving write request to a second value.

[0018] Optionally, the order-preserving write request queue construction unit is further configured to: in response to the existence of a preset IO scheduling, perform order-preserving scheduling processing based on the multiple order-preserving write requests to construct the order-preserving write request queue; in response to the absence of the preset IO scheduling, construct the order-preserving write request queue according to the generation order of the multiple order-preserving write requests.

[0019] Optionally, the order-preserving write request queue construction unit is further configured to: insert the multiple order-preserving write requests into a first-in-first-out queue in the order of generation to obtain an initial order-preserving write request queue; construct a normal write request queue by applying the preset IO scheduling to multiple normal write requests, wherein the normal write request is generated based on other data to be written before the preset write function runs; and merge the initial order-preserving write request queue with the normal write request queue according to the write address of each normal write request and the write address of each order-preserving write request to obtain the order-preserving write request queue.

[0020] Optionally, the sending unit is also configured to: obtain the thread number of the order-preserving information of the order-preserving write request, and perform a hash search based on the thread number to obtain the state of the thread corresponding to the thread number from a preset hash table; in response to the state of the thread being a first state, send the order-preserving command and the order-preserving write request to the storage device in sequence.

[0021] Optionally, the sending unit is also configured to: after sending the order-preserving command and the order-preserving write request to the storage device in sequence, set the state of the thread in the preset hash table to a second state, and also set the states of other threads in the preset hash table that meet preset conditions to the second state, wherein the preset condition indicates that the state of the thread is a first state.

[0022] Optionally, the sending unit is further configured to: in response to not obtaining the state of the thread corresponding to the thread number from the preset hash table, set the state of the thread to a second state, and add the thread number and the state of the thread to the preset hash table.

[0023] Optionally, the sending unit is also configured to: in response to setting the state of the thread to the second state, or the state of the thread obtained from the preset hash table is not the first state, determine the order-preserving value in the order-preserving information of the order-preserving write request; based on the order-preserving value, send the order-preserving write request to the storage device, and reset the state of the thread.

[0024] Optionally, the sending unit is further configured to, in response to the order preservation value being a second value, set the state of the thread to a third state; and in response to the order preservation value being a first value, set the state of the thread to a first state.

[0025] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the data processing method as described above.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the data processing method as described above is implemented.

[0027] According to the data processing method, data processing device and electronic device disclosed in the present invention, the order of data writing can be guaranteed by calling a preset write function, and the state of the order-preserving write thread can be maintained, thereby reducing the overhead caused by the occurrence of redundant order-preserving write commands, saving the queue resources and computing resources of the storage device, thereby improving the performance and efficiency of order-preserving write under multi-threading and maintaining high throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart illustrating a data processing method according to an embodiment of the present disclosure.

[0029] Figure 2 It is a schematic diagram showing the structure of a data processing system according to an embodiment of the present disclosure.

[0030] Figure 3 is a schematic diagram showing an example of generating an order-preserving write request according to an embodiment of the present disclosure.

[0031] Figure 4 is a schematic diagram showing an example of generating an order-preserving write request queue according to an embodiment of the present disclosure.

[0032] Figure 5 It is a schematic diagram showing the structure of a data processing device according to an embodiment of the present disclosure.

[0033] Figure 6 is a schematic structural diagram showing an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present disclosure. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present disclosure, except for operations that must occur in a specific order. In addition, for greater clarity and conciseness, descriptions of features known in the art may be omitted.

[0035] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, many of which will be clear after understanding the disclosure of the present disclosure.

[0036] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0037] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0038] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0039] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0041] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0042] Embodiments of the present disclosure provide a data processing method, a data processing device, and an electronic device.

[0043] Figure 1 is a flowchart illustrating a data processing method according to an embodiment of the present disclosure.

[0044] Reference Figure 1 In step S101, in response to the execution of the preset write function, multiple order-preserving write requests including order-preserving information are generated for the data to be written. Here, when an application (or a thread) calls the preset write function (i.e., executes the preset write function), the order-preserving information is added to the data to be written, and a corresponding order-preserving write request is generated.

[0045] Refer to the following Figure 2 Describes in detail an example of running a preset write function. Figure 2 Schematic diagram showing the structure of a data processing system according to an embodiment of the present disclosure. Figure 2 , the data processing system includes a file system, a block layer, and a device driver. In actual applications, the file system may be an F2FS (Flash-Friendly File System) file system, and the preset write function may be lbarrier(). A Devbarrier mount option may be added to the F2FS file system to call the lbarrier() function to provide an order-preserving write interface. When the Devbarrier mount option is turned on, the lbarrier() function is called to add order-preserving information of the data to be written to generate an order-preserving write request. In this way, even if the application in the F2FS file system calls the fsync() function, the lbarrier() function may be executed in the actual process, so that the application can use the lbarrier() function without modifying the source code.

[0046] According to an embodiment of the present disclosure, the data to be written includes user data and metadata. Therefore, in step S101, multiple first order-preserving write requests including order-preserving information can be generated for the user data, wherein the order-preserving information of each first order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the first order-preserving write request; and multiple second order-preserving write requests including order-preserving information can be generated for the metadata, wherein the order-preserving information of each second order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the second order-preserving write request.

[0047] According to an embodiment of the present disclosure, for each first order-preserving write request, in response to the first order-preserving write request being the last one generated among multiple first order-preserving write requests, the order-preserving value of the first order-preserving write request is set to a first value (for example, but not limited to 1), otherwise the order-preserving value of the first order-preserving write request is set to a second value (for example, but not limited to 0); for each second order-preserving write request, in response to the second order-preserving write request being the last one generated among multiple second order-preserving write requests, the order-preserving value of the second order-preserving write request is set to the first value, otherwise the order-preserving value of the second order-preserving write request is set to the second value.

[0048] As an example, when an application (or thread) calls a preset write function (i.e., the lbarrier() function), the file system may create a first order-preserving write request with order-preserving information for user data, and the number of the first order-preserving write requests is the same as the number of user data. At the same time, the file system may create a second order-preserving write request with order-preserving information for metadata, and the number of the second order-preserving write requests is the same as the number of metadata. Here, the order-preserving information of the first order-preserving write request and the second order-preserving write request may include the thread number of the thread where the data to be written is located and the order-preserving value corresponding to the order in which the order-preserving write requests are generated.

[0049] As described above, the order-preserving value in the corresponding order-preserving information can be determined according to the generation order of each first order-preserving write request. Specifically, multiple first order-preserving write requests can be sorted according to the generation order, the order-preserving value of the last first order-preserving write request can be set to the first value (for example, but not limited to 1), and the order-preserving values ​​of the first order-preserving write requests other than the last first order-preserving write request can be set to the second value (for example, but not limited to 0). Similarly, the order-preserving value in the order-preserving information can be determined according to the generation order of each second order-preserving write request. That is, multiple second order-preserving write requests can be sorted according to the generation order, the order-preserving value of the last second order-preserving write request can be set to the first value, and the order-preserving values ​​of the remaining second order-preserving write requests can be set to the second value.

[0050] Figure 3 is a schematic diagram showing an example of generating an order-preserving write request according to an embodiment of the present disclosure. Figure 3 , when thread A (where A is the thread number) calls the lbarrier() function, the file system may generate first order-preserving write requests ① and ② based on each user data, the order-preserving information of the first order-preserving write request ① includes thread number A and order-preserving value 0, and the order-preserving information of the first order-preserving write request ② includes thread number A and order-preserving value 1. In addition, the file system may generate a second order-preserving write request ③ based on the metadata, and the order-preserving information of the second order-preserving write request ③ includes thread number A and order-preserving value 1.

[0051] In step S102, an order-preserving write request queue is constructed based on multiple order-preserving write requests.

[0052] Refer again Figure 2 The block layer can receive a normal write request directly generated based on the data to be written, or it can receive an order-preserving write request generated in response to the execution of a preset write function. When the block layer receives a normal write request, it can send the normal write request directly to the device driver. When the file system sends multiple generated order-preserving write requests to the block layer, the block layer can build an order-preserving write request queue based on the received order-preserving write requests.

[0053] According to an embodiment of the present disclosure, step S102 may specifically include the following steps: in response to the existence of a preset IO schedule, performing order-preserving scheduling processing based on multiple order-preserving write requests, and constructing an order-preserving write request queue; in response to the absence of a preset IO schedule, constructing an order-preserving write request queue according to the generation order of multiple order-preserving write requests. Here, the preset IO schedule can be implemented by the block layer using an IO scheduler. The IO scheduler can select a noop scheduler, a cfq scheduler, a Deadline scheduler, and an Anticipatory scheduler, etc., which is not limited by the present disclosure. If the block layer does not use the IO scheduler, it means that there is no preset IO schedule. If the block layer uses the IO scheduler, it means that there is a preset IO schedule. When the block layer does not use the IO scheduler (that is, there is no preset IO schedule), the order-preserving write request queue can be simply constructed according to the generation order of the order-preserving write requests.

[0054] On the other hand, when the block layer uses an IO scheduler (i.e., there is a preset IO schedule), order-preserving scheduling can be performed based on multiple order-preserving write requests to build an order-preserving write request queue through the following steps: insert multiple order-preserving write requests into a first-in-first-out queue in the order of generation to obtain an initial order-preserving write request queue; construct a normal write request queue by applying the preset IO schedule to multiple normal write requests, wherein the normal write request is generated based on other data to be written before the preset write function runs; and merge the initial order-preserving write request queue with the normal write request queue according to the write address of each normal write request and the write address of each order-preserving write request to obtain an order-preserving write request queue.

[0055] Figure 4 is a schematic diagram showing an example of generating an order-preserving write request queue according to an embodiment of the present disclosure. Figure 4 ,When there is a preset IO schedule, multiple order-preserving write requests can be inserted into the first-in-first-out queue in the order of generation to obtain the initial order-preserving write request queue, such as Figure 4 Then, the normal write request can be processed according to the preset IO scheduling, and a normal write request queue can be constructed, such as Figure 4Then, it is determined whether the write address of each order-preserving write request and the write address of each normal write request are continuous, and at the same time, it is determined whether the write address of each normal write request is continuous. Next, the normal write requests with continuous addresses can be merged first, and then the normal write requests with continuous addresses and the order-preserving write requests can be merged to obtain the final order-preserving write request queue, as shown in FIG. Figure 4 As shown in 403.

[0056] Return to reference Figure 1 In step S103, for each order-preserving write request in the order-preserving write request queue, based on the order-preserving information of the order-preserving write request, the order-preserving command and the order-preserving write request are sent to the storage device, or only the order-preserving write request is sent to the storage device.

[0057] Specifically, for any order-preserving write request, the thread number of the order-preserving information of the order-preserving write request can be obtained first, and a hash search can be performed based on the thread number to obtain the state of the thread corresponding to the thread number from the preset hash table; then, in response to the thread state being the first state, the order-preserving command and the order-preserving write request can be sent to the storage device in sequence.

[0058] Here, if Figure 2 As shown, the order-preserving write request queue can be output from the block layer to the device driver, and the device driver obtains the thread number of the order-preserving write request based on the order-preserving information of the order-preserving write request received in the order-preserving write request queue. Then, the thread state corresponding to the thread number can be searched in the hash table according to the thread number. When the searched thread state is the first state, the order-preserving command can be sent first, and then the order-preserving write request can be sent. Here, the thread state being the first state can indicate that the thread state S is P2, where P2 represents the need to send an order-preserving command. The order-preserving command can be expressed as a BWC (BarrierWrite Command) command. By sending the BWC command from the device driver to the storage device, the storage device can implement order-preserving write based on the BWC command. Here, the storage device can be a storage device that supports the BWC command.

[0059] According to an embodiment of the present disclosure, after the order-preserving command and the order-preserving write request are sent to the storage device in sequence, the state of the thread in the preset hash table can be set to the second state, and the states of other threads in the preset hash table that meet the preset conditions can also be set to the second state, wherein the preset condition indicates that the state of the thread is the first state.

[0060] As an example, the thread state being the second state can indicate that the thread state S is P0, where P0 represents that the thread is in the initial state. After the order-preserving command and the order-preserving write request are sent to the storage device in sequence, the thread state of the thread where the order-preserving command is located is set to the second state, that is, the initial state, and the thread in the hash table whose thread state is the first state (that is, P2) is deleted (that is, the thread state of the thread state in the first state is also set to the second state). In this way, when there are multiple threads executing order-preserving writes, only the current thread is supported to send BWC commands, thereby saving queue resources and computing resources of the storage device and improving the performance of order-preserving writes.

[0061] On the other hand, in response to not obtaining the state of the thread corresponding to the thread number from the preset hash table, the state of the thread can be set to the second state, and the thread number and the state of the thread can be added to the preset hash table.

[0062] Here, when the thread state corresponding to the thread number is not obtained in the preset hash table, the thread state of the thread is set to the second state (i.e. P0, the initial state), and the thread state and thread number of the thread are added to the preset hash table so that other order-preserving write requests for the thread can determine the corresponding thread state.

[0063] According to an embodiment of the present disclosure, in response to setting the state of a thread to a second state, or the state of the thread obtained from a preset hash table is not the first state, an order-preserving value in the order-preserving information of an order-preserving write request can be determined; based on the order-preserving value, the order-preserving write request is sent to a storage device, and the state of the thread is reset.

[0064] According to an embodiment of the present disclosure, the state of the thread can be reset through the following steps: in response to the sequence preservation value being the second value, the state of the thread is set to the third state (i.e., P1); in response to the sequence preservation value being the first value, the state of the thread is set to the first state (i.e., P2).

[0065] Here, when the state of the thread is set to the second state, or when the thread state found in the preset hash table according to the thread number is not the first state (i.e., P2) but other states (e.g., the second state P0 or the third state P1), the state of the thread can be reset according to the order-preserving value in the order-preserving information, and the order-preserving write request is sent to the storage device. Specifically, when the thread state is P0 or P1, the order-preserving value of the order-preserving write request is obtained. When the order-preserving value of the first order-preserving write request is the second value (i.e., 0), it is determined that the first order-preserving write request is not the last first order-preserving write request, the first order-preserving write request is sent to the storage device, and the state of the thread where the first order-preserving write request is located is reset to the third state (i.e., P1). When the order-preserving value of the first order-preserving write request is the first value (i.e., 1), it is determined that the order-preserving write request is the last first order-preserving write request, the first order-preserving write request is sent to the storage device, and the state of the thread where the first order-preserving write request is located is reset to the first state (i.e., P2). For the second order-preserving write request, the second order-preserving write request may be sent and the state of the thread where the second order-preserving write request is located may be reset in a manner similar to the manner of processing the first order-preserving write request.

[0066] According to the data processing method of the embodiment of the present disclosure, the order of data writing can be guaranteed by calling a preset write function, and the state of the order-preserving write thread can be maintained, reducing the overhead caused by sending redundant order-preserving write commands, saving queue resources and computing resources of the storage device, thereby improving the performance and efficiency of order-preserving write under multi-threading and maintaining high throughput.

[0067] Figure 5 It is a schematic diagram showing the structure of a data processing device according to an embodiment of the present disclosure.

[0068] Reference Figure 5 According to an embodiment of the present disclosure, a data processing device 500 includes an order-preserving write request generating unit 501 , an order-preserving write request queue building unit 502 , and a sending unit 503 .

[0069] The order-preserving write request generation unit 501 generates multiple order-preserving write requests including order-preserving information for the data to be written in response to the operation of the preset write function. The order-preserving write request queue construction unit 502 constructs an order-preserving write request queue based on the multiple order-preserving write requests. The sending unit 503 sends the order-preserving command and the order-preserving write request to the storage device based on the order-preserving information of the order-preserving write request for each order-preserving write request in the order-preserving write request queue, or only sends the order-preserving write request to the storage device.

[0070] According to an embodiment of the present disclosure, the data to be written includes user data and metadata. The order-preserving write request generation unit 501 can generate multiple first order-preserving write requests including order-preserving information for the user data, wherein the order-preserving information of each first order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the first order-preserving write request; and the order-preserving write request generation unit 501 can generate multiple second order-preserving write requests including order-preserving information for the metadata, wherein the order-preserving information of each second order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the second order-preserving write request.

[0071] According to an embodiment of the present disclosure, the order-preserving write request generation unit 501 may, for each first order-preserving write request, in response to the first order-preserving write request being the last one generated among multiple first order-preserving write requests, set the order-preserving value of the first order-preserving write request to the first value, otherwise set the order-preserving value of the first order-preserving write request to the second value; and the order-preserving write request generation unit 501 may, for each second order-preserving write request, in response to the second order-preserving write request being the last one generated among multiple second order-preserving write requests, set the order-preserving value of the second order-preserving write request to the first value, otherwise set the order-preserving value of the second order-preserving write request to the second value.

[0072] According to an embodiment of the present disclosure, the order-preserving write request queue construction unit 502 can, in response to the existence of a preset IO schedule, perform order-preserving scheduling processing based on multiple order-preserving write requests to construct an order-preserving write request queue; in response to the absence of a preset IO schedule, construct an order-preserving write request queue according to the generation order of multiple order-preserving write requests.

[0073] According to an embodiment of the present disclosure, the order-preserving write request queue construction unit 502 can insert multiple order-preserving write requests into a first-in-first-out queue in the order of generation to obtain an initial order-preserving write request queue; construct a normal write request queue by applying a preset IO schedule to multiple normal write requests, wherein the normal write request is generated based on other data to be written before the preset write function runs; and merge the initial order-preserving write request queue with the normal write request queue according to the write address of each normal write request and the write address of each order-preserving write request to obtain an order-preserving write request queue.

[0074] According to an embodiment of the present disclosure, the sending unit 503 can obtain the thread number of the order-preserving information of the order-preserving write request, and perform a hash search based on the thread number to obtain the state of the thread corresponding to the thread number from a preset hash table; in response to the state of the thread being the first state, the order-preserving command and the order-preserving write request are sent to the storage device in sequence.

[0075] According to an embodiment of the present disclosure, after sending the order-preserving command and the order-preserving write request to the storage device in sequence, the sending unit 503 may set the state of the thread in the preset hash table to the second state, and also set the state of other threads in the preset hash table that meet the preset conditions to the second state, wherein the preset condition indicates that the state of the thread is the first state.

[0076] According to an embodiment of the present disclosure, in response to not obtaining the state of the thread corresponding to the thread number from the preset hash table, the sending unit 503 can set the state of the thread to the second state and add the thread number and the state of the thread to the preset hash table.

[0077] According to an embodiment of the present disclosure, the sending unit 503 may determine the order-preserving value in the order-preserving information of the order-preserving write request in response to setting the state of the thread to the second state, or the state of the thread obtained from the preset hash table is not the first state; based on the order-preserving value, send the order-preserving write request to the storage device, and reset the state of the thread.

[0078] According to an embodiment of the present disclosure, the sending unit 503 may set the thread state to the third state in response to the sequence preservation value being the second value; and set the thread state to the first state in response to the sequence preservation value being the first value.

[0079] Figure 6 is a block diagram showing an electronic device according to an embodiment of the present disclosure. Figure 6 The electronic device 600 includes: a processor 601; a memory 602, wherein the memory 602 stores computer executable instructions, wherein when the computer executable instructions are executed by the processor 601, the processor 601 is prompted to execute the data processing method according to the exemplary embodiment of the present disclosure.

[0080] As an example, the electronic device 600 can be a PC computer, a tablet device, a personal digital assistant, a smart phone or other devices capable of executing the above instructions. Here, the electronic device 600 is not necessarily a single electronic device, but can also be any device or circuit that can execute the above instructions (or instruction sets) individually or jointly. The electronic device 600 can also be a part of an integrated control system or a system manager, or can be configured as a portable electronic device interconnected with an interface locally or remotely (for example, via wireless transmission). In addition, the electronic device 600 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device 600 can be connected to each other via a bus and / or a network.

[0081] Processor 601 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, processor 601 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0082] The processor 601 may execute instructions or codes stored in the memory 602, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0083] The memory 602 may be integrated with the processor 601, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 602 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 602 and the processor 601 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 601 can read files stored in the memory 602.

[0084] According to an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor, the processor is prompted to execute a data processing method according to an embodiment of the present disclosure.

[0085] The data processing method according to the embodiment of the present disclosure can be written as a computer program and stored on a computer-readable storage medium. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. In one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0086] According to the data processing method, data processing device and electronic device disclosed in the present invention, an order-preserving write request with order-preserving information is generated by using a preset write function, and the order-preserving command and the order-preserving write request are sent to the storage device according to the order-preserving information, or only the order-preserving write request is sent to the storage device. This can ensure the order of data writing and maintain the status of the order-preserving write thread, reduce the overhead caused by sending redundant order-preserving write commands, save the queue resources and computing resources of the storage device, thereby improving the performance and efficiency of order-preserving write under multi-threading and maintaining high throughput.

[0087] The specific implementation methods of the present disclosure have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments may be modified and varied without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents. These modifications and variations should also be within the scope of protection of the claims of the present disclosure.

Claims

1. A data processing method, characterized in that: The data processing method comprises: In response to the execution of the preset write function, generating a plurality of order-preserving write requests including order-preserving information for the data to be written; Building an order-preserving write request queue based on the multiple order-preserving write requests; For each order-preserving write request in the order-preserving write request queue, based on the order-preserving information of the order-preserving write request, an order-preserving command and the order-preserving write request are sent to a storage device, or only the order-preserving write request is sent to a storage device.

2. The data processing method according to claim 1, characterized in that: The data to be written includes user data and metadata, The step of generating a plurality of order-preserving write requests including order-preserving information for the data to be written comprises: Generate a plurality of first order-preserving write requests including order-preserving information for the user data, wherein the order-preserving information of each first order-preserving write request includes an order-preserving value determined based on a thread number of the data to be written and a generation order of the first order-preserving write requests; A plurality of second order-preserving write requests including order-preserving information are generated for the metadata, wherein the order-preserving information of each second order-preserving write request includes an order-preserving value determined based on the thread number of the data to be written and the generation order of the second order-preserving write requests.

3. The data processing method according to claim 2, characterized in that: For each first order-preserving write request, in response to the first order-preserving write request being the last one generated among the multiple first order-preserving write requests, setting the order-preserving value of the first order-preserving write request to a first value, otherwise setting the order-preserving value of the first order-preserving write request to a second value; For each second order-preserving write request, in response to the second order-preserving write request being the last one generated among the multiple second order-preserving write requests, the order-preserving value of the second order-preserving write request is set to the first value, otherwise the order-preserving value of the second order-preserving write request is set to the second value.

4. The data processing method according to claim 1, characterized in that: The step of constructing an order-preserving write request queue based on the multiple order-preserving write requests includes: In response to the existence of a preset IO schedule, performing order-preserving scheduling processing based on the multiple order-preserving write requests, and constructing the order-preserving write request queue; In response to the absence of the preset IO schedule, the order-preserving write request queue is constructed according to the generation order of the multiple order-preserving write requests.

5. The data processing method according to claim 4, characterized in that: In response to the existence of a preset IO schedule, performing order-preserving scheduling processing based on the multiple order-preserving write requests, and constructing the order-preserving write request queue comprises: Inserting the plurality of order-preserving write requests into a first-in-first-out queue in the order in which they are generated, to obtain an initial order-preserving write request queue; Building a normal write request queue by applying the preset IO scheduling to a plurality of normal write requests, wherein the normal write request is generated based on other to-be-written data before the preset write function is run; According to the write address of each normal write request and the write address of each order-preserving write request, the initial order-preserving write request queue is merged with the normal write request queue to obtain the order-preserving write request queue.

6. The data processing method according to claim 2, characterized in that: The step of sending an order-preserving command and the order-preserving write request to a storage device based on the order-preserving information of the order-preserving write request, or sending only the order-preserving write request to the storage device, comprises: Obtaining a thread number of the order-preserving information of the order-preserving write request, and performing a hash search based on the thread number to obtain a state of a thread corresponding to the thread number from a preset hash table; In response to the state of the thread being the first state, an order-preserving command and the order-preserving write request are sent to a storage device in sequence.

7. The data processing method according to claim 6, characterized in that: The step of sending the order-preserving command and the order-preserving write request to the storage device based on the order-preserving information of the order-preserving write request, or sending only the order-preserving write request to the storage device, further includes: After the order-preserving command and the order-preserving write request are sent to the storage device in sequence, the state of the thread in the preset hash table is set to the second state, and the states of other threads in the preset hash table that meet the preset conditions are also set to the second state, wherein the preset conditions indicate that the state of the thread is the first state.

8. The data processing method according to claim 6, characterized in that: The step of sending the order-preserving command and the order-preserving write request to the storage device based on the order-preserving information of the order-preserving write request, or sending only the order-preserving write request to the storage device, further includes: In response to not obtaining the state of the thread corresponding to the thread number from the preset hash table, setting the state of the thread to a second state, and adding the thread number and the state of the thread to the preset hash table.

9. The data processing method according to claim 8, characterized in that: The step of sending the order-preserving command and the order-preserving write request to the storage device based on the order-preserving information of the order-preserving write request, or sending only the order-preserving write request to the storage device, further includes: In response to setting the state of the thread to the second state, or the state of the thread obtained from the preset hash table is not the first state, determining an order-preserving value in the order-preserving information of the order-preserving write request; Based on the order-preserving value, the order-preserving write request is sent to a storage device, and the state of the thread is reset.

10. The data processing method according to claim 9, characterized in that: The step of resetting the state of the thread comprises: In response to the sequence preservation value being a second value, setting the state of the thread to a third state; In response to the sequence preservation value being a first value, setting the state of the thread to a first state.

11. A data processing device, characterized in that: The data processing device comprises: An order-preserving write request generating unit, configured to generate a plurality of order-preserving write requests including order-preserving information for data to be written in response to the execution of a preset write function; an order-preserving write request queue construction unit, configured to construct an order-preserving write request queue based on the multiple order-preserving write requests; The sending unit is configured to send an order-preserving command and the order-preserving write request to the storage device for each order-preserving write request in the order-preserving write request queue based on the order-preserving information of the order-preserving write request, or to send only the order-preserving write request to the storage device.

12. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

Images