Data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method, device and equipment and a storage medium, which are applied to a preset network architecture including a metadata service device, a storage service device and a plurality of client devices. The method comprises the following steps: the client device determines whether target data exists in the read-write cache thereof, and if the target data exists, determines whether the target data is invalid; if the target data is not invalid, the target data is read from the read-write cache to the processor; if the target data does not exist or is invalid, a target data reading request is sent to the metadata service device. The metadata service device receives the target data reading request, determines whether there is a write access client device, and if there is not, sends a target data first reading permission response to the target client device. The client device reads the target data from the storage service device and stores the target data in the read-write cache thereof in response to the target data first reading permission response. The application can realize high-speed data reading and writing of high-performance computing in a multi-client device scenario.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] In artificial intelligence technology, model training is usually carried out on computing nodes, which in turn read and store computational data through the storage nodes they are connected to. With the development of high-performance computing, higher requirements are now being placed on backend distributed storage.

[0003] Currently, the high-speed data read / write requirements in high-performance computing scenarios are primarily met through the client-side persistent caching (PCC) scheme of the Lustre parallel file system. In the PCC scheme, compute nodes act as clients of the parallel file system, using high-performance disks. Storage nodes act as servers of the parallel file system, using low-speed disks. Clients asynchronously cache data from the server into a read / write buffer layer, allowing them to quickly read data during high-performance computing. However, to ensure data consistency, the same data can only be cached in the read / write buffer layer of one client. If multiple clients cache the same data from the server, that data will become invalid from the client that cached it first. This prevents high-speed data read / write in multi-client scenarios, impacting the efficiency of high-performance computing.

[0004] In summary, the existing Lustre parallel file system cannot achieve high-speed data read and write in multi-client scenarios. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium to solve the technical problem that the Lustre parallel file system in the prior art cannot achieve high-speed data read and write in multi-client scenarios.

[0006] In a first aspect, this application provides a data processing method applied to a metadata service device, the metadata service device being located in a preset network architecture, the preset network architecture further including a storage service device and multiple client devices, the method comprising:

[0007] Receive a target data write request sent by a target client device; the target client device is one of multiple client devices, and the target data write request includes a target data identifier;

[0008] Based on the target data identifier, determine whether there is a write access client device among the multiple client devices that is performing a write operation on the target data;

[0009] if the write access client device exists, sending a target data write permission response to the target client device, the target data write permission response being used for instructing the target client device to write the target data into the storage service device;

[0010] if the read access client device exists, determining whether a read access client device performing a read operation on the target data exists in the plurality of client devices according to the target data identifier;

[0011] if the read access client device does not exist, sending a target data write permission response to the target client device, the target data write permission response being used for instructing the target client device to write the target data into the storage service device;

[0012] if the read access client device exists, sending a target data invalidation instruction to the read access client device and sending a target data write permission response to the target client device, the target data invalidation instruction being used for instructing the read access client device to perform invalidation processing on the target data.

[0013] Optionally, the method as described above, the determining whether a write access client device performing a write operation on the target data exists in the plurality of client devices according to the target data identifier comprises:

[0014] obtaining a write access device list corresponding to the target data according to the target data identifier;

[0015] if the write access device list is empty, determining that the write access client device does not exist;

[0016] if the write access device list is not empty, determining that the write access client device exists.

[0017] Optionally, the method as described above, after the sending of the target data write permission response to the target client device, further comprises:

[0018] adding an identifier of the target client device into the write access device list.

[0019] Optionally, the method as described above, after the sending of the target data write permission response to the target client device, further comprises:

[0020] receiving a target data write completion response sent by the target client device;

[0021] in response to the target data write completion response, deleting the identifier of the target client device from the write access device list.

[0022] Optionally, the method as described above, the determining whether a read access client device performing a read operation on the target data exists in the plurality of client devices according to the target data identifier comprises:

[0023] identifying a read access device list corresponding to the target data according to the target data identifier;

[0024] if the read access device list is empty, determining that the read access client device does not exist;

[0025] if the read access device list is not empty, determining that the read access client device exists.

[0026] Optionally, the method as described above, after the target data invalidation instruction is sent to the read access client device, further comprises:

[0027] deleting the identifier of the read access client device from the read access device list.

[0028] Optionally, the method as described above, further comprises:

[0029] receiving a target data read request sent by a target client device; the target data read request comprises a target data identifier and a cache start-stop identifier of the target client device; the cache start-stop identifier is cache enabled or cache disabled;

[0030] determining whether a write access client device performing a write operation on the target data exists in a plurality of client devices according to the target data identifier;

[0031] if the write access client device exists, sending a target data read restriction response to the target client device;

[0032] if the write access client device does not exist and the cache start-stop identifier is cache enabled, sending a target data first read permission response to the target client device, the target data first read permission response comprising a cache read identifier, so that the target client device reads the target data from the storage service device and stores it in its read-write cache;

[0033] if the write access client device does not exist and the cache start-stop identifier is cache disabled, sending a target data second read permission response to the target client device, the target data second read permission response comprising a direct read identifier, so that the target client device reads the target data from the storage service device and stores it in its processor.

[0034] Optionally, the method as described above, after the target data first read permission response is sent to the target client device, further comprises:

[0035] adding the identifier of the target client device to the read access device list.

[0036] Optionally, the method as described above, further comprises:

[0037] receiving a target data deletion request sent by a target client device; the target data deletion request comprising a target data identifier;

[0038] determining, according to the target data identifier, whether there is a read access client device performing a read operation on the target data among a plurality of client devices;

[0039] if there is no read access client device, sending a target data deletion instruction to a storage service device to make the storage service device delete the target data;

[0040] if there is the read access client device, sending a target data invalidation instruction to the read access client device to make the read access client device perform invalidation processing on the target data;

[0041] receiving a target data deletion response sent by the read access client device;

[0042] in response to the target data deletion response, sending a target data deletion instruction to a storage service device to make the storage service device delete the target data;

[0043] receiving a target data deletion response sent by a storage service device;

[0044] in response to the target data deletion response, sending a target data deletion success response to the target client device.

[0045] In a second aspect, the application provides a data processing method applied to a client device, wherein the client device is located in a preset network architecture, and the preset network architecture further comprises a metadata service device and a storage service device, and the data reading method comprises:

[0046] determining whether the target data exists in the read-write cache;

[0047] if it is determined that the target data exists in the read-write cache, determining whether the target data is invalid;

[0048] if it is determined that the target data is not invalid, reading the target data from the read-write cache to the processor;

[0049] if it is determined that the target data does not exist in the read-write cache or that the target data is invalid, sending a target data reading request to the metadata service device; the target data reading request comprising a target data identifier and a cache enabling identifier of the client device, for instructing the metadata service device to determine, according to the target data identifier, whether there is a write access client device performing a write operation on the target data among a plurality of client devices; if there is no write access client device, sending a target data first reading permission response to the target client device; the target data first reading permission response comprising a cache reading identifier;

[0050] reading the target data from the storage service device and storing the target data into the read-write cache in response to the target data first read permission response.

[0051] In a third aspect, the present application provides a data processing apparatus applied to a metadata service device, wherein the metadata service device is located in a preset network architecture, and the preset network architecture further comprises a storage service device and a plurality of client devices, and the apparatus comprises:

[0052] a receiving module configured to receive a target data write request sent by a target client device, wherein the target client device is one of the plurality of client devices, and the target data write request comprises a target data identifier;

[0053] a first determining module configured to determine, according to the target data identifier, whether there is a write access client device performing a write operation on the target data in the plurality of client devices;

[0054] a first sending module configured to send, if there is the write access client device, a target data write restriction response to the target client device;

[0055] a second determining module configured to determine, if there is not the write access client device, whether there is a read access client device performing a read operation on the target data in the plurality of client devices according to the target data identifier;

[0056] a second sending module configured to send, if there is not the read access client device, a target data write permission response to the target client device, wherein the target data write permission response is used to instruct the target client device to write the target data into the storage service device;

[0057] a third sending module configured to send, if there is the read access client device, a target data invalidation instruction to the read access client device and a target data write permission response to the target client device, wherein the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data.

[0058] In a fourth aspect, the present application provides a data processing apparatus applied to a client device, wherein the client device is located in a preset network architecture, and the preset network architecture further comprises a metadata service device and a storage service device, and the apparatus comprises:

[0059] a first determining module configured to determine whether target data exists in a read-write cache;

[0060] a second determining module configured to determine, if it is determined that the target data exists in the read-write cache, whether the target data is invalid;

[0061] The first reading module is configured to read the target data from the read-write cache into the processor if it is determined that the target data is not invalidated.

[0062] The sending module is configured to send a target data reading request to the metadata service device if it is determined that the target data does not exist in the read-write cache or that the target data is invalidated; the target data reading request comprises a target data identifier and a cache enabling identifier of the client device, and is used to instruct the metadata service device to determine whether there is a write access client device performing a write operation on the target data among a plurality of client devices according to the target data identifier; send a target data reading restriction response to the target client device if the write access client device exists; and send a target data first reading permission response to the target client device if the write access client device does not exist; the target data first reading permission response comprises a cache reading identifier.

[0063] The second reading module is configured to read the target data from the storage service device and store the target data into the read-write cache of the target client device in response to the target data first reading permission response.

[0064] In a fifth aspect, the present application provides a metadata service device, comprising a storage, a processor and a transceiver;

[0065] The storage, the processor and the transceiver are electrically interconnected;

[0066] The storage stores computer execution instructions;

[0067] The transceiver is used to transceive data;

[0068] The processor executes the computer execution instructions stored in the storage to implement the method according to the first aspect.

[0069] In a sixth aspect, the present application provides a client device, comprising a storage, a processor and a transceiver;

[0070] The storage, the processor and the transceiver are electrically interconnected;

[0071] The storage stores computer execution instructions;

[0072] The transceiver is used to transceive data;

[0073] The processor executes the computer execution instructions stored in the storage to implement the method according to the second aspect.

[0074] In a seventh aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions; the computer execution instructions are executed by a processor to implement the method according to the first aspect or the second aspect.

[0075] In an eighth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method according to the first aspect or the second aspect.

[0076] The data processing method, device, equipment and storage medium provided by the present application can avoid multiple client devices performing write operations on target data at the same time, and ensure data consistency of the target data in the storage service device and the multiple client devices. If there is no read access client device, a target data write permission response is sent to the target client device; the target data write permission response is used to instruct the target client device to write the target data into the storage service device; if there is the read access client device, a target data invalidation instruction is sent to the read access client device, and a target data write permission response is sent to the target client device; the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data. The target data can be cached in the multiple client devices at the same time while ensuring data consistency of the target data in the storage service device and the multiple client devices. Furthermore, the present application can realize high-speed data read and write of high-performance computing in a multi-client device scenario. BRIEF DESCRIPTION OF DRAWINGS

[0077] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the present application.

[0078] Figure 1 A network architecture diagram corresponding to an application scenario of the data processing method provided by the embodiment of the present application;

[0079] Figure 2 A flowchart of the data processing method provided by the first embodiment of the present application;

[0080] Figure 3 A flowchart of the data processing method provided by the second embodiment of the present application;

[0081] Figure 4A flowchart of a data processing method provided for Embodiment Three of the present application is shown in FIG. 3.

[0082] Figure 5 A flowchart of a data processing method provided for Embodiment Four of the present application is shown in FIG. 4.

[0083] Figure 6 A structural diagram of a data processing device provided for Embodiment Five of the present application is shown in FIG. 5.

[0084] Figure 7 A structural diagram of a data processing device provided for Embodiment Six of the present application is shown in FIG. 6.

[0085] Figure 8 A structural diagram of a metadata service device provided for Embodiment Seven of the present application is shown in FIG. 7.

[0086] Figure 9 A structural diagram of a client device provided for Embodiment Seven of the present application is shown in FIG. 8.

[0087] The specific embodiments of the present application have been shown in the above-described drawings, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0088] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same reference numbers in different drawings represent the same or similar elements unless otherwise represented. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0089] First, the professional terms involved in the present application are explained.

[0090] Lustre is an open-source, distributed parallel file system software platform.

[0091] Persistent Cache on Client (PCC) is the full name of the client persistent cache in English, and the abbreviation is PCC.

[0092] Non-Volatile Memory Express (NVME) is the full name of the non-volatile memory standard in English, and the abbreviation is NVME. It is a storage access and transmission protocol.

[0093] Wireless bandwidth, English full name for InfiniBand, English abbreviation for IB, is a computer network communication standard for high-performance computing, with extremely high throughput and extremely low delay.

[0094] Input / output, English full name for Input / Output, English abbreviation for I / O, refers to the input and output of data between internal memory and external memory or other peripheral devices.

[0095] Transmission Control Protocol, English full name for Transmission Control Protocol, English abbreviation for TCP, is a connection-oriented, reliable, byte-stream-based transport layer communication protocol.

[0096] Lustre parallel file system, under the hardware conditions of high-performance disk NVME plus IB network, the time of single I / O processing can be controlled within hundreds of microseconds, meeting the high-speed data read-write performance requirements of high-performance computing, but the hardware requirements of this scheme are too high. If TCP network and low-speed disk are used, it is impossible to meet the high-speed data read-write performance requirements of high-performance computing. Therefore, at present, the main way to meet the high-speed data read-write performance requirements of high-performance computing while reducing the hardware condition requirements is through the Lustre PCC scheme.

[0097] In the Lustre PCC scheme, the computing node is the lustre client, the storage node is the lustre server, the lustre client uses high-performance disk, the lustre server uses low-speed disk, and the read-write buffer layer is divided in the high-performance disk of the lustre client. Before the lustre client needs to process the data in the lustre server, the data in the lustre server is cached asynchronously to the read-write buffer layer by the lustre client, and then the lustre client can quickly read data from its read-write buffer layer when it needs to process data in the lustre server. At the same time, the lustre client can write the data generated by the operation into the read-write buffer layer and asynchronously write it into the lustre server, meeting the high-speed data read-write performance requirements of high-performance computing.

[0098] But in the Lustre PCC scheme, in order to ensure data consistency, the same data in the lustre server can only be cached in the read-write buffer layer of one Lustre client. If multiple lustre clients cache the same data from the lustre server, this data will be invalidated from the lustre client that caches this data first. This makes the Lustre PCC scheme unable to achieve high-speed data read-write in a multi-client scenario, affecting the efficiency of high-performance computing.

[0099] To solve the technical problems in the prior art, the present application proposes the following inventive concept: if multiple client devices can be allowed to cache data at the same time, and the cached data in the client devices is only allowed to be written by the client devices when there is a client device writing the data, then the data read efficiency between multiple computing nodes and storage nodes can be improved while ensuring data consistency, and the high-speed data read and write requirements of high-performance computing in a multi-client scenario can be met.

[0100] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0101] Figure 1 The application scenario of the data processing method provided by the embodiments of the present application corresponds to a network architecture diagram. As shown in Figure 1 The network architecture corresponding to the application scenario provided by the embodiments of the present application includes a metadata service device 10, a storage service device 11 and multiple client devices.

[0102] The metadata service device 10, the storage service device 11 and the multiple client devices are all configured with a lustre file system.

[0103] The metadata service device 10 receives a target data write request sent by a target client device 12. The target client device 12 is one of the multiple client devices. The target data write request includes a target data identifier.

[0104] The metadata service device 10 determines whether there is a write access client device in the multiple client devices that performs a write operation on the target data according to the target data identifier.

[0105] If there is a write access client device, a target data write restriction response is sent to the target client device 12.

[0106] If there is no write access client device, it is determined whether there is a read access client device 13 in the multiple client devices that performs a read operation on the target data according to the target data identifier.

[0107] If there is no read access client device 12, a target data write permission response is sent to the target client device 12; the target data write permission response is used to instruct the target client device 12 to write the target data to the storage service device 11.

[0108] If the read access client device 13 exists, the target data invalidation instruction is sent to the read access client device 13, and the target data write permission response is sent to the target client device 12; the target data invalidation instruction is used to instruct the read access client device 13 to perform invalidation processing on the target data.

[0109] Embodiment one

[0110] Figure 2 A flowchart of a data processing method provided by the present application is provided. As shown in the figure, the execution subject of the data processing method provided by the present application is a data processing device, which is located in a metadata service device. The metadata service device is located in a preset network architecture, and the preset network architecture further includes a storage service device and a plurality of client devices. The data processing method provided by the present application includes steps 201 to 206. Figure 2

[0111] Step 201, receiving a target data write request sent by a target client device; the target client device is one of the plurality of client devices, and the target data write request includes a target data identifier.

[0112] In an application scenario of the present application, the client device can be a computing node, which is used for training an artificial intelligence model. The storage service device can be a storage node, which is used for storing training data and model parameters of the artificial intelligence model, etc. The metadata service device is used for managing the data stored in the storage service device. The target data can be a model parameter of the artificial intelligence model. After the artificial intelligence model is trained, new model parameters are generated, and the client device needs to update the old model parameters to the new model parameters. Therefore, the client device can send a target data write request to the metadata service device, and the target write request includes a target data identifier.

[0113] The metadata service device receives the target data write request sent by the target client device, and determines whether the target client device can write the target data into the storage service device. If yes, a target data write permission response is sent to the target client device. After receiving the target data write permission response sent by the metadata service device, the target client device can write the new model parameters into the storage service device, so as to overwrite the old model parameters stored in the storage service device.

[0114] Step 202, determining whether there is a write access client device in the plurality of client devices which performs a write operation on the target data according to the target data identifier.

[0115] ​In this embodiment, the metadata service device stores metadata of all preset data stored in the storage service device, and the metadata includes a write access device list corresponding to the data. When a client device writes data into the storage service device, the identifier of the client device is added to the write access device list corresponding to the data; and when the client device finishes writing data into the storage service device, the identifier of the client device is deleted from the write access device list corresponding to the data.

[0116] If the write access device list corresponding to the target data is empty, it indicates that there is no write access client device in the plurality of client devices to perform a write operation on the target data.

[0117] If the write access device list corresponding to the target data is not empty, it indicates that there is a write access client device in the plurality of client devices to perform a write operation on the target data. If the write access device list corresponding to the target data is not empty, the number of write access client devices is one.

[0118] Optionally, step 202 further includes steps 301 to 303.

[0119] In step 301, the write access device list corresponding to the target data is obtained according to the target data identifier.

[0120] In this embodiment, the metadata service device can query the metadata of the target data from the metadata of all preset data stored in the storage service device according to the target data identifier, and further obtain the write access device list corresponding to the target data from the metadata of the target data.

[0121] In step 302, if the write access device list is empty, it is determined that there is no write access client device.

[0122] If the write access device list corresponding to the target data is empty, the metadata service device determines that there is no write access client device in the plurality of client devices to perform a write operation on the target data.

[0123] In step 303, if the write access device list is not empty, it is determined that there is a write access client device.

[0124] If the write access device list corresponding to the target data is not empty, the metadata service device determines that there is a write access client device in the plurality of client devices to perform a write operation on the target data.

[0125] The data processing method provided in the embodiment can quickly determine whether there is a write access client device performing a write operation on the target data in the plurality of client devices through the write access device list corresponding to the target data, so that the metadata service device responds to the target client device target data write request, and ensures the data consistency of the target data in the storage service device and the plurality of client devices.

[0126] In step 203, if there is a write access client device, a target data write restriction response is sent to the target client device.

[0127] In the embodiment, if there is a write access client device performing a write operation on the target data in the plurality of client devices, in order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, and avoid data conflict or error caused by simultaneous modification or write of the target data by the plurality of client devices, the metadata service device sends a target data write restriction response to the target client device to restrict the target client device from writing the target data into the storage service device. The target data write restriction response can include a target data identifier.

[0128] After receiving the target data write restriction response sent by the metadata service device, the target client device will not write the target data into the storage service device. The target client device can send a target data write request to the storage service device again after a preset time delay.

[0129] In step 204, if there is no write access client device, it is determined whether there is a read access client device performing a read operation on the target data in the plurality of client devices according to the target data identifier.

[0130] In the embodiment, if there is no write access client device performing a write operation on the target data in the plurality of client devices, in order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, it is further needed to determine whether there is a read access client device performing a read operation on the target data in the plurality of client devices.

[0131] In this embodiment, the metadata of the data further comprises a read access device list corresponding to the data. When a client device directly reads data from the storage service device, the identifier of the client device is added to the read access device list corresponding to the data; when the client device directly reads data from the storage service device ends, the identifier of the client device is deleted from the read access device list corresponding to the data. When a client device reads data from the storage service device and stores the data in the read-write cache of the client device, the identifier of the client device is added to the read access device list corresponding to the data; the identifier of the client device is deleted from the read access device list corresponding to the data when the data is deleted from the read-write cache of the client device or the client device marks the data in the read-write cache of the client device as invalid.

[0132] If the read access device list corresponding to the target data is empty, it means that there is no read access client device in the plurality of client devices to perform a read operation on the target data, that is, there is no read access client device in the plurality of client devices to read the target data or to cache the target data.

[0133] If the read access device list corresponding to the target data is not empty, it means that there is a read access client device in the plurality of client devices to perform a read operation on the target data, that is, there is a read access client device in the plurality of client devices to read the target data or to cache the target data. If the read access device list corresponding to the target data is not empty, the number of read access client devices is at least one.

[0134] Optionally, step 204 comprises steps 401 to 403.

[0135] Step 401: Obtain the read access device list corresponding to the target data according to the target data identifier.

[0136] In this embodiment, the metadata service device can query the metadata of the target data from the metadata of all preset data stored in the storage service device according to the target data identifier, and further obtain the read access device list corresponding to the target data from the metadata of the target data.

[0137] Step 402: If the read access device list is empty, it is determined that there is no read access client device.

[0138] If the read access device list corresponding to the target data is empty, the metadata service device determines that there is no read access client device in the plurality of client devices to perform a read operation on the target data.

[0139] Step 403: If the read access device list is not empty, it is determined that there is a read access client device.

[0140] If the read access device list corresponding to the target data is not empty, the metadata service device determines that there is a read access client device in the plurality of client devices that performs a read operation on the target data.

[0141] The data processing method provided in this embodiment can quickly determine whether there is a read access client device in the plurality of client devices that performs a read operation on the target data through the read access device list corresponding to the target data, so that the metadata service device responds to the target client device's target data write request, and ensures the data consistency of the target data in the storage service device and the plurality of client devices.

[0142] In step 205, if there is no read access client device, a target data write permission response is sent to the target client device; the target data write permission response is used to instruct the target client device to write the target data into the storage service device.

[0143] In this embodiment, if there is no read access client device in the plurality of client devices that performs a read operation on the target data, the metadata service device sends a target data write permission response to the target client device, and the target data write permission response can include the target data identifier.

[0144] The target client device receives the target data write permission response sent by the metadata service device, and writes the target data into the storage service device in response to the target data write permission response.

[0145] Optionally, the target data write permission response can include a storage address of the target data in the storage service device, so that the target client device writes the target data into the storage service device.

[0146] Optionally, the target client device stores metadata of all preset data stored in the storage service device, and the metadata further includes a storage address of the data in the storage service device. After receiving the target data write permission response, the target client device can query the storage address of the target data in the storage service device from the metadata of all preset data stored in the storage service device according to the target data identifier, and then writes the target data into the storage service device according to the storage address of the target data in the storage service device.

[0147] In step 206, if there is a read access client device, a target data invalidation instruction is sent to the read access client device, and a target data write permission response is sent to the target client device; the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data.

[0148] In this embodiment, if there is a read access client device performing a read operation on the target data among the plurality of client devices, in order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, the metadata service device can send a target data invalidation instruction to the read access client device, so that the read access client device performs invalidation processing on the target data. The target data invalidation instruction can include a target data identifier.

[0149] Optionally, after receiving the target data invalidation instruction sent by the metadata service device, the read access client device can delete or mark the target data in the read-write cache of the read access client device as invalid according to the target data identifier. The data marked as invalid in the read access client device will not be read by the processor of the read access client device, thereby ensuring the data consistency of the target data in the storage service device and the plurality of client devices. The processor can be a central CPU or a graphics processor GPU.

[0150] Optionally, after the read access client device completes the invalidation processing on the target data, the read access client device can send a target data invalidation response to the metadata service device. The metadata service device can send a target data write permission response to the target client device after receiving the target data invalidation response sent by the read access client device, or can send the target data write permission response to the target client device at the same time as sending the target data invalidation instruction to the read access client device.

[0151] The data processing method provided in the embodiment can avoid simultaneous write operations on the target data by the multiple client devices, and ensure data consistency of the target data in the storage service device and the multiple client devices. If there is no write access client device, it is determined whether there is a read access client device performing a read operation on the target data in the multiple client devices according to the target data identifier. If there is no read access client device, a target data write permission response is sent to the target client device. The target data write permission response is used to instruct the target client device to write the target data into the storage service device. If there is a read access client device, a target data invalidation instruction is sent to the read access client device, and a target data write permission response is sent to the target client device. The target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data. The target data can be simultaneously cached in the multiple client devices while ensuring data consistency of the target data in the storage service device and the multiple client devices. Furthermore, the application can realize high-speed data read and write in the high-performance computing of the multiple client device scenario.

[0152] Optionally, after the step 205 of "sending the target client device a target data write permission response", the method further includes a step 501.

[0153] Optionally, after the step 206 of "sending the target client device a target data write permission response", the method further includes a step 501.

[0154] The step 501 adds the identifier of the target client device to the write access device list.

[0155] In the embodiment, after receiving the target data write permission response, the target client device writes the target data into the storage service device. Therefore, the identifier of the target client device is added to the write access device list corresponding to the target data. In the process of writing the target data into the storage service device by the target client device, the metadata service device limits other client devices in the multiple client devices from writing the target data into the storage service device, thereby ensuring data consistency of the target data in the storage service device and the multiple client devices.

[0156] The data processing method provided in the embodiment can add the identifier of the target client device to the write access device list after sending the target data write permission response to the target client device, can update the write access device list corresponding to the target data in real time, can provide high-speed data read and write for high-performance computing, and can ensure data consistency of the target data in the storage service device and the plurality of client devices.

[0157] Optionally, after the step of sending the target data write permission response to the target client device in step 205, steps 601 to 602 are further included.

[0158] Optionally, after the step of sending the target data write permission response to the target client device in step 206, steps 601 to 602 are further included.

[0159] In step 601, a target data write completion response sent by the target client device is received.

[0160] In step 602, in response to the target data write completion response, the identifier of the target client device is deleted from the write access device list.

[0161] In the embodiment, after the target client device successfully writes the target data into the storage service device, the target data write completion response can be sent to the metadata service device. The metadata service device receives the target data write completion response sent by the target client device, and in response to the target data write completion response, deletes the identifier of the target client device from the write access device list corresponding to the target data. In this way, after the target client device successfully writes the target data into the storage service device, the metadata service device can allow one of the plurality of client devices to write the target data into the storage service device, or allow any of the plurality of client devices to read the target data from the storage service device, thereby ensuring data consistency of the target data in the storage service device and the plurality of client devices.

[0162] The data processing method provided in the embodiment can add the identifier of the target client device to the write access device list after sending the target data write permission response to the target client device, can update the write access device list corresponding to the target data in real time, can provide high-speed data read and write for high-performance computing, and can ensure data consistency of the target data in the storage service device and the plurality of client devices.

[0163] Optionally, after the step of sending the target data invalidation instruction to the read access client device in step 206, step 701 is further included.

[0164] In step 701, the identifier of the read access client device is deleted from the read access device list.

[0165] Since the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data, the read access client device will no longer read the target data from its read-write cache for calculation, and the data consistency of the target data in the storage service device and the plurality of client devices can be guaranteed, so the identification of the read access client device can be deleted from the read access device list.

[0166] Optionally, after receiving the target data invalidation instruction sent by the metadata service device, the read access client device performs invalidation processing on the target data, and can send a target data invalidation response to the metadata service device after completing the invalidation processing on the target data. The invalidation processing performed by the read access client device on the target data can be deleting the target data from the read-write cache of the read access client device, or marking the target data as invalid.

[0167] The metadata service device can receive the target data invalidation response sent by the read access client device, and delete the identification of the read access client device from the read access device list in response to the target data invalidation response.

[0168] The data processing method provided by the embodiment can update the read access device list corresponding to the target data in real time by deleting the identification of the read access client device from the read access device list after sending the target data invalidation instruction to the read access client device, and can provide high-speed data read-write for high-performance computing while ensuring the data consistency of the target data in the storage service device and the plurality of client devices.

[0169] Embodiment Two

[0170] Figure 3 A flowchart of the data processing method provided by Embodiment Two of the present application is shown in FIG. 8. As shown in FIG. 8, the data processing method provided by the embodiment, on the basis of Embodiment One, further includes steps 801 to 805. Figure 3

[0171] Step 801: receiving a target data read request sent by a target client device; the target data read request includes a target data identification and a cache start-stop identification of the target client device; the cache start-stop identification is cache enabled or cache disabled.

[0172] In an application scenario of the present application, the target data can be model parameters of an artificial intelligence model. During the training of the artificial intelligence model by the client device, the model parameters need to be read, and therefore a target data read request can be sent to the metadata service device to read the target data from the storage service device.

[0173] ​In this embodiment, the cache start-stop identifier is used to mark whether there is a read-write cache in the target client device, and the read-write cache is used to realize the read-write buffering between the client device and the storage service device, and realize the high-speed data read-write of high-performance computing. The user can configure whether the client device enables read-write buffering by configuring the cache start-stop identifier in the client device.

[0174] When the target client device needs to read the target data, if the cache start-stop identifier of the target client device is enabled, the target client device determines whether the target data exists in its read-write cache, and if the target data exists in its read-write cache and is not marked as invalid, the target client device directly reads the target data from its read-write cache to its processor.

[0175] If the target data does not exist in the read-write cache of the target client device or is marked as invalid, the target client device sends a target data read request to the metadata service device.

[0176] The metadata service device receives the target data read request sent by the target client device, and determines whether the target client device can read the target data from the storage service device.

[0177] If the metadata service device determines that the target client device can read the target data from the storage service device, a first read permission response is sent to the target client device, so that the target client device reads the target data from the storage service device and stores it in the read-write cache of the target client device. Alternatively, a second read permission response is sent to the target client device, so that the target client device reads the target data from the storage service device and stores it in the processor of the target client device.

[0178] Step 802, according to the target data identifier, determine whether there is a write access client device that performs a write operation on the target data in the plurality of client devices.

[0179] In this embodiment, the specific implementation of step 802 can refer to step 202, which will not be repeated here.

[0180] Step 803, if there is a write access client device, a target data read restriction response is sent to the target client device.

[0181] It can be understood that if the target data is being written by a client device other than the target client device in the plurality of client devices, in order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, the metadata service device determines that the target client device cannot read the target data from the storage service device.

[0182] In the embodiment, if there is a write access client device performing a write operation on the target data in the plurality of client devices, in order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, avoid data conflict or error caused by other client devices in the plurality of client devices reading the target data from the storage service device when the write access client device writes the target data to the storage service device, the metadata service device determines that the target client device cannot read the target data from the storage service device, sends a target data reading limitation response to the target client device, and limits the target client from reading the target data from the storage service device. The target data reading limitation response can include the target data identifier.

[0183] After the target client device receives the target data reading limitation response sent by the metadata service device, the target client device can send a target data reading request to the metadata service device again after a preset time delay.

[0184] If there is no write access client device and the cache start-stop identifier is cache enabled, a target data first reading permission response is sent to the target client device, and the target data first reading permission response includes a cache reading identifier, so that the target client device reads the target data from the storage service device and stores the target data in the read-write cache of the target client device.

[0185] If the metadata service device determines that the target client device can read the target data from the storage service device, the metadata service device needs to determine the way in which the target client reads the target data from the storage service device according to the cache start-stop identifier of the target client device. If the cache start-stop identifier of the target client device is cache enabled, it indicates that the target client device has a read-write cache, and the metadata service device sends a target data first reading permission response to the target client device.

[0186] The target data first reading permission response includes a cache reading identifier, and can also include a storage address of the target data in the storage service device.

[0187] After the target client device receives the target data first reading permission response sent by the metadata service device, the target client device reads the target data from the storage service device and stores the target data in the read-write cache of the target client device. The operation of the target client device reading the target data from the storage service device and storing the target data in the read-write cache of the target client device can be asynchronous operation with the operation of the target client device processing the target data.

[0188] If the cache start-stop identifier of the target client device is cache disabled, it indicates that the target client device does not have a read-write cache, and the metadata service device sends a target data second read permission response to the target client device.

[0189] If the cache start-stop identifier of the target client device is cache disabled, it indicates that the target client device does not have a read-write cache, and the metadata service device sends a target data second read permission response to the target client device.

[0190] The target data second read permission response includes a direct read identifier, and can also include a storage address of the target data in the storage service device. After receiving the target data second read permission response sent by the metadata service device, the target client device reads the target data from the storage service device and stores the target data into the processor of the target client device.

[0191] The data processing method provided in the embodiment can ensure data consistency of the target data in the storage service device and the plurality of client devices, and achieve simultaneous reading of the same target data by the plurality of client devices, thereby achieving high-speed data read-write for high-performance computing in the plurality of client device scenario.

[0192] Optionally, after the step of sending the target data first read permission response to the target client device in step 804, the method further includes step 901.

[0193] In step 901, the identifier of the target client device is added to the read access device list.

[0194] In the embodiment, after receiving the target data first read permission response, the target client device reads the target data from the storage service device and stores the target data into the read-write cache, so that the target data can be quickly read from the read-write cache to the processor for calculation when the target data is needed in the calculation process. Therefore, in order to ensure data consistency of the target data in the storage service device and the plurality of client devices, the identifier of the target client device needs to be added to the read access device list, so that when other client devices except the target client device write the target data into the storage service device, the target data in the read-write cache of the target client device is invalidated.

[0195] The data processing method provided in the embodiment can add the identifier of the target client device to the read access device list after sending the target data first read permission response to the target client device, can update the read access device list corresponding to the target data in real time, can provide high-speed data read and write for high-performance computing, and can ensure data consistency of the target data in the storage service device and the plurality of client devices.

[0196] Embodiment Three

[0197] Figure 4 A flowchart of the data processing method provided in Embodiment Three of the application is shown in FIG. 10. As shown in FIG. 10, the data processing method provided in the embodiment further includes steps 1001 to 1008 on the basis of Embodiment One or Two. Figure 4

[0198] Step 1001 receives a target data deletion request sent by a target client device; the target data deletion request includes a target data identifier.

[0199] In an application scenario of the application, if the target client device needs to delete the target data, the target client device sends a target data deletion request to the metadata service device. The metadata service device receives the target data deletion request sent by the target client device.

[0200] Step 1002 determines whether there is a read access client device that performs a read operation on the target data in the plurality of client devices according to the target data identifier.

[0201] In the embodiment, the specific implementation of step 1002 can be referred to step 204, and details are not described herein.

[0202] Step 1003 sends a target data deletion instruction to the storage service device to make the storage service device delete the target data if there is no read access client device.

[0203] In the embodiment, if there is no read access client device that performs a read operation on the target data in the plurality of client devices, the metadata service device can control the storage service device to delete the target data from the storage space of the storage service device. Specifically, the metadata service device can send a target data deletion instruction to the storage service device. The target data deletion instruction can include the target data identifier.

[0204] The metadata service device can also store metadata of all preset data stored by the storage service device, and can query the storage address of the target data in the storage service device from the metadata of all preset data stored by the storage service device according to the target data identifier, and then delete the target data according to the storage address of the target data in the storage service device.

[0205] ​The storage service device can send a target data deletion response to the metadata service device after deleting the target data.

[0206] If the read access client device exists, the metadata service device sends a target data invalidation instruction to the read access client device to make the read access client device perform invalidation processing on the target data.

[0207] If the read access client device exists in the plurality of client devices and performs a read operation on the target data, it indicates that the target data has been cached in the read-write cache of the read access client device. In order to ensure the data consistency of the target data in the storage service device and the plurality of client devices, the read access client device needs to perform invalidation processing on the target data that has been cached. Therefore, the metadata service device sends a target data invalidation instruction to the read access client device to make the read access client device perform invalidation processing on the target data. For details, refer to the description in step 206.

[0208] Step 1005, receiving a target data invalidation response sent by the read access client device.

[0209] In this embodiment, the read access client device can send a target data invalidation response to the metadata service device after completing the invalidation processing on the target data.

[0210] Step 1006, in response to the target data invalidation response, sending a target data deletion instruction to the storage service device to make the storage service device delete the target data.

[0211] In this embodiment, the metadata service device sends a target data deletion instruction to the storage service device to make the storage service device delete the target data after receiving the target data invalidation response. If the read access client device is multiple, the metadata service device needs to send a target data deletion instruction to the storage service device after receiving the target data invalidation response sent by each read access client device.

[0212] Step 1007, receiving a target data deletion response sent by the storage service device.

[0213] Step 1008, in response to the target data deletion response, sending a target data deletion success response to the target client device.

[0214] In this embodiment, the metadata service device receives the target data deletion response sent by the storage service device, which indicates that the target data has been successfully deleted from the storage service device. Therefore, the metadata service device can send a target data deletion success response to the target client device.

[0215] The data processing method provided in the embodiment can ensure the data consistency of the target data in the storage service device and the plurality of client devices by invalidating the cached data in the read-write cache of the read access client device when deleting the data.

[0216] Embodiment Four

[0217] Figure 5 A flowchart of the data processing method provided in Embodiment Four of the present application is shown in FIG. 11. As shown in FIG. 11, the data processing method provided in the embodiment is executed by a data processing device, which is located in a client device. The client device is located in a preset network architecture, which further includes a metadata service device and a storage service device. The method includes steps 1101 to 1105. Figure 5

[0218] Step 1101: Determine whether the target data exists in the read-write cache.

[0219] In the embodiment, the read-write cache is the read-write cache of the client device. The client device stores the metadata of the cached data stored in the read-write cache thereof. The metadata of the cached data includes the cached data identifier and the storage address of the cached data in the read-write cache. The client device can determine whether the target data exists in the read-write cache according to the metadata of the cached data stored in the read-write cache thereof. For example, if the target data identifier exists in the metadata of the cached data stored in the read-write cache of the client device, it indicates that the target data exists in the read-write cache of the client device. If the target data identifier does not exist in the metadata of the cached data stored in the read-write cache of the client device, it indicates that the target data does not exist in the read-write cache of the client device.

[0220] If the target data exists in the read-write cache of the client device, the client device can obtain the storage address of the target data in the read-write cache thereof from the metadata of the cached data stored in the read-write cache thereof.

[0221] Step 1102: If it is determined that the target data exists in the read-write cache, determine whether the target data is invalid.

[0222] The metadata of the cached data stored in the read-write cache of the client device can further include an invalid identifier or a valid identifier. If the metadata of the target data includes the invalid identifier, it is determined that the target data is invalid. At this time, it is possible that the preset network architecture has other devices in addition to the client device performing write operation on the target data. In order to ensure the data consistency of the target data in the preset network architecture, the target data is marked as invalid. If the metadata of the target data does not include the invalid identifier or includes the valid identifier, it is determined that the target data is not invalid.

[0223] ​Step 1103, if it is determined that the target data is not invalid, reading the target data from the read-write cache to the processor.

[0224] If the target data is not invalid, the client device can read the target data from the read-write cache to the processor to meet the high-speed data reading requirement of high-performance computing.

[0225] Step 1104, if it is determined that the target data does not exist in the read-write cache or the target data is invalid, sending a target data reading request to the metadata service device; the target data reading request includes a target data identifier and a cache enabling identifier of the client device, used to instruct the metadata service device to determine whether there is a write access client device performing a write operation on the target data among a plurality of client devices according to the target data identifier; if there is no write access client device, sending a target data first reading permission response to the target client device; the target data first reading permission response includes a cache reading identifier.

[0226] If the target data does not exist in the read-write cache or the target data is invalid, the target data needs to be obtained from the storage service device, therefore, a target data reading request is sent to the metadata service device to cache the target data to the read-write cache of the client device, so that the target data can be directly read from the read-write cache when it is needed to be read next time, meeting the high-speed data reading requirement of high-performance computing.

[0227] In this embodiment, whether there is a write access client device performing a write operation on the target data among a plurality of client devices is determined according to the target data identifier; if there is no write access client device, a target data first reading permission response is sent to the target client device; the target data first reading permission response includes a cache reading identifier. The action performed by the metadata service device after receiving the target data reading request is not described here.

[0228] Step 1105, in response to the target data first reading permission response, reading the target data from the storage service device and storing it to the read-write cache thereof.

[0229] Optionally, the target data reading request is also used to instruct the metadata service device to send a target data reading limitation response to the target client device if there is a write access client device; step 1105 further includes step 1106.

[0230] Step 1106, in response to the target data reading limitation response, sending the target data reading request to the metadata service device again after a preset time delay.

[0231] The data processing method provided by the embodiment determines whether the target data exists in the read-write cache; if it is determined that the target data exists in the read-write cache, it is determined whether the target data is invalid; if it is determined that the target data is not invalid, the target data is read from the read-write cache to the processor; if it is determined that the target data does not exist in the read-write cache or that the target data is invalid, a target data reading request is sent to the metadata service device; in response to a target data first reading permission response, the target data is read from the storage service device and stored in the read-write cache thereof. Since the target data can be directly read from the read-write cache or cached in the read-write cache for next reading, high-speed data reading for high-performance computing can be realized in a multi-client device scenario.

[0232] Embodiment five

[0233] Figure 6 The structural schematic diagram of the data processing apparatus provided by Embodiment Five of the present application is shown. As shown in the figure, the data processing apparatus provided by the embodiment is applied to a metadata service device, the metadata service device is located in a preset network architecture, the preset network architecture further includes a storage service device and a plurality of client devices, and the data processing apparatus 60 includes a receiving module 61, a first determination module 62, a first sending module 63, a second determination module 64, a second sending module 66 and a third sending module 66. Figure 6

[0234] The receiving module 61 is configured to receive a target data writing request sent by a target client device; the target client device is one of the plurality of client devices, and the target data writing request includes a target data identifier.

[0235] The first determination module 62 is configured to determine, according to the target data identifier, whether there is a write access client device in the plurality of client devices that performs a write operation on the target data.

[0236] The first sending module 63 is configured to send a target data writing restriction response to the target client device if there is a write access client device.

[0237] The second determination module 64 is configured to determine, according to the target data identifier, whether there is a read access client device in the plurality of client devices that performs a read operation on the target data if there is no write access client device.

[0238] The second sending module 65 is configured to send a target data writing permission response to the target client device if there is no read access client device; the target data writing permission response is used to instruct the target client device to write the target data to the storage service device.

[0239] ​The third sending module 66 is configured to send a target data invalidation instruction to the read access client device if the read access client device exists, and send a target data write permission response to the target client device; the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data.

[0240] Optionally, the first determining module is specifically configured to:

[0241] obtain a write access device list corresponding to the target data according to the target data identifier;

[0242] if the write access device list is empty, it is determined that there is no write access client device;

[0243] if the write access device list is not empty, it is determined that there is a write access client device.

[0244] Optionally, the data processing apparatus further comprises a list modification module, and the list modification is configured to:

[0245] add the identifier of the target client device to the write access device list.

[0246] Optionally, the list modification is further configured to:

[0247] receive a target data write completion response sent by the target client device;

[0248] in response to the target data write completion response, delete the identifier of the target client device from the write access device list.

[0249] Optionally, the second determining module is specifically configured to:

[0250] obtain a read access device list corresponding to the target data according to the target data identifier;

[0251] if the read access device list is empty, it is determined that there is no read access client device;

[0252] if the read access device list is not empty, it is determined that there is a read access client device.

[0253] Optionally, the list modification is further configured to:

[0254] delete the identifier of the read access client device from the read access device list.

[0255] Optionally, the data processing apparatus further comprises a data reading module, and the data reading module is configured to:

[0256] receive a target data reading request sent by the target client device; the target data reading request comprises a target data identifier and a cache start-stop identifier of the target client device; the cache start-stop identifier is cache enabled or cache disabled;

[0257] determining whether there is a write access client device performing a write operation on the target data among the plurality of client devices according to the target data identifier;

[0258] if there is a write access client device, sending a target data read restriction response to the target client device;

[0259] if there is no write access client device and the cache start-stop identifier is cache enabled, sending a target data first read permission response to the target client device, the target data first read permission response comprising a cache read identifier, so that the target client device reads the target data from the storage service device and stores the target data in its read-write cache;

[0260] if there is no write access client device and the cache start-stop identifier is cache disabled, sending a target data second read permission response to the target client device, the target data second read permission response comprising a direct read identifier, so that the target client device reads the target data from the storage service device and stores the target data in its processor.

[0261] Optionally, the list modification is further used for:

[0262] adding the identifier of the target client device to the read access device list.

[0263] Optionally, the data processing apparatus further comprises a data deletion module, the data deletion module being configured to:

[0264] receive a target data deletion request sent by the target client device; the target data deletion request comprising a target data identifier;

[0265] determining whether there is a read access client device performing a read operation on the target data among the plurality of client devices according to the target data identifier;

[0266] if there is no read access client device, sending a target data deletion instruction to the storage service device, so that the storage service device deletes the target data;

[0267] if there is a read access client device, sending a target data invalidation instruction to the read access client device, so that the read access client device performs invalidation processing on the target data;

[0268] receiving a target data deletion response sent by the read access client device;

[0269] in response to the target data deletion response, sending a target data deletion instruction to the storage service device, so that the storage service device deletes the target data;

[0270] receiving a target data deletion response sent by the storage service device;

[0271] In response to the target data deletion response, a target data deletion success response is sent to the target client device.

[0272] The data processing apparatus provided by the embodiment can execute any one of the data processing methods provided by any one of the first to third embodiments, and the specific implementation manner and principle are similar, which will not be described here.

[0273] Embodiment six

[0274] Figure 7 The structure schematic diagram of the data processing apparatus provided by the embodiment six of the present application is shown. As shown in the figure, the data processing apparatus provided by the embodiment is applied to a client device, the client device is located in a preset network architecture, the preset network architecture further includes a metadata service device and a storage service device, and the data processing apparatus 70 includes a first determination module 71, a second determination module 72, a first reading module 73, a sending module 74 and a second reading module 75. Figure 7 The first determination module 71 is configured to determine whether the target data exists in the read-write cache.

[0275] The second determination module 72 is configured to, if it is determined that the target data exists in the read-write cache, determine whether the target data is invalid.

[0276] The first reading module 73 is configured to, if it is determined that the target data is not invalid, read the target data from the read-write cache to the processor.

[0277] The sending module 74 is configured to, if it is determined that the target data does not exist in the read-write cache or it is determined that the target data is invalid, send a target data reading request to the metadata service device; the target data reading request includes a target data identifier and a cache enabling identifier of the client device, and is used to instruct the metadata service device to: determine whether there is a write access client device performing a write operation on the target data among a plurality of client devices according to the target data identifier; if there is a write access client device, send a target data reading restriction response to the target client device; if there is no write access client device, send a target data first reading permission response to the target client device; the target data first reading permission response includes a cache reading identifier.

[0278] The second reading module 75 is configured to, in response to the target data first reading permission response, read the target data from the storage service device and store the target data in the read-write cache.

[0279]

[0280] ​Optionally, the target data reading request is further configured to instruct the metadata service device to send a target data reading restriction response to the target client device if the write access client device exists.

[0281] The data processing apparatus provided in the embodiment can execute any one of the data processing methods provided in the above-mentioned embodiment four, and the specific implementation manner and principle are similar, and thus will not be described here again.

[0282] Embodiment seven

[0283] Figure 8 A structural schematic diagram of the metadata service device provided in the embodiment seven of the present application is shown. The metadata service device is located in a preset network architecture, and the preset network architecture further includes a storage service device and a plurality of client devices. Figure 8 As shown in the figure, the metadata service device 80 provided in the embodiment includes a memory 81, a processor 82 and a transceiver 83.

[0284] The memory 81, the processor 82 and the transceiver 83 are circuit-interconnected.

[0285] The memory 81 stores computer execution instructions.

[0286] The transceiver 83 is configured to transceive data.

[0287] The processor 82 executes the computer execution instructions stored in the memory 81, and realizes any one of the data processing methods provided in the above-mentioned embodiments one to three, and the specific implementation manner and principle are similar, and thus will not be described here again.

[0288] Figure 9 A structural schematic diagram of the client device provided in the embodiment seven of the present application is shown. The client device is located in a preset network architecture, and the preset network architecture further includes a metadata service device and a storage service device. Figure 9 As shown in the figure, the metadata service device 90 provided in the embodiment includes a memory 91, a processor 92 and a transceiver 93.

[0289] The memory 91, the processor 92 and the transceiver 93 are circuit-interconnected.

[0290] The memory 91 stores computer execution instructions.

[0291] The transceiver 93 is configured to transceive data.

[0292] The processor 92 executes the computer execution instructions stored in the memory 91, and realizes any one of the data processing methods provided in the above-mentioned embodiment four, and the specific implementation manner and principle are similar, and thus will not be described here again.

[0293] The memory, the processor and the transceiver can be electrically interconnected through a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 8 and Figure 9 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0294] Embodiments of the present application also provide a computer readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the data processing method provided by any one of the above embodiments. The specific implementation manners and principles are similar, and will not be described here. Exemplarily, the computer readable storage medium can be a read-only memory, a random access memory, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0295] Embodiments of the present application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the data processing method provided by any one of the above embodiments. The specific implementation manners and principles are similar, and will not be described here.

[0296] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0297] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0298] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0299] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0300] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0301] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0302] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application

[0303] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains or can relate. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0304] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A data processing method, characterized by, The method is applied to a metadata service device, the metadata service device is located in a preset network architecture, the preset network architecture further comprises a storage service device and a plurality of client devices, and the method comprises the following steps: receiving a target data write request sent by a target client device; the target client device is one of the plurality of client devices, and the target data write request comprises a target data identifier; determining whether a write access client device that performs a write operation on the target data exists in the plurality of client devices according to the target data identifier; if the write access client device exists, sending a target data write restriction response to the target client device; if the write access client device does not exist, determining whether a read access client device that performs a read operation on the target data exists in the plurality of client devices according to the target data identifier; if the read access client device does not exist, sending a target data write permission response to the target client device; the target data write permission response is used for instructing the target client device to write the target data into the storage service device; if the read access client device exists, sending a target data invalidation instruction to the read access client device and sending a target data write permission response to the target client device; the target data invalidation instruction is used for instructing the read access client device to perform invalidation processing on the target data.

2. The method of claim 1, wherein, The step of determining whether the write access client device exists in the plurality of client devices according to the target data identifier comprises the following steps: obtaining a write access device list corresponding to the target data according to the target data identifier; if the write access device list is empty, determining that the write access client device does not exist; if the write access device list is not empty, determining that the write access client device exists.

3. The method of claim 2, wherein, After the step of sending the target data write permission response to the target client device, the method further comprises the following steps: adding an identifier of the target client device into the write access device list.

4. The method of claim 2, wherein, After the step of sending the target data write permission response to the target client device, the method further comprises the following steps: receiving a target data write completion response sent by the target client device; in response to the target data write completion response, deleting the identifier of the target client device from the write access device list.

5. The method of claim 1, wherein, The step of determining whether the read access client device exists in the plurality of client devices according to the target data identifier comprises the following steps: obtaining a read access device list corresponding to the target data according to the target data identifier; if the read access device list is empty, determining that the read access client device does not exist; if the read access device list is not empty, determining that the read access client device exists.

6. The method of claim 5, wherein, After the step of sending the target data invalidation instruction to the read access client device, the method further comprises the following step: deleting the identifier of the read access client device from the read access device list.

7. The method of claim 5, wherein, The method further comprises the following steps: receiving a target data read request sent by a target client device; the target data read request comprises a target data identifier and a cache start-stop identifier of the target client device; the cache start-stop identifier is cache enabled or cache disabled; determining whether there is a write access client device performing a write operation on the target data among the plurality of client devices according to the target data identifier; if there is the write access client device, sending a target data read restriction response to the target client device; if there is no write access client device and the cache start-stop identifier is cache enabled, sending a target data first read permission response to the target client device, the target data first read permission response comprising a cache read identifier, so that the target client device reads the target data from the storage service device and stores the target data in the read-write cache of the target client device; if there is no write access client device and the cache start-stop identifier is cache disabled, sending a target data second read permission response to the target client device, the target data second read permission response comprising a direct read identifier, so that the target client device reads the target data from the storage service device and stores the target data in the processor of the target client device.

8. The method of claim 7, wherein, after the sending of the target data first read permission response to the target client device, the method further comprises: adding the identifier of the target client device to the read access device list.

9. The method of claim 1, wherein, the method further comprises: receiving a target data deletion request sent by the target client device, the target data deletion request comprising a target data identifier; determining whether there is a read access client device performing a read operation on the target data among the plurality of client devices according to the target data identifier; if there is no read access client device, sending a target data deletion instruction to the storage service device, so that the storage service device deletes the target data; if there is the read access client device, sending a target data invalidation instruction to the read access client device, so that the read access client device performs invalidation processing on the target data; receiving a target data deletion response sent by the read access client device; in response to the target data deletion response, sending a target data deletion instruction to the storage service device, so that the storage service device deletes the target data; receiving a target data deletion response sent by the storage service device; in response to the target data deletion response, sending a target data deletion success response to the target client device.

10. A data processing method, characterized by, applied to a client device, the client device being located in a preset network architecture, the preset network architecture further comprising a metadata service device and a storage service device, the method comprising: determining whether the target data exists in the read-write cache; if it is determined that the target data exists in the read-write cache, determining whether the target data is invalid; if it is determined that the target data is not invalid, reading the target data from the read-write cache to the processor; If it is determined that the target data does not exist in the read-write cache or that the target data is invalid, a target data read request is sent to the metadata service device; the target data read request includes a target data identifier and a cache enabling identifier of the client device, and is used to instruct the metadata service device to determine, according to the target data identifier, whether there is a write access client device performing a write operation on the target data in the plurality of client devices; if there is no write access client device, a target data first read permission response is sent to the target client device; the target data first read permission response includes a cache read identifier; In response to the target data first read permission response, the target data is read from the storage service device and stored in the read-write cache of the target client device.

11. A data processing apparatus, characterized by The metadata service device is applied to a preset network architecture, and the preset network architecture further includes a storage service device and a plurality of client devices, and the device includes: A receiving module is configured to receive a target data write request sent by a target client device; the target client device is one of the plurality of client devices, and the target data write request includes a target data identifier; A first determining module is configured to determine, according to the target data identifier, whether there is a write access client device performing a write operation on the target data in the plurality of client devices; A first sending module is configured to send, if there is the write access client device, a target data write limitation response to the target client device; A second determining module is configured to determine, if there is no write access client device, whether there is a read access client device performing a read operation on the target data in the plurality of client devices according to the target data identifier; A second sending module is configured to send, if there is no read access client device, a target data write permission response to the target client device; the target data write permission response is used to instruct the target client device to write the target data to the storage service device; A third sending module is configured to send, if there is the read access client device, a target data invalidation instruction to the read access client device and a target data write permission response to the target client device; the target data invalidation instruction is used to instruct the read access client device to perform invalidation processing on the target data.

12. A metadata service device, characterized by comprise: a memory, a processor and a transceiver; the memory, the processor and the transceiver are electrically interconnected; the memory stores computer execution instructions; the transceiver is used to transceive data; the processor executes the computer execution instructions stored in the memory to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method according to any one of claims 1 to 10.

Images