Data processing method and device and storage medium

Abstract

The invention discloses a data processing method and device and a storage medium, and relates to the technical field of computer storage, and the method comprises the steps: receiving a data writing request sent by a first client, and determining each first client object and to-be-written data of each first client object according to the data writing request; performing data fragmentation on the to-be-written data of each first client object to obtain a fragmented data set corresponding to each first client object; and based on an exclusive lock mechanism and an optimistic lock mechanism, executing a write-in operation on the fragmented data set corresponding to each first client object so as to write the fragmented data set corresponding to each first client object into the corresponding storage node, thereby improving the concurrent processing capability of the system and improving the user experience. And the data consistency under the condition of high concurrency in the distributed storage is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer storage technology, and in particular to a data processing method, device and storage medium. Background Art

[0002] Distributed data storage systems have become an important part of big data processing and cloud computing. Such systems achieve high data availability and scalability by storing data in multiple physical or logical nodes. However, while this distributed architecture brings many advantages, it also introduces complex data consistency issues.

[0003] In a distributed data storage system, data consistency means that the data between multiple nodes must be logically consistent, which is the key to ensuring data accuracy and reliability. To achieve this goal, the system usually adopts a synchronous update strategy, that is, storing the same data copies on multiple nodes and ensuring the consistency between these copies through a synchronization mechanism.

[0004] However, in high-concurrency situations, when multiple write requests access and modify data at the same time, data conflicts may occur, resulting in inconsistent data between corresponding nodes.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a data processing method, device and storage medium, aiming to solve the technical problem of data conflict in high concurrency situations in distributed storage.

[0007] To achieve the above purpose, the present application proposes a data processing method, which includes:

[0008] receiving a data writing request sent by a first client, and determining each first client object and data to be written of each first client object according to the data writing request;

[0009] Slice the data to be written of each first client object respectively to obtain a slicing data set corresponding to each first client object;

[0010] Based on the exclusive lock mechanism and the optimistic lock mechanism, a write operation is performed on the shard data set corresponding to each first client object, so as to write the shard data set corresponding to each first client object into the corresponding storage node.

[0011] In one embodiment, based on optimistic concurrency control, each shard data in the shard data set is written into the corresponding storage node using an exclusive lock;

[0012] After the data is successfully written, the exclusive lock applied to the shard data is released.

[0013] In one embodiment, receiving a modification request for target data sent by a second client;

[0014] Verify the modification permission of the second client object;

[0015] If the second client object has the modification authority of the target data, a copy of the target data is copied to obtain the backup target data;

[0016] Performing a modification operation on the backup target data to obtain the modified backup target data;

[0017] The modified backup target data is used to replace the target data in the storage node corresponding to the target data.

[0018] In one embodiment, after the thread of the second client obtains the write lock, it performs a write operation and replaces the target data in the corresponding storage node with the modified backup target data;

[0019] The second client thread acquires the read lock while holding the write lock;

[0020] After acquiring the read lock, the thread of the second client releases the write lock and the read lock in sequence.

[0021] In one embodiment, a data download request sent by a third client is received, and a third client object of the third client and an index identifier of target data are determined according to the data download request;

[0022] Verify download access rights of the third client object;

[0023] If the third client object has download access rights, the corresponding shard data is obtained from the corresponding storage node based on the index identifier of the target data to obtain the target data.

[0024] In one embodiment, the index identifiers of each shard data associated with the index identifier of the target data are obtained;

[0025] Get the storage node corresponding to the index identifier of each shard data;

[0026] Read the corresponding shard data from the corresponding storage node and splice them to obtain the target data.

[0027] In one embodiment, a data writing request sent by a first client object is received, and the data writing request is parsed to obtain encrypted data;

[0028] The encrypted data is used as the data to be written to the first client object.

[0029] In one embodiment, the encrypted data is data encrypted using end-to-end encryption technology.

[0030] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method as described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method as described above are implemented.

[0032] One or more technical solutions proposed in this application have at least the following technical effects:

[0033] By using optimistic locks and exclusive locks, while ensuring that data in the distributed storage system can only be modified by a single transaction to prevent data conflicts, the waiting time of other transactions is shortened, the concurrent processing capability of the distributed storage system is improved, and data consistency in the distributed storage system under high concurrency is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 This is a schematic diagram of the system framework of the data processing method of this application;

[0037] Figure 2 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0038] Figure 3 A detailed flow chart of step S30 in the first embodiment of the data processing method of this application;

[0039] Figure 4 A flowchart of the second embodiment of the data processing method of the present application is provided;

[0040] Figure 5 A detailed flow chart of step S05 in the second embodiment of the data processing method of this application;

[0041] Figure 6 A flowchart of the third embodiment of the data processing method of the present application is provided;

[0042] Figure 7 A detailed flow chart of step A30 in the third embodiment of the data processing method of the present application;

[0043] Figure 8 A detailed flow chart of step S10 in the third embodiment of the data processing method of this application;

[0044] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Distributed data storage systems have become an important part of big data processing and cloud computing. Such systems achieve high data availability and scalability by storing data in multiple physical or logical nodes. However, while this distributed architecture brings many advantages, it also introduces complex data consistency issues.

[0049] In a distributed data storage system, data consistency means that the data between multiple nodes must be logically consistent, which is the key to ensuring data accuracy and reliability. To achieve this goal, the system usually adopts a synchronous update strategy, that is, storing the same data copies on multiple nodes and ensuring the consistency between these copies through a synchronization mechanism.

[0050] However, in high-concurrency situations, when multiple write requests access and modify data at the same time, data conflicts may occur, resulting in inconsistent data between corresponding nodes.

[0051] In response to the above problems, the main solution of the present application is: receiving a data write request sent by the first client, and determining each first client object and the data to be written of each first client object according to the data write request; sharding the data to be written of each first client object respectively to obtain a sharded data set corresponding to each first client object; based on an exclusive lock mechanism and an optimistic lock mechanism, performing a write operation on the sharded data set corresponding to each first client object to write the sharded data set corresponding to each first client object into a corresponding storage node.

[0052] The solution of this application, by using optimistic concurrency control, exclusive locks and lock downgrade mechanisms, ensures that data can only be modified by a single transaction to prevent data conflicts, while reducing the waiting time of other transactions, improving the system's concurrent processing capabilities, and ensuring data consistency in high concurrency situations in distributed storage.

[0053] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a distributed storage system, etc. The following uses a distributed storage system as an example to illustrate this embodiment and the following embodiments.

[0054] It should be noted that, refer to Figure 1 This embodiment uses a distributed storage system for distributed data storage and uses a client management platform to interact with the distributed storage system. The distributed storage system includes a processing server and multiple node servers, each of which has a backup server. The client is a client terminal device. Figure 1 The number of node servers shown is for illustrative purposes only and does not constitute a specific limitation on the present application.

[0055] It should be noted that the first client, the second client, and the third client mentioned below may be two identical clients, three identical clients, or three different clients.

[0056] Based on this, the present application embodiment provides a data processing method, referring to Figure 2 , Figure 2 This is a flowchart of the first embodiment of the data processing method of the present application.

[0057] In this embodiment, the data processing method includes steps S10 to S30:

[0058] Step S10: receiving a data write request sent by a first client, and determining each first client object and data to be written of each first client object according to the data write request.

[0059] It should be noted that the client object is a user who has successfully registered in the client management platform. To ensure the account security of the client object, this embodiment adopts a multi-factor identity authentication mechanism, such as user password, SMS verification code, email link and other verification levels to reduce the risk of illegal access to the client object's account, effectively improving the account security of the client object and reducing potential security risks.

[0060] The above-mentioned data write request refers to a data upload request initiated by the client object to the distributed storage system through the client management platform. The above-mentioned first client object is the client object that makes the data write request, and the above-mentioned first client is the terminal device where the first client object is located.

[0061] The data to be written refers to the data and data path to be uploaded by the first client object, wherein the data format includes but is not limited to word, excel, PowerPoint, and txt. The data to be written corresponding to each first client object is different.

[0062] In a feasible embodiment, the first client object logs into its account, uploads the data and data path to be uploaded on the client management platform, and clicks the upload request. The client management platform will send the data, data path and account information to the distributed storage system. The distributed storage system will receive the data write request of the first client object. The data write request only serves as a prompt information. After the distributed storage system receives the data write request, it can obtain the uploaded data, data path and account information of the first client object linked to the data write request.

[0063] It should be noted that the account information of the first client object is associated with the data uploaded by it to perform data permission management operations. The data uploaded by the first client object is visible only to itself. Considering the flexibility of the system, this embodiment sets up a security management center in the client management platform to interact with the distributed storage system. The security management center sets up a permission management module to allow client objects to flexibly set access rights for different data, where the above access rights include but are not limited to specifying that a specific client object has read-only, download, and modify permissions for data. In addition, an audit module is also set up in the security management center to record the data operation behavior of the client object for later auditing and problem tracking.

[0064] Step S20, sharding the data to be written of each first client object to obtain a sharded data set corresponding to each first client object.

[0065] It should be noted that data sharding, also known as partitioning, is a method of dividing large data sets into smaller, more manageable data subsets. In distributed storage systems, data sharding processing aims to distribute data to multiple independent nodes to achieve load balancing, improve storage efficiency and access speed.

[0066] In this embodiment, the number of shards is determined by the size of the data to be written, so that after the processing server in the distributed system receives the data to be written, it divides the above data to be written according to the set number of shards, thereby obtaining a shard data set, which is conducive to the subsequent storage of the shard data on the node server.

[0067] Step S30: Based on the exclusive lock mechanism and the optimistic lock mechanism, a write operation is performed on the shard data set corresponding to each first client object to write the shard data set corresponding to each first client object into the corresponding storage node.

[0068] It should be noted that the exclusive lock, also known as the write lock, or X lock for short, is a basic lock type used to protect data objects from concurrent modifications in database transaction processing. That is, when transaction T adds an exclusive lock to data object A, the lock will prevent any other transaction from adding any type of lock to A until T releases the lock on A. This means that while T holds the exclusive lock, other transactions can neither read nor modify A. This mechanism ensures data consistency and integrity and prevents conflicts and data inconsistencies caused by concurrent writes.

[0069] Optimistic locking is different from exclusive locking. It is a concurrency control method and a concurrency control mechanism, mainly used in multi-threaded environments to ensure data consistency. Its core idea is to "optimistically read data in multiple transactions, and verify whether there is a conflict when submitting. If not, submit; if so, roll back and try again."

[0070] In this embodiment, the shard data is locked by using an exclusive lock to ensure that the shard data can be completely and correctly transmitted to the node server. At the same time, an optimistic locking mechanism is used to enable the system to simultaneously receive the data to be written of multiple first client objects, avoid duplicate data storage, improve the concurrent processing capability of the distributed storage system, and ensure data consistency.

[0071] Further, see Figure 3 , step S30 also includes steps S31-S32:

[0072] Step S31, based on optimistic concurrency control, each shard data in the shard data set is written into the corresponding storage node using an exclusive lock.

[0073] It should be noted that optimistic concurrency control (OCC) is a concurrency control method applied to transaction systems. Its principle is that during the working phase of a transaction, i.e., the execution phase of the upload operation, the distributed storage system receives the uploaded data. When the transaction reaches the commit phase, the distributed storage system performs data conflict detection. If a conflict is detected, the transaction will be aborted and re-executed.

[0074] In this embodiment, optimistic concurrency control is used to allow multiple first client objects to write the data to be uploaded to the distributed storage system in parallel and synchronously, and conflict detection is performed only when the first client object executes the transaction submission of data upload, thereby improving the concurrency performance of the system.

[0075] Step S32, after the data is successfully written, release the exclusive lock applied to the shard data.

[0076] It should be noted that in a distributed storage system, the sharding mechanism is to improve the efficiency of data storage and the speed of access, while the exclusive lock is to ensure the consistency and integrity of data during the writing process.

[0077] In this embodiment, after the write operation is completed, the distributed storage system needs to check the write result to ensure that the data has been successfully written to the corresponding node server. This can be achieved by verifying whether the written data is consistent with the original data, checking the write status of the node server, etc. When the distributed storage system confirms that the shard has been successfully written to the node server and there is no other data to be written, the distributed storage system can release the exclusive lock applied to the shard to ensure that other transactions can perform normal read and write operations on the shard data.

[0078] In some embodiments of the present application, reference Figure 4 The data processing method further includes steps S01 to S05:

[0079] Step S01: receiving a target data modification request sent by a second client.

[0080] It should be noted that the above-mentioned client object refers to the same as the client object described in step S10, which will not be repeated here. The second client object is the client object that makes the modification request in this embodiment, and the second client is the terminal device where the above-mentioned second client object is located.

[0081] The modification request is modification request information that the second client object requests the distributed storage system to modify designated data through the client management platform.

[0082] Step S02: verifying the modification authority of the second client object.

[0083] The definition of permissions has been described in step S10 and will not be repeated here. Verifying the modification permission means verifying whether the second client object has the modification permission for the target data it requests to modify. If so, the second client object can modify the target data. If not, the second client object cannot modify the target data, and the target data may even be read-only or invisible to the second client object.

[0084] In a feasible embodiment, if the above-mentioned second client object is also the first client object, and the target data it requests to modify is the data to be written that it requests to write when it is the first client object, then as mentioned in step S10, the data uploaded by the first client object is only visible to itself, and the second client object at this time has all access rights to the target data, including but not limited to modifying, downloading or sharing with other designated client objects. That is, in this case, the second client object can pass the permission verification and have the permission to modify the target data.

[0085] In other embodiments, if the second client object does not obtain the modification access authorization for the target data from the first client object of the target data, the second client object does not have the modification permission for the target data it requests to modify, cannot pass the permission verification, and cannot modify the target data, thereby ensuring that sensitive information is only accessed by authorized users, effectively reducing the risk of data leakage.

[0086] Step S03: if the second client object has the modification authority of the target data, a copy of the target data is made to obtain the backup target data.

[0087] Step S04: performing a modification operation on the backup target data to obtain modified backup target data.

[0088] Step S05: Use the modified backup target data to replace the target data in the storage node corresponding to the target data.

[0089] As described in steps S03 to S05, when the second client object has modification authority for the target data that it requests to modify, in a feasible embodiment, the distributed storage system will copy a copy of the target data, i.e., the backup target data, and the modification operation of the target data by the second client object will be performed on the backup target data to obtain the modified backup target data, which will then be used to replace the target data in the storage node where the target data is located, to ensure that the modified data meets expectations and avoid irreversible damage to the original data.

[0090] Further, see Figure 5 , step S05 also includes steps S051 to S053:

[0091] Step S051: After the thread of the second client obtains the write lock, it performs a write operation and replaces the target data in the corresponding storage node with the modified backup target data.

[0092] Step S052: The thread of the second client obtains the read lock while maintaining the write lock.

[0093] Step S053: After acquiring the read lock, the thread of the second client releases the write lock and the read lock in sequence.

[0094] As described in steps S051 to S053, the read-write lock switching process involves replacing the target data in the corresponding storage node with the modified backup target data. In one embodiment, after the second client object obtains the write lock to perform the write operation to complete the modification of the target data, it obtains the read lock while maintaining the write lock; and then releases the write lock and the read lock in sequence. On the one hand, this read-write lock switching method allows other threads to obtain the read lock immediately after the write lock is released, reducing lock competition and improving the concurrency performance of the system; on the other hand, this read-write lock switching method ensures that the current thread can immediately obtain the read lock and read the data after holding the write lock for the write operation, thereby preventing other threads from tampering with the data after the write operation is completed, which helps maintain data consistency in a multi-threaded environment.

[0095] This embodiment is implemented based on the first embodiment and the second embodiment. In this embodiment, after step S30 of the data processing method, refer to Figure 6 , further comprising steps A10 to A30:

[0096] Step A10: receiving a data download request sent by a third client, and determining a third client object of the third client and an index identifier of target data according to the data download request.

[0097] It should be noted that the above-mentioned client object refers to the same as the client object described in step S10, which will not be repeated here. The third client object is the client object that makes a data download request in this embodiment, and the third client is the terminal device where the above-mentioned third client object is located.

[0098] The data download request is download request information that the third client object requests the distributed storage system to download designated data through the client management platform.

[0099] In a distributed storage system, there is usually a metadata management system that records the information of each shard, such as shard ID, storage location, size, status, etc. When a file needs to be accessed, the system will search for the corresponding shard information in the metadata management system based on the file identifier, such as file name, file ID, etc., and the sharding strategy, where the sharding strategy includes but is not limited to range-based sharding, hash-based sharding, and list-based sharding.

[0100] The above-mentioned index identifier is the identification information corresponding to the target data requested to be downloaded by the third client object, so as to instruct the metadata management system to find the corresponding segment of the target data and complete the combination of the segments to obtain a complete data. The above-mentioned index identifier can be the ID or file name of the target data, or it can be an index identifier of a certain index mechanism set according to the segment storage rules.

[0101] In this embodiment, the third client object requests the distributed storage system to download target data through the client management platform. The distributed storage system receives the data download request for the target data sent by the third client object on the client management platform. The data download request only serves as a prompt information. After the distributed storage system receives the data download request, it can obtain the target data ID and account information of the third client object linked to the data download request.

[0102] Step A20: verifying the download access permission of the third client object.

[0103] The definition of permissions has been described in step S10 and will not be repeated here. Verifying the download access permission means verifying whether the third client object has the download access permission for the target data it requests to download. If so, the third client object can download the target data. If not, the third client object cannot download the target data, and the target data may even be read-only or invisible to the third client object.

[0104] In a feasible embodiment, if the above-mentioned third client object is also the first client object, and the target data it requests to download is the data to be written that it requests to write when it is the first client object, then as mentioned in step S10, the data uploaded by the first client object is only visible to itself, and the third client object at this time has all access rights to the target data, including but not limited to modifying, downloading or sharing with other designated client objects. That is, in this case, the third client object can pass the permission verification and have the download access right to the target data.

[0105] In another embodiment, if the third client object does not obtain the download access authorization for the target data from the first client object of the target data, the third client object does not have download access permission for the target data it requests to download, cannot pass the permission verification, and cannot download the target data, thereby ensuring that sensitive information is only accessed by authorized users, effectively reducing the risk of data leakage.

[0106] Step A30: If the third client object has download access rights, based on the index identifier of the target data, the corresponding shard data is obtained from the corresponding storage node to obtain the target data.

[0107] Further, see Figure 7 , step A30 includes steps A31 to A33:

[0108] Step A31, obtaining the index identifiers of each shard data associated with the index identifier of the target data.

[0109] In a distributed storage system, index identifiers are a key technology for quickly locating and accessing data stored on multiple nodes. The specific form and implementation of index identifiers may vary depending on different distributed storage systems and application scenarios. Index identifiers are usually unique to ensure that the shards of the target data can be accurately located in the distributed storage system, thereby combining them into the complete target data. Common types of index identifiers include, but are not limited to, hash indexes, B-tree / B+-tree indexes, and local and global indexes.

[0110] Step A32, obtaining the storage node corresponding to the index identifier of each shard data.

[0111] In a feasible embodiment, in the process of searching for corresponding shards, based on the index identifier of the target data, the distributed storage system first queries the metadata management system to obtain the storage locations and numbers of all shards of the target data, and then communicates with the nodes storing these shards to obtain the shard data.

[0112] Step A33, read the corresponding shard data from the corresponding storage node and splice them to obtain the target data.

[0113] In a feasible embodiment, after obtaining all the required shards, the distributed storage system needs to sort and merge these shards to restore the original file content. During the merging process, the distributed storage system may verify each shard to ensure the correctness of the data. If a shard is found to be damaged or lost, the distributed storage system may try to restore the data from other copies or backups. Once all shards are correctly sorted and merged, the distributed storage system will generate the final file and provide it to the user or application for use.

[0114] In this embodiment, when the third client object has the download access right to the target data it requests to download, the distributed storage system obtains the shards corresponding to the target data through the metadata management system based on the file name corresponding to the target data it requests to download, based on the hash sharding strategy, to combine into a complete target data, and provide it to the third client object, wherein the hash sharding strategy in the distributed storage system maps the shard data to different node servers to realize data processing. At the same time, by setting a backup server for each node server, when the node server fails or the data is damaged, the backup data on the backup server can still provide services, reducing the risk of data loss. Since this embodiment uses end-to-end encryption technology to encrypt data, after obtaining the complete target data, it is necessary to use the private key allocated to the download end to decrypt the combined complete target data in order to obtain the target data in a readable format, thereby ensuring the security of the data.

[0115] Further, see Figure 8 , step S10 also includes steps S11 to S12:

[0116] Step S11, receiving a data writing request sent by a first client object, and parsing the data writing request to obtain encrypted data.

[0117] In this embodiment, the data to be uploaded of the first client object is encrypted using a public key through end-to-end encryption technology. This encryption technology adopts a dynamic key management system, so that the generation, distribution and storage of keys are automated and secure. In addition, by using a channel encryption protocol, it is ensured that the above-mentioned data to be uploaded can be transmitted to the distributed storage system through a secure communication channel, wherein the channel encryption protocol includes but is not limited to the SSL / TLS encryption protocol, IPSec protocol, SSH protocol and WireGuard protocol to avoid being eavesdropped or tampered with during the transmission process, thereby ensuring the security of data storage. The processing server in the distributed storage system receives the data write request of the first client object sent by the client management platform, and obtains the encrypted data linked to the data write request to ensure the security and correctness of the data during the upload process, wherein the above-mentioned encrypted data includes but is not limited to the account information of the first client object, the file name and file path of the data to be uploaded.

[0118] Step S12: Use the encrypted data as the data to be written into the first client object.

[0119] In this embodiment, the distributed storage system does not perform decryption operations on data transmitted using end-to-end encryption technology. The encryption and decryption of the data are respectively performed by the first client object and the third client object, i.e., the data upload end and the data download end. The first client object and the third client object have a key pair, and no decryption operations are performed during the data transmission and storage process to prevent the data from being eavesdropped or tampered with by a third party.

[0120] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0121] The present application provides a data processing device, the data processing device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1. The data processing device provided by the present application may be Figure 1 The processing server in .

[0122] Reference below Fig. 9, which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0123] like Fig. 9 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0124] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0125] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of data conflict in distributed storage under high concurrency. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0126] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0127] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0128] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0129] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0130] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0131] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device: receives a data write request sent by a first client, determines each first client object and the data to be written of each first client object according to the data write request; performs data sharding on the data to be written of each first client object to obtain a sharded data set corresponding to each first client object; and performs a write operation on the sharded data set corresponding to each first client object based on an exclusive lock mechanism and an optimistic lock mechanism, so as to write the sharded data set corresponding to each first client object into a corresponding storage node.

[0132] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0135] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of data conflicts in high concurrency in distributed storage. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0136] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: receiving a data writing request sent by a first client, and determining each first client object and data to be written for each first client object according to the data writing request; Slice the data to be written of each of the first client objects respectively to obtain a slicing data set corresponding to each of the first client objects; Based on the exclusive lock mechanism and the optimistic lock mechanism, a write operation is performed on the shard data set corresponding to each of the first client objects, so as to write the shard data set corresponding to each of the first client objects into the corresponding storage node.

2. The data processing method according to claim 1, characterized in that: The step of performing a write operation on the shard data set corresponding to each of the first client objects based on an exclusive lock mechanism and an optimistic lock mechanism to write the shard data set corresponding to each of the first client objects into a corresponding storage node includes: Based on optimistic concurrency control, each shard data in the shard data set is written into the corresponding storage node using an exclusive lock; After the data is successfully written, the exclusive lock applied to the shard data is released.

3. The data processing method according to claim 1, characterized in that: The data processing method further includes: receiving a modification request for target data sent by a second client; Verify the modification permission of the second client object; If the second client object has the modification permission of the target data, a copy of the target data is copied to obtain the backup target data; Performing a modification operation on the backup target data to obtain modified backup target data; The modified backup target data is used to replace the target data in the storage node corresponding to the target data.

4. The data processing method according to claim 3, characterized in that: The step of using the modified backup target data to replace the target data in the corresponding storage node includes: After the thread of the second client obtains the write lock, it performs a write operation to replace the target data in the corresponding storage node with the modified backup target data; The thread of the second client acquires the read lock while maintaining the write lock; After acquiring the read lock, the thread of the second client releases the write lock and the read lock in sequence.

5. The data processing method according to claim 3, characterized in that: After the step of performing a write operation on the shard data set corresponding to each of the first client objects based on the exclusive lock mechanism and the optimistic lock mechanism to write the shard data set corresponding to each of the first client objects into the corresponding storage node, the method further includes: receiving a data download request sent by a third client, and determining a third client object of the third client and an index identifier of target data according to the data download request; Verifying the download access permission of the third client object; If the third client object has the download access right, the corresponding shard data is obtained from the corresponding storage node based on the index identifier of the target data to obtain the target data.

6. The data processing method according to claim 5, characterized in that: The step of obtaining the target data by obtaining the corresponding shard data from the corresponding storage node based on the index identifier of the target data comprises: Obtain the index identifiers of each shard data associated with the index identifier of the target data; Obtain the storage node corresponding to the index identifier of each shard data; The corresponding shard data is read from the corresponding storage node for splicing to obtain the target data.

7. The data processing method according to claim 5, characterized in that: The step of receiving a data writing request sent by a first client and determining each first client object and the data to be written of each first client object according to the data writing request comprises: receiving a data writing request sent by the first client object, and parsing the data writing request to obtain encrypted data; The encrypted data is used as the data to be written to the first client object.

8. The data processing method according to claim 7, characterized in that: The encrypted data is data encrypted using end-to-end encryption technology.

9. A data processing device, characterized in that: The data processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images