Data consistency processing method and device based on distributed file system

Abstract

The invention provides a data consistency processing method and device based on a distributed file system, and the method comprises the steps: locking a path for writing target data through a mode of creating a lock file for the target data under a target path; whether other users compete for the same path in parallel or not is judged by judging whether the temporary lock file can be successfully created or not, if yes, a competition detection strategy is started, whether a conflicting fixed lock file exists or not is continuously determined, and if the conflicting fixed lock file exists, it is indicated that other users have already established the temporary lock file, and then the temporary lock file is successfully created. The temporary lock file is renamed as the fixed lock file, in order to avoid concurrent operation conflict with other users, a new temporary lock file is reestablished, a new temporary lock file which does not conflict is continuously tried out, and the temporary lock file is renamed as the fixed lock file; and finally, the target data is updated to a path corresponding to the fixed lock file after the temporary lock file is renamed, so that the consistency of the data is effectively guaranteed.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data consistency processing method and device based on a distributed file system. Background Art

[0002] A distributed file system is a data storage system that stores data in a distributed manner on multiple servers. HDFS (Hadoop Distributed File System) is a common distributed file system suitable for Hadoop big data computing engines. HDFS stores data in a distributed manner on multiple nodes in the entire data cluster in the form of data blocks.

[0003] In the actual use of HDFS, data inconsistency may easily occur due to concurrent operation conflicts, resulting in inconsistent data obtained by different users accessing the same folder, affecting the user experience. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a data consistency processing method and device based on a distributed file system to reduce the problem of data inconsistency in the distributed file system and ensure user experience.

[0005] In a first aspect, an embodiment of the present application provides a data consistency processing method based on a distributed file system, wherein the method comprises:

[0006] Receiving a data request from a user, wherein the data request carries target data to be written;

[0007] Creating a temporary lock file for the target data under the target path, and if the temporary lock file is successfully created, renaming the temporary lock file to a fixed lock file based on the target data using an atomic naming rule, wherein the lock file is used to lock the path corresponding to the lock file;

[0008] If the creation of the temporary lock file fails, a contention detection strategy is initiated to determine whether there is a conflicting fixed lock file;

[0009] If it exists, a new temporary lock file is created for the target data under the target path again, and it is determined again whether the new temporary lock file is created successfully. If the new temporary lock file fails to be created, the competition detection strategy is started again until the new temporary lock file does not have a conflicting fixed lock file;

[0010] If it does not exist, rename the temporary lock file to a fixed lock file based on the target data using an atomic naming rule;

[0011] The target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0012] In some possible embodiments, the temporary lock file and the fixed lock file are file locks in the distributed file system, the temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, wherein the second time period is longer than the first time period.

[0013] In some possible embodiments, after the step of updating the target data to a path corresponding to the fixed lock file after the temporary lock file is renamed, the method further includes:

[0014] Clean up the temporary lock file and delete the fixed lock file.

[0015] In some possible embodiments, the method further includes:

[0016] If the temporary lock file is renamed to a fixed lock file based on the target data by using the atomic naming rule, the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed;

[0017] If renaming the temporary lock file to a fixed lock file based on the target data using the atomic naming rule fails, the temporary lock file is cleared.

[0018] In a second aspect, an embodiment of the present application provides a data consistency processing device based on a distributed file system, wherein the device includes:

[0019] A receiving module, used for receiving a data request from a user, wherein the data request carries target data to be written;

[0020] A file naming processing module, used to create a temporary lock file for the target data under the target path, and if the temporary lock file is successfully created, the temporary lock file is renamed to a fixed lock file based on the target data using an atomic naming rule, wherein the lock file is used to lock the path corresponding to the lock file;

[0021] A contention detection module is used to start a contention detection strategy if the creation of the temporary lock file fails, and determine whether there is a conflicting fixed lock file; if so, create a new temporary lock file for the target data under the target path again, and determine again whether the new temporary lock file is successfully created; if the creation of the new temporary lock file fails, start the contention detection strategy again until the new temporary lock file does not have a conflicting fixed lock file;

[0022] The file naming processing module is further used to rename the temporary lock file to a fixed lock file based on the target data by using an atomic naming rule if there is no conflicting fixed lock file;

[0023] The data processing module is used to update the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0024] In some possible embodiments, the temporary lock file and the fixed lock file are file locks in the distributed file system, the temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, wherein the second time period is longer than the first time period.

[0025] In some possible embodiments, after the step of updating the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed, the data processing module is further used to:

[0026] Clean up the temporary lock file and delete the fixed lock file.

[0027] In some possible embodiments, the data processing module is specifically used to:

[0028] If the temporary lock file is renamed to a fixed lock file based on the target data by using the atomic naming rule, the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed;

[0029] If renaming the temporary lock file to a fixed lock file based on the target data using the atomic naming rule fails, the temporary lock file is cleared.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0031] Processor; and

[0032] Memory for storing programs,

[0033] The program includes instructions, and when the instructions are executed by the processor, the processor executes the data consistency processing method based on the distributed file system described in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the data consistency processing method based on a distributed file system described in the first aspect.

[0035] Beneficial effects of this application:

[0036] The present application provides a data consistency processing method and device based on a distributed file system, wherein the method locks the path used to write the target data by creating a lock file for the target data under the target path, and determines whether there are other users competing for the same path in parallel by whether the temporary lock file can be successfully created. If so, the competition detection strategy is started to continue to determine whether there are conflicting fixed lock files. If there are conflicting fixed lock files, it indicates that other users have already established the temporary lock file, and the temporary lock file is renamed to the fixed lock file. At this time, in order to avoid concurrent operation conflicts with other users, a new temporary lock file is re-established, and a new temporary lock file that does not conflict at all is continuously tried, and the temporary lock file is renamed to the fixed lock file, and finally the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0037] In this way, by using temporary lock files and fixed lock files for contention detection, different writing results for the same data during high-concurrency processing by different users can be avoided. When different users access the path corresponding to the same fixed lock file, they will only see the same copy of the data, which can effectively ensure the consistency of the data from the user's perspective and help to ensure the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 A schematic diagram of a process flow of a data consistency processing method based on a distributed file system provided in an embodiment of the present application is shown;

[0040] Figure 2 Another schematic diagram of a data consistency processing method based on a distributed file system provided in an embodiment of the present application is shown;

[0041] Figure 3A schematic diagram of a logical structure of a data consistency processing device based on a distributed file system provided in an embodiment of the present application is shown;

[0042] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0044] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0045] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0046] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0047] As described in the background technology, the HDFS distributed file system may cause data inconsistency due to concurrent operation conflicts during use. In order to solve the data inconsistency, the prior art often uses tools such as Zookeeper (an open source distributed coordination service), Google Spanner (a database system), and Chubby (a distributed lock service) to ensure the consistency of data in the distributed file system and ensure that all users or clients can see consistent data.

[0048] However, Zookeeper is relatively complex and requires running an independent service cluster. In addition, the configuration and management of Zookeeper are relatively complex. Google Spanner requires complex infrastructure and high costs, and has poor versatility. Chubby is an internal technology that is not open to external users. In addition, the implementation and maintenance of Chubby require high computing resources and are difficult to maintain.

[0049] In view of this, in the first aspect, the present application provides a data consistency processing method based on a distributed file system, which is applied to any electronic device with a data consistency processing function based on a distributed file system, including but not limited to a personal mobile terminal, a computer or a server. Figure 1 As shown, the method comprises the following steps:

[0050] S11, receiving a data request input by a user, wherein the data request carries target data to be written;

[0051] S12, creating a temporary lock file for the target data in the target path;

[0052] S13, determine whether the temporary lock file is created successfully, if successful, execute step S15; if failed, execute step S14;

[0053] S14, start the contention detection strategy to determine whether there is a conflicting fixed lock file; if so, return to step S12 until the created temporary lock file does not contain a conflicting fixed lock file; if not, execute step S15.

[0054] S15. Rename the temporary lock file to a fixed lock file based on the target data using an atomic naming rule; then execute step S16; wherein the lock file is used to lock the path corresponding to the lock file.

[0055] S16. Update the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0056] In an embodiment of the present application, a lock file is created for the target data under the target path to lock the path used to write the target data. Whether the temporary lock file can be successfully created is used to determine whether there are other users competing for the same path in parallel. If so, a competition detection strategy is initiated to continue to determine whether there is a conflicting fixed lock file. If there is a conflicting fixed lock file, it indicates that other users have already established the temporary lock file, and the temporary lock file has been renamed to the fixed lock file. At this time, in order to avoid concurrent operation conflicts with other users, a new temporary lock file is re-established, and a new temporary lock file that does not conflict at all is continuously tried, and the temporary lock file is renamed to the fixed lock file, and finally the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0057] In this way, by using temporary lock files and fixed lock files for contention detection, different writing results for the same data during high-concurrency processing by different users can be avoided. When different users access the path corresponding to the same fixed lock file, they will only see the same copy of the data, which can effectively ensure the consistency of the data from the user's perspective and help to ensure the user experience.

[0058] The above steps S11 to S16 are described in detail below with reference to specific examples:

[0059] In the embodiment of the present application, data consistency processing is performed on data in a distributed file system, where a distributed file system is different from an ordinary application or a single server. A distributed file system is a large data cluster. Taking HDFS as an example, the HDFS distributed file system is the core component of Hadoop, a big data computing engine. HDFS can realize reliable storage, processing, and calculation of big data on multiple hardware devices in the cluster.

[0060] As an implementation method, the data consistency processing method based on a distributed file system provided by the present application can be applied to any storage node in an HDFS cluster. As another implementation method, the data consistency processing method based on a distributed file system provided by the present application can be applied to any client in a data system implemented based on an HDFS cluster. Based on this, when executing step S11, data requests passed in by users according to their own needs can be received from each node or client. Since the present application is committed to ensuring data consistency, in this article, the type of data request is mainly a data write request, and the above-mentioned node or client can also receive requests other than write requests, such as read requests, query requests, etc.

[0061] As an example, if a user requires each client in the HDFS distributed file system to maintain the same system time, that is, the consistency of the system time needs to be determined, each client can achieve the effect of consistent system time by obtaining the preset reference time. Based on this, a user can preset a reference time path, and each client agrees to read the reference time stored under the reference time path from the reference time path, and set the reference time as the system time, so that the system consistency of each client can be achieved. At this time, the user can pass a data write request to the reference time path, and carry the reference time to be set in the data write request. The reference time is the target data.

[0062] Based on this, as an implementation method, when executing step S12, a temporary lock file is created for the target data under the target path. In an embodiment of the present application, a lock file is a mechanism for protecting data in a multi-threaded or multi-process environment, and can be specifically used to lock the path corresponding to the lock file. Once the lock file is successfully created, the path corresponding to the lock file is occupied, and other threads or processes can no longer apply to use the path. In an embodiment of the present application, a total of two types of lock files are involved: a temporary lock file (English name temlockfile) and a fixed lock file (lockfile), wherein the temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, and the relationship between the first time period and the second time period is: the second time period is longer than the first time period.

[0063] In the embodiment of the present application, the temporary lock file locks the path corresponding to the temporary lock file for a first time period, indicating that if a thread or process successfully creates the first temporary lock file at a certain time node, other threads or processes cannot continue to apply to use the path corresponding to the first temporary lock file within the first time period after the time node. In the embodiment of the present application, creating a temporary lock file for the target data under the target path is not created arbitrarily, but is created based on the file name of the target data.

[0064] Exemplarily, if the target data is a song named "Planting the Sun", the title of the song can be used as a reference to create a temporary lock file. Exemplarily, the temporary lock file is "Planting the Sun.temp", and the subscript suffix .temp indicates that the file is a temporary lock file. Similarly, taking the above-mentioned system time as an example, if the file name is reference time, a temporary lock file "reference time.temp" can be created for the reference time. Based on this, when executing step S13, it can be determined whether the temporary lock file is successfully created by detecting whether the created temporary lock file exists successfully under the target path. If it is created successfully, a temporary lock file exists under the path.

[0065] In the embodiment of the present application, the method of creating a lock file based on the file name of the target data and the method of naming the lock file with the file name of the target data are named or created based on the atomicity of the target data. The atomicity of the target data refers to the target data itself, which will not be randomly modified. An atomic operation specifically refers to a series of operations of the present application based on the target data. The entire atomic operation also has a characteristic: either it is completely executed successfully or it is not executed at all.

[0066] When step S13 is executed, if it is determined that the creation of the temporary lock file fails, step S14 is executed to start a contention detection strategy. This contention detection strategy is flexibly set by the operation and maintenance personnel of the HDFS distributed file system. Specifically, it can detect whether there is the same file name according to the file name, and specifically check whether there is a temporary lock file with the same file name. If it exists, it indicates that the temporary lock file with this file name has been occupied by other threads or processes, that is, there is a conflict in the temporary lock file and the temporary lock file cannot be successfully created.

[0067] Exemplarily, if user A wants to write an MP3 file in path X, and the file name of this MP3 file is Planting the Sun, at this time, a temporary lock file Planting the Sun.temp is created for user A. User B also wants to write an MP3 file in path X, and the file name of this MP3 is also called Planting the Sun. At this time, the file name of another temporary lock file to be created for user B is also called Planting the Sun.temp. If the Planting the Sun.temp exists when user B writes, it indicates that there are other users also requesting to write the MP3 file named Planting the Sun, and there will be a conflict in the temporary lock file. At this time, the temporary lock file Planting the Sun.temp cannot be successfully created for this user B.

[0068] As another implementation manner, since the embodiment of the present application also includes the process of renaming the temporary lock file to a fixed lock file, it is very easy to have a conflict between the temporary lock file and the fixed lock file when the conversion of the temporary lock file to the fixed lock file is completed. Among them, the conversion of the temporary lock file to the fixed lock file is specifically obtained by executing step S15 to perform a file rename. As an implementation manner, when step S15 is executed, using the atomic naming rule, the temporary lock file is renamed to a fixed lock file, which can be to delete the file name representing temporariness in the temporary lock file. Exemplarily, taking the above-mentioned.temp as the file suffix to identify the temporary lock file, when renaming the temporary lock file to a fixed lock file, the.temp can be directly deleted. Exemplarily, taking the above-mentioned temporary lock file Planting the Sun.temp as an example, renaming this temporary lock file to a fixed lock file can be directly modifying the file name from "Planting the Sun.temp" to "Planting the Sun".

[0069] Since the fixed lock file is directly converted based on the temporary lock file, the difference between the two may only be the suffix. Therefore, the temporary lock file cannot be successfully created. Another factor is the existence of a conflicting fixed lock file. Therefore, by further executing step S14, the competition detection strategy can be started to determine whether there is a conflicting fixed lock file. If so, the currently created temporary lock file cannot be used anymore, and the process returns to execute step S12 to create a new temporary lock file for the target data. At this point, creating a new temporary lock file for the target data is still based on the file name of the target data itself. For example, taking the conflict in the above-mentioned Seed Sun.temp as an example, a new temporary lock file can be re-established with "Seed Sun V1.temp".

[0070] In an embodiment of the present application, the core processing idea for concurrent conflicts is "for the same data path, whoever updates first gets the credit". Exemplarily, taking the above-mentioned writing of the seeding sun data as an example, if it is found in advance that no one is using path X, a temporary lock file seeding sun.temp is directly created at this time. However, in this process, there is another user who is also creating it, but the seeding sun.temp temporary lock file has been created successfully, then the user will no longer be able to create seeding sun.temp. Then, the temporary lock file is renamed as a fixed lock file to change the lock duration of the thread or process. Based on this, execute step S16 to update the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed. Specifically, the target data can be uploaded to the path corresponding to the renamed fixed lock file.

[0071] In the embodiment of the present application, atomic rename is the transactional guarantee of HDFS itself, that is, there are two states: either completely successful or failed without affecting the system status at all. The purpose of atomic rename is to ensure that when a temporary lock file is renamed to a fixed lock file, there can and can only be one state, either success or failure. In this way, it can be ensured that when user A tries to rename a temporary lock file, if the rename is not successful and the same temporary lock file name is preempted by user B, at this time, user A's temporary lock rename fails and automatically rolls back to wait for the next round, and will not mistakenly believe that he has successfully renamed it.

[0072] Exemplarily, taking the above-mentioned target data of "Planting the Sun" as an example, after renaming "Planting the Sun.temp" to "Planting the Sun", the MP3 file of "Planting the Sun" can be uploaded to the path corresponding to the fixed lock file of "Planting the Sun". Among them, the upload operation is not atomic. If the upload fails, it will automatically roll back, and the path corresponding to the fixed lock file of "Planting the Sun" is still empty. If the MP3 file of "Planting the Sun" is uploaded successfully, the original temporary lock file "Planting the Sun.temp" will become the corresponding fixed lock file of "Planting the Sun". If the upload fails, the temporary lock file "Planting the Sun.temp" will be deleted. If there are other users who have been trying to create the temporary lock file "Planting the Sun.temp", and at this time other users find that the previous user's upload of the MP3 file of "Planting the Sun" fails, the temporary lock file "Planting the Sun.temp" that other users try to create will be created successfully immediately. Otherwise, other users can only wait for the current user to upload successfully and change the fixed lock file before they can create the temporary lock file of "Planting the Sun.temp" again. If the current user's upload fails, then it will continue to preempt the temporary lock file of "Planting the Sun.temp".

[0073] As an implementation manner, during the process of updating the target data to the path corresponding to the fixed lock file after renaming the temporary lock file, if the target data is successfully updated, the temporary lock file will be automatically changed to a fixed lock file. If the update of the target data fails, the temporary lock file will be automatically invalidated. At this time, the temporary lock file can be deleted to reduce the locking of the thread or process by the temporary lock file, so as to release the thread or process resources.

[0074] In some embodiments, after step S16, the method further includes:

[0075] Clean the temporary lock file and delete the fixed lock file.

[0076] In some embodiments, the method further includes:

[0077] If the temporary lock file is successfully renamed to a fixed lock file based on the target data using the atomic naming rule, update the target data to the path corresponding to the fixed lock file after renaming the temporary lock file;

[0078] If the temporary lock file fails to be renamed to a fixed lock file based on the target data using the atomic naming rule, clean the temporary lock file.

[0079] Specifically, as Figure 2 shown, the data consistency processing method based on a distributed file system provided by the embodiments of the present application includes the following steps:

[0080] Execute step S21 to try to create a temporary lock file temlockfile. If the creation is successful, execute step S22 to rename the temporary lock file to the fixed lock file lockfile. If it fails, it indicates that there is another node that is also being updated. At this time, execute step S23 to start the competition detection strategy to check whether there is a conflicting fixed lock file lockfile. If it exists, execute step S24 to try to create another temporary lock file for competition. If it does not exist, execute step S25 to continue to rename the temporary lock file to the fixed lock file normally. After the attempt to create another temporary lock file in step S24 is successful, execute step S26 to rename the new temporary lock file to the fixed lock file. After the temporary lock file is successfully renamed to the fixed lock file in steps S22 and S26, execute step S27 to write the target data to the path corresponding to the fixed lock file, and then execute step S28 to complete the data update and delete the fixed lock file. If the renaming of the temporary lock file to the fixed lock file in step S22 or step S26 fails, execute step S29 to clean up the corresponding temporary lock file.

[0081] Among them, when there is a conflicting fixed lock file, an attempt to create another temporary lock file is specifically to create a temporary lock file in accordance with the principle of global uniqueness. Among them, the principle of global uniqueness means that under this principle, no other temporary lock file is allowed to be the same as the created temporary lock file. Exemplary, if there are three user groups: user A, user B, user C, and their respective resource permissions are: / root / a, / root / b, / root / c. Users in the three user groups seize the resources of their respective user groups, and there is no conflict between different user groups. The same user groups compare whether the temporary lock files are the same. If they are the same, it indicates that the temporary lock file has been preempted by other users in the same user group.

[0082] The data consistency processing method based on the distributed file system provided by this application is selected. With the help of native naming rules, when different users compete concurrently, a first-come-first-served method is adopted to create a temporary lock file, thereby avoiding data inconsistency caused by concurrent user operations and ensuring the order and consistency of operations of different users. In addition, when the creation of a temporary lock file fails, the competition detection strategy is started to check the status of the existing file, and then try to create a new temporary lock file, and the atomicity and validity of the operation are guaranteed by the renaming operation.

[0083] In a second aspect, an embodiment of the present application provides a data consistency processing device based on a distributed file system, wherein, Figure 3 As shown, the device 30 comprises:

[0084] The receiving module 301 is used to receive a data request from a user, wherein the data request carries target data to be written;

[0085] A file naming processing module 302 is used to create a temporary lock file for the target data under the target path. If the temporary lock file is successfully created, the temporary lock file is renamed to a fixed lock file based on the target data using an atomic naming rule, wherein the lock file is used to lock the path corresponding to the lock file;

[0086] The contention detection module 303 is used to start the contention detection strategy if the creation of the temporary lock file fails, and determine whether there is a conflicting fixed lock file; if so, create a new temporary lock file for the target data under the target path again, and determine again whether the new temporary lock file is successfully created; if the creation of the new temporary lock file fails, start the contention detection strategy again until the new temporary lock file does not have a conflicting fixed lock file;

[0087] The file naming processing module 302 is further configured to rename the temporary lock file to a fixed lock file based on the target data by using an atomic naming rule if there is no conflicting fixed lock file;

[0088] The data processing module 304 is used to update the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed.

[0089] In some possible embodiments, the temporary lock file and the fixed lock file are file locks in the distributed file system, the temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, wherein the second time period is longer than the first time period.

[0090] In some possible embodiments, after the step of updating the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed, the data processing module is further used to:

[0091] Clean up the temporary lock file and delete the fixed lock file.

[0092] In some possible embodiments, the data processing module is specifically used to:

[0093] If the temporary lock file is renamed to a fixed lock file based on the target data by using the atomic naming rule, the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed;

[0094] If renaming the temporary lock file to a fixed lock file based on the target data using the atomic naming rule fails, the temporary lock file is cleared.

[0095] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0096] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0097] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0098] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0099] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0100] refer to Figure 4 , the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0101] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0102] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0103] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned data consistency processing method based on a distributed file system may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the aforementioned data consistency processing method based on a distributed file system in any other appropriate manner (e.g., by means of firmware).

[0104] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0108] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0109] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data consistency processing method based on a distributed file system, characterized in that: The method comprises: Receiving a data request from a user, wherein the data request carries target data to be written; Creating a temporary lock file for the target data under the target path, and if the temporary lock file is successfully created, renaming the temporary lock file to a fixed lock file based on the target data using an atomic naming rule, wherein the lock file is used to lock the path corresponding to the lock file; If the creation of the temporary lock file fails, a contention detection strategy is initiated to determine whether there is a conflicting fixed lock file; If it exists, a new temporary lock file is created for the target data under the target path again, and it is determined again whether the new temporary lock file is created successfully. If the new temporary lock file fails to be created, the competition detection strategy is started again until the new temporary lock file does not have a conflicting fixed lock file; If it does not exist, rename the temporary lock file to a fixed lock file based on the target data using an atomic naming rule; The target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed.

2. The method according to claim 1, characterized in that The temporary lock file and the fixed lock file are file locks in the distributed file system. The temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, wherein the second time period is longer than the first time period.

3. The method according to claim 2, characterized in that After the step of updating the target data to a path corresponding to the fixed lock file after the temporary lock file is renamed, the method further includes: Clean up the temporary lock file and delete the fixed lock file.

4. The method according to claim 1, characterized in that: The method further comprises: If the temporary lock file is renamed to a fixed lock file based on the target data by using the atomic naming rule, the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed; If renaming the temporary lock file to a fixed lock file based on the target data using the atomic naming rule fails, the temporary lock file is cleared.

5. A data consistency processing device based on a distributed file system, characterized in that: The device comprises: A receiving module, used for receiving a data request from a user, wherein the data request carries target data to be written; A file naming processing module, used to create a temporary lock file for the target data under the target path, and if the temporary lock file is successfully created, the temporary lock file is renamed to a fixed lock file based on the target data using an atomic naming rule, wherein the lock file is used to lock the path corresponding to the lock file; A contention detection module is used to start a contention detection strategy if the creation of the temporary lock file fails, and determine whether there is a conflicting fixed lock file; if so, create a new temporary lock file for the target data under the target path again, and determine again whether the new temporary lock file is successfully created; if the creation of the new temporary lock file fails, start the contention detection strategy again until the new temporary lock file does not have a conflicting fixed lock file; The file naming processing module is further used to rename the temporary lock file to a fixed lock file based on the target data by using an atomic naming rule if there is no conflicting fixed lock file; The data processing module is used to update the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed.

6. The device according to claim 5, characterized in that The temporary lock file and the fixed lock file are file locks in the distributed file system. The temporary lock file is used to lock the path corresponding to the temporary lock file for a first time period, and the fixed lock file is used to lock the path corresponding to the fixed lock file for a second time period, wherein the second time period is longer than the first time period.

7. The device according to claim 6, characterized in that After the step of updating the target data to the path corresponding to the fixed lock file after the temporary lock file is renamed, the data processing module is further used to: Clean up the temporary lock file and delete the fixed lock file.

8. The device according to claim 5, characterized in that The data processing module is specifically used for: If the temporary lock file is renamed to a fixed lock file based on the target data by using the atomic naming rule, the target data is updated to the path corresponding to the fixed lock file after the temporary lock file is renamed; If renaming the temporary lock file to a fixed lock file based on the target data using the atomic naming rule fails, the temporary lock file is cleared.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-4.

Images