Data writing method and device, equipment, medium and program product

Abstract

The invention provides a data writing method which can be applied to the technical field of big data. The data writing method comprises the following steps: in response to received write-in data sent by a coordinator, simulating to write the write-in data into a storage node based on the write-in data and a first mirror data set to obtain a pre-write-in simulation result, the first mirror data set is obtained by copying an original storage data set stored in the storage node; sending the pre-write simulation result to a coordinator; a submission instruction sent by the coordinator is received, the write-in data is written into the original storage data set, and the submission instruction is generated under the condition that the coordinator determines that the pre-write-in simulation result and at least one other pre-write-in simulation result represent successful pre-write-in. The at least one other pre-write simulation result is from at least one other storage node different from the storage node. The invention further provides a data writing device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically to a data writing method, device, equipment, medium and program product. Background Art

[0002] With the advent of the big data era, multiple forms of data are needed to meet complex data usage needs. Since a database can usually only store one form of data, in order to store multiple different forms of data, it is necessary to set up multiple storage nodes based on different types of databases.

[0003] However, when writing data to multiple storage nodes simultaneously, if one or more of the multiple storage nodes fail to write data successfully due to an abnormality, data consistency among the multiple storage nodes cannot be guaranteed. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data writing method, apparatus, device, medium and program product.

[0005] According to the first aspect of the present disclosure, a data writing method is provided, comprising: in response to receiving write data sent by a coordinator, based on the write data and a first mirror data set, simulating writing the write data into the storage node to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; sending the pre-write simulation result to the coordinator; and receiving a commit instruction sent by the coordinator to write the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when determining that the pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0006] According to an embodiment of the present disclosure, based on the above-mentioned write data and the first mirror data set, the above-mentioned write data is simulated to be written into the above-mentioned storage node to obtain a pre-write simulation result, including: writing the above-mentioned write data into the above-mentioned first mirror data set to obtain a second mirror data set; performing validity verification on the above-mentioned second mirror data set to obtain a validity verification result; in the case where it is determined that the above-mentioned validity verification result indicates that the above-mentioned second mirror data set is valid, determining that the above-mentioned pre-write simulation result indicates that the pre-write is successful; and in the case where it is determined that the above-mentioned validity verification result indicates that the above-mentioned second mirror data set is not valid, determining that the above-mentioned pre-write simulation result indicates that the pre-write fails.

[0007] According to an embodiment of the present disclosure, the above-mentioned method also includes: in response to receiving a cancel commit instruction sent by the above-mentioned coordinator, deleting the above-mentioned second mirror data set, wherein the above-mentioned cancel commit instruction is generated by the above-mentioned coordinator when determining that any pre-write simulation result among the above-mentioned pre-write simulation result and at least one other pre-write simulation result represents a pre-write failure.

[0008] According to an embodiment of the present disclosure, the above-mentioned validity verification is performed on the above-mentioned second mirror data set to obtain a validity verification result, including: performing the same preprocessing on the above-mentioned write data in the above-mentioned second mirror data set and the above-mentioned first mirror data set, respectively, and then performing validity verification to obtain a first validity verification result, wherein the above-mentioned same preprocessing includes at least one of the following: semantic extraction, node data identification; and obtaining the above-mentioned validity verification result based on the above-mentioned first validity verification result and the second validity verification result, wherein the above-mentioned second validity verification result is obtained based on data logic verification performed on the above-mentioned write data.

[0009] According to an embodiment of the present disclosure, the above-mentioned write data is obtained through the following operations: the above-mentioned coordinator, in response to receiving the original write data sent by the client, performs data processing on the above-mentioned original write data based on the respective database types of the above-mentioned storage node and at least one other storage node different from the above-mentioned storage node, to obtain write data corresponding to each of the above-mentioned database types.

[0010] According to an embodiment of the present disclosure, the above-mentioned database type includes a graph database type, and the above-mentioned data processing is performed on the above-mentioned original write data to obtain write data corresponding to each of the above-mentioned database types, including: based on at least one entity involved in the above-mentioned original write data, generating node data corresponding to each of the above-mentioned entities; based on the above-mentioned original write data, obtaining edge data for characterizing the association relationship; and based on the above-mentioned node data and the above-mentioned edge data, generating write data corresponding to the above-mentioned graph database type.

[0011] According to an embodiment of the present disclosure, the above-mentioned database type includes a document database type, and the above-mentioned data processing is performed on the above-mentioned original write data to obtain write data corresponding to each of the above-mentioned database types, including: based on the attribute information of the above-mentioned original write data, determining a classification field for identifying the above-mentioned original write data; and based on the above-mentioned original write data and the above-mentioned classification field, generating write data corresponding to the above-mentioned document database type.

[0012] According to an embodiment of the present disclosure, the method further includes: determining a write result of writing the write data to the original storage data set, and sending the write result to the coordinator; and in response to receiving a rollback instruction sent by the coordinator, rolling back the original storage data set using the first mirror data set, wherein the rollback instruction is generated by the coordinator when determining that any one of the write result and at least one other write result indicates a write failure.

[0013] A second aspect of the present disclosure provides a data writing device, including: a first writing module, for responding to receiving write data sent by a coordinator, and simulating writing the write data into the storage node based on the write data and a first mirror data set to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; a sending module, for sending the pre-write simulation result to the coordinator; and a second writing module, for receiving a commit instruction sent by the coordinator, and writing the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when determining that the pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0014] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0017] According to an embodiment of the present disclosure, a first mirror data set is obtained by copying the original data storage set, the first mirror data set is used to simulate the data storage status of the original data storage set in the storage node, and the first mirror data set is used to simulate writing the write data to the storage node, so as to obtain a pre-commit simulation effect that can characterize whether the write data can be successfully written to the storage node. The write data is formally written to the original storage set only when the pre-write results of multiple storage nodes all indicate that the pre-write is successful, so as to avoid the failure of writing data due to the abnormality of any storage node, resulting in data inconsistency between multiple storage nodes, and ensure the data consistency between multiple storage nodes while avoiding the impact on the original storage data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 The application scenario diagram of the data writing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0020] Figure 2 A flowchart of a data writing method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3 Schematically shows a flowchart of the pre-submission stage according to an embodiment of the present disclosure;

[0022] Figure 4 A flowchart of the submission phase according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 Schematically shows a flowchart of the pre-submission stage according to another embodiment of the present disclosure;

[0024] Figure 6 Schematically shows a flowchart of a submission phase according to another embodiment of the present disclosure;

[0025] Figure 7 A flowchart of a data writing method according to a specific embodiment of the present disclosure is schematically shown;

[0026] Figure 8 A structural block diagram of a data writing device according to an embodiment of the present disclosure is schematically shown; and

[0027] Fig. 9 A block diagram of an electronic device suitable for implementing a data writing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0033] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0035] When writing data to multiple storage nodes at the same time, if one or more of the multiple storage nodes fail to write data successfully due to an exception, the data consistency between the multiple storage nodes cannot be guaranteed, which in turn affects the accuracy and reliability of the data.

[0036] In order to ensure data consistency, a method based on message queue coordination, two-phase commit protocol and delayed synchronization is usually adopted. The method based on message queue coordination requires the introduction of a message queue as a coordinator, and all write requests are placed in the message queue. After the consumer obtains the operation request from the queue, it uses the transactional consumption mode to write data to multiple storage nodes in sequence. However, the method based on message queue coordination divides transaction processing into two stages, which not only brings additional time overhead and throughput bottlenecks, but also requires the consumer process to run continuously. Once a failure occurs, it may cause data loss or duplicate writing.

[0037] The two-phase commit protocol-based approach introduces a transaction coordinator and uses the two-phase commit protocol to uniformly schedule write operations of multiple storage units to achieve strict transaction consistency. However, most non-relational databases lack support for the interface specification protocol (XA protocol) for distributed transaction processing, making it impossible to implement the two-phase commit protocol. The kernel of the non-relational database storage node needs to be modified, so the cost of implementing the two-phase commit protocol-based approach in non-relational databases is high. In addition, even if the storage node supports the XA protocol, the two-phase commit protocol requires locking the original storage data set in the storage node, resulting in congestion and low read and write efficiency.

[0038] The delayed synchronization method requires that the write operation be completed on one storage node first, and then asynchronously replicated to other storage nodes after submission. Although this method does not require direct coordination of multiple storage nodes, it has the problem of eventual consistency. Even if incremental synchronization is used, data inconsistency within the synchronization time window cannot be avoided, and it cannot meet the scenarios with high data real-time and consistency.

[0039] The embodiment of the present disclosure provides a data writing method, in response to receiving the write data sent by the coordinator, based on the write data and the first mirror data set, simulating writing the write data into the storage node to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; sending the pre-write simulation result to the coordinator; and receiving the commit instruction sent by the coordinator, writing the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both represent the success of the pre-write, and at least one other pre-write simulation result comes from at least one other storage node different from the storage node. The embodiment of the present application simulates writing the write data into the storage node by copying the first mirror data set based on the original storage data set, and only formally writes the write data to the storage node when it is determined that the pre-write is successful, which not only ensures the real-time data consistency between multiple storage nodes, but also ensures the performance of the storage node.

[0040] Figure 1 The application scenario diagram of the data writing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0041] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a coordinator 105, a first storage node 106, and a second storage node 107. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the coordinator 105, and between the coordinator 105 and the first storage node 106, the second storage node 107. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0042] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the coordinator 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0044] The coordinator 105 can be used to forward data processing requests issued by the first terminal device 101, the second terminal device 102 or the third terminal device 103 to the first storage node 106 or the second storage node 107, coordinate the first storage node 106 and the second storage node 107, and feed back the processing results of the first storage node 106 and the second storage node 107 (for example, web pages, information, or data obtained or generated according to user requests, etc.) to the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0045] The first storage node 106 and the second storage node 107 are used to store data, process data processing requests sent by the coordinator 105 , and return storage results to the coordinator 105 .

[0046] It should be noted that the data writing method provided in the embodiment of the present disclosure can generally be executed by the first storage node 106 and the second storage node 107. Accordingly, the data writing device provided in the embodiment of the present disclosure can generally be set in the first storage node 106 and the second storage node 107. The data writing method provided in the embodiment of the present disclosure can also be executed by a database or database cluster that is different from the first storage node 106 and the second storage node 107 and can communicate with the coordinator 105. Accordingly, the data writing device provided in the embodiment of the present disclosure can also be set in a database or database cluster that is different from the first storage node 106 and the second storage node 107 and can communicate with the coordinator 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, coordinators, and storage nodes in the embodiment is only for illustration. Any number of terminal devices, networks, coordinators, and storage nodes may be provided according to implementation requirements.

[0048] The following will be based on Figure 1 The scene described by Figure 2~Figure 7 The data writing method of the disclosed embodiment is described in detail.

[0049] Figure 2 The flowchart of the data writing method according to the embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the data writing method of this embodiment includes operations S210 to S230.

[0051] In operation S210, in response to receiving the write data sent by the coordinator, based on the write data and the first mirror data set, the write data is simulated to be written into the storage node to obtain a pre-write simulation result.

[0052] According to an embodiment of the present disclosure, a storage node may be any storage node in a heterogeneous database cluster. The heterogeneous database cluster may be implemented based on a variety of different types of databases, specifically, it may include relational databases, and non-relational databases such as graph databases, document databases, etc.

[0053] According to an embodiment of the present disclosure, the first mirror data set is obtained by copying the original storage data set stored in the storage node. Specifically, the original storage data set can be stored in the disk of the storage node, and the first mirror data set can be stored in the local cache of the storage node. In the process of copying the original storage data set to obtain the first mirror data set, an environment identical to the original storage data set can be established in the local cache of the storage node, and then a connection between the disk and the local cache is established, and the original storage set is copied from the disk to the local cache through the connection to obtain the first mirror data set.

[0054] According to an embodiment of the present disclosure, the writing of write data into the storage node is simulated based on the first mirror data set. Since the first mirror data set is copied from the original storage data set, the first mirror data set can simulate the data storage state of the original data storage set in the storage node.

[0055] According to the embodiments of the present disclosure, by using the first mirror data set for pre-writing, it is possible to determine in advance whether the pre-writing simulation result of whether the write data can be written to the storage node before the write data is actually written to the storage node. And because the pre-writing operation is performed on the basis of the first mirror data set, there is no need to modify the storage node to support the XA protocol, and there is no need to perform operations such as locking on the original storage data set. The user can still access the original storage data set normally, reducing the impact on the storage node and the original storage data set.

[0056] In operation S220 , the pre-write simulation result is sent to the coordinator.

[0057] According to an embodiment of the present disclosure, since the coordinator is used to coordinate between multiple storage nodes, the coordinator can be used to count the pre-write simulation results of each of the multiple storage nodes, and further coordinate between the multiple storage nodes based on the pre-write simulation results to ensure consistency between the multiple storage nodes.

[0058] In operation S230 , a commit instruction sent by the coordinator is received, and the write data is written into the original storage data set.

[0059] According to an embodiment of the present disclosure, a commit instruction is generated by a coordinator when it is determined that a pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0060] According to an embodiment of the present disclosure, when the pre-write simulation results of multiple storage nodes are all written successfully, it means that the write data can be successfully written to the storage node. At this time, the coordinator can send commit instructions for committing the write data to the multiple storage nodes respectively.

[0061] According to an embodiment of the present disclosure, in response to receiving a commit instruction, the storage node confirms that the write data can be formally persisted and can write the write data into a set for storing the original storage data.

[0062] According to an embodiment of the present disclosure, a first mirror data set is obtained by copying the original data storage set, the first mirror data set is used to simulate the data storage status of the original data storage set in the storage node, and the first mirror data set is used to simulate writing the write data to the storage node, so as to obtain a pre-commit simulation effect that can characterize whether the write data can be successfully written to the storage node. The write data is formally written to the original storage set only when the pre-write results of multiple storage nodes all indicate that the pre-write is successful, so as to avoid the failure of writing data due to the abnormality of any storage node, resulting in data inconsistency between multiple storage nodes, and ensure the data consistency between multiple storage nodes while avoiding the impact on the original storage data set.

[0063] According to an embodiment of the present disclosure, based on the write data and the first mirror data set, a simulation is performed to write the write data into a storage node to obtain a pre-write simulation result, including: writing the write data into the first mirror data set to obtain a second mirror data set; performing validity verification on the second mirror data set to obtain a validity verification result; in the case where it is determined that the validity verification result indicates that the second mirror data set has validity, determining that the pre-write simulation result indicates that the pre-write is successful; and in the case where it is determined that the validity verification result indicates that the second mirror data set does not have validity, determining that the pre-write simulation result indicates that the pre-write has failed.

[0064] According to an embodiment of the present disclosure, since the second mirror data set is the result of writing the write data into the first mirror data set, the second mirror data set can simulate the data storage state of writing the write data to the storage node. The process of verifying the validity of the second mirror data set can also simulate the process of verifying the validity of the original data storage set after writing the write data into the original data storage set. The validity of the second mirror data set can equally indicate that the original storage data set after writing the write data is valid, and the non-validity of the second mirror data set can equally indicate that the original storage data set after writing the write data is not valid.

[0065] According to an embodiment of the present disclosure, when verifying the validity of the second mirror data, the validity of the written data itself can be verified, or the validity of the written data can be verified based on the first mirror data set in the second mirror data set, so as to avoid the second mirror data set being invalid due to a conflict between the written data and the first mirror data set.

[0066] According to an embodiment of the present disclosure, when it is determined that the second mirror data set is valid, it means that the write data can be written to the storage node normally, and the pre-write simulation result is determined to be a pre-write success; when it is determined that the second mirror data set is not valid, it means that the write data cannot be written to the storage node normally, and the pre-write simulation result is determined to be a pre-write failure.

[0067] According to an embodiment of the present disclosure, by verifying the validity of the second mirror data set, the validity verification of the original storage data set after writing the write data is simulated. Before the write data is officially written to the storage node, it is determined whether the write data will cause storage node write abnormalities, thereby ensuring the consistency of data between multiple storage nodes.

[0068] According to an embodiment of the present disclosure, write data is obtained through the following operations: in response to receiving the original write data sent by the client, the coordinator processes the original write data based on the database types of the storage node and at least one other storage node different from the storage node to obtain write data corresponding to each database type.

[0069] According to an embodiment of the present disclosure, the coordinator may obtain original write data from a write request sent by a client.

[0070] According to an embodiment of the present disclosure, since multiple storage nodes each correspond to a different database type and have different data storage formats, it is necessary to process the original write data sent by the client and convert it into write data corresponding to the database type. Specifically, a coordinator can be used to process the original write data.

[0071] According to an embodiment of the present disclosure, a coordinator is used to process the original write data to obtain write data corresponding to the database type, thereby achieving decoupling between multiple storage nodes and clients and being able to uniformly coordinate the data storage process across storage nodes.

[0072] Figure 3 The flowchart of the pre-submission stage according to an embodiment of the present disclosure is schematically shown.

[0073] like Figure 3 As shown, the pre-submission stage includes operations S310 to S340.

[0074] In operation S310 , the client sends a write request to the coordinator.

[0075] In operation S320 , the coordinator sends write data to the first storage node and the second storage node, respectively.

[0076] According to an embodiment of the present disclosure, the write data is obtained by the coordinator performing data processing on the original write data in the write request.

[0077] In operation S330 , the first storage node and the second storage node perform validity verification respectively.

[0078] According to an embodiment of the present disclosure, the first storage node and the second storage node may respectively write write data into their respective first mirror data, simulate writing the write data into the storage node, obtain second mirror data, and verify the validity of the second mirror data.

[0079] In operation S340, the first storage node and the second storage node respectively return a pre-write simulation result indicating that the pre-write is successful to the coordinator.

[0080] According to an embodiment of the present disclosure, the first storage node and the second storage node determine that the pre-write is successful when determining that their respective second mirror data are valid.

[0081] Figure 4 The flowchart of the submission phase according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 4 As shown, the submission phase includes operations S410 to S430.

[0083] In operation S410 , the coordinator sends a commit instruction to the first storage node and the second storage node respectively.

[0084] In operation S420, the first storage node and the second storage node respectively send a write result indicating that the write is successful to the coordinator.

[0085] According to an embodiment of the present disclosure, in response to receiving a commit instruction, the first storage node and the second storage node write the write data into their respective original storage data sets.

[0086] According to an embodiment of the present disclosure, both the first storage node and the second storage node successfully write the write data into their respective original storage data sets, and send a write result indicating the success of the write to the coordinator.

[0087] In operation S430 , the coordinator sends a submission result indicating a successful submission to the client.

[0088] According to an embodiment of the present disclosure, when the coordinator determines that multiple write results all represent successful write, it returns a submission result representing successful submission to the client.

[0089] According to an embodiment of the present disclosure, in response to receiving a cancel commit instruction sent by the coordinator, the second mirror data set is deleted.

[0090] According to an embodiment of the present disclosure, the uncommit instruction is generated by the coordinator when it is determined that any one of the pre-write simulation result and at least one other pre-write simulation result indicates a pre-write failure.

[0091] According to an embodiment of the present disclosure, when the pre-write simulation result indicates that the pre-write has failed, it indicates that the storage node to which the write data can be written cannot be written normally, and therefore, in order to ensure data consistency, it is necessary to abandon the submission of the write data. When the coordinator determines that any pre-write simulation result indicates that the pre-write has failed, it generates a cancel submission instruction, that is, the write data is not written to the original storage data set.

[0092] According to an embodiment of the present disclosure, since the pre-write process writes the write data into the first mirror data set and does not affect the original storage data set, there is no need to operate the original storage data set. It is only necessary to delete the second mirror data set stored in the storage node where the pre-write is successful.

[0093] According to an embodiment of the present disclosure, in the case where there is a storage node that fails in pre-writing, the submission of write data is canceled to ensure data consistency among multiple storage nodes.

[0094] Figure 5 The flowchart of the pre-submission stage according to another embodiment of the present disclosure is schematically shown.

[0095] like Figure 5 The pre-submission stage includes operations S510 to S560.

[0096] In operation S510 , the client sends a write request to the coordinator.

[0097] In operation S520 , the coordinator sends write data to the first storage node and the second storage node, respectively.

[0098] According to an embodiment of the present disclosure, the write data is obtained by the coordinator performing data processing on the original write data in the write request.

[0099] In operation S530 , the first storage node and the second storage node perform validity verification respectively.

[0100] According to an embodiment of the present disclosure, the first storage node and the second storage node may write write data into their respective first mirror data, simulate writing the write data into the storage node, obtain their respective second mirror data, and verify the validity of the second mirror data.

[0101] In operation S540, the first storage node returns a pre-write simulation result indicating a pre-write failure to the coordinator, and the second storage node returns a pre-write simulation result indicating a pre-write success to the coordinator.

[0102] According to an embodiment of the present disclosure, the first storage node determines that the second mirror data is not valid and determines that the pre-write fails; the second storage node determines that the second mirror data is valid and determines that the pre-write succeeds.

[0103] In operation S550, the coordinator sends a cancel commit instruction to the first storage node and the second storage node respectively.

[0104] According to an embodiment of the present disclosure, the coordinator determines that a plurality of pre-write simulation results include a pre-write simulation result indicating a pre-write failure, and generates and sends a cancel submission instruction.

[0105] In operation S560 , the coordinator sends a submission result indicating a submission failure to the client.

[0106] According to an embodiment of the present disclosure, in order to avoid the storage node failing to return the pre-write simulation result to the coordinator in time due to the storage node not receiving the write data, the coordinator can also send the write data to the storage node again if the pre-write simulation result is not received within a preset time to avoid blocking.

[0107] According to an embodiment of the present disclosure, the second mirror data set is validated to obtain a validity verification result, including: performing the same preprocessing on the written data in the second mirror data set and the first mirror data set, respectively, and then performing validity verification to obtain a first validity verification result; and obtaining a validity verification result based on the first validity verification result and the second validity verification result.

[0108] According to an embodiment of the present disclosure, in order to avoid conflicts between written data and the first mirror data set, it is necessary to match the written data based on the first mirror data set. However, since the storage node can be implemented based on a non-relational database, there is a problem of inconsistency in the form of the written data and the multiple first mirror data in the first mirror data set, and it is necessary to pre-process the written data and the multiple first mirror data in the first mirror data set for subsequent matching operations.

[0109] According to an embodiment of the present disclosure, the same preprocessing includes at least one of the following: semantic extraction and node data identification. Semantic extraction can be used to extract semantic features of the written data and the plurality of first mirror data in the first mirror data set, respectively. Specifically, entity objects involved in the written data and the first mirror data can be extracted as semantic features.

[0110] According to an embodiment of the present disclosure, the first validity verification result can be used to characterize whether the written data conflicts with the first mirror data set. Specifically, feature matching can be performed based on the semantic features of the written data and the semantic features of each of the multiple first mirror data to obtain multiple target first mirror data that match the semantic features of the written data, and determine whether the written data conflicts with the target first mirror data. Since the written data and the target first mirror data have the same or similar semantic features, such as being related to the same physical object, the target first mirror data can be used to determine whether the written data conflicts with the first mirror data set, and then determine whether the written data conflicts with the original storage data set.

[0111] According to an embodiment of the present disclosure, when there is a conflict between the written data and the first mirror data set, the first validity verification result is determined to represent that the written data is not valid; when there is no conflict between the written data and the first mirror data set, the first validity verification result is determined to represent that the written data is valid.

[0112] According to an embodiment of the present disclosure, when the storage node is implemented based on a graph database, the first mirror data set includes multiple node data, each node data represents an entity object, so the preprocessing of the written data and the first mirror data set can also be implemented through node data identification. Specifically, the node data in the written data and the node data involved in each first mirror data in the first mirror data set can be identified.

[0113] According to an embodiment of the present disclosure, for node data in write data, it can be determined whether node data representing the same entity object as the write data already exists in the first mirror data set. If node data representing the same entity object as the write data already exists in the first mirror data set, it is determined that node data used to represent the entity object already exists in the first mirror data set, then there is a conflict between the write data and the first mirror data set, and the pre-write simulation result is a write failure.

[0114] According to an embodiment of the present disclosure, for edge data in the write data, two entity objects corresponding to the edge data can be determined, and then it can be determined whether edge data of the association relationship between the two entity objects represented by the write data already exists in the first mirror data set. In the case that edge data of the association relationship between the two entity objects represented by the write data already exists in the first mirror data set, it is determined that edge data of the association relationship between the two entity objects already exists in the first mirror data set, then there is a conflict between the write data and the first mirror data set, and the pre-write simulation result is a pre-write failure.

[0115] According to the embodiments of the present disclosure, the second validity verification result is obtained based on data logic verification of the written data. Specifically, it can be determined whether there is a situation where necessary information in the written data is empty, etc. For example, in the operation and maintenance scenario, if there is no entity operation and maintenance object information in the written data, it is impossible to determine the entity operation and maintenance object targeted by the written data, and the written data is invalid.

[0116] According to an embodiment of the present disclosure, before writing the write data into the first mirror data, the validity of the write data may be checked. Specifically, it may be verified whether the write data meets a preset range or a predetermined data format.

[0117] According to the embodiments of the present disclosure, it is not necessary to store the verification rules for verifying the validity of the second mirror data set and the verification code for implementing the verification process in the storage node, but the storage node can be expanded in a pluggable form. Specifically, in the operation and maintenance scenario, the verification rules and verification codes can be stored in the operation and maintenance server, and the storage node verifies the second mirror data set by calling the verification rules and verification codes stored in the operation and maintenance server through an interface. At the same time, since the verification rules and verification codes are stored in the operation and maintenance server, the operation and maintenance personnel can customize the verification rules and verification codes according to business needs, which is convenient for the operation and maintenance personnel to adjust the verification rules and verification codes, and has strong flexibility.

[0118] According to an embodiment of the present disclosure, when any one of the first validity verification result and the second validity verification result indicates that the second mirror data set is not valid, it is determined that the validity verification result indicates that the second mirror data set is not valid.

[0119] According to an embodiment of the present disclosure, by determining whether there is a conflict between the written data and the first mirror data set and performing data logic verification on the written data, the validity of the written data is fully verified, thereby further ensuring data consistency.

[0120] According to the embodiments of the present disclosure, in the operation and maintenance scenario, it is usually necessary to use a graph database and a search engine at the same time to store and retrieve the relational data and document data of the entity operation and maintenance object. The graph database is good at expressing and querying complex entity association relationships, while the document database can efficiently implement the indexing and searching of full-text content. Combining the two can more completely describe and manage the operation and maintenance objects, and provide accurate and timely data support for operation and maintenance decisions.

[0121] According to an embodiment of the present disclosure, the database type includes a graph database type, and data processing is performed on the original write data to obtain write data corresponding to each database type, including: generating node data corresponding to each entity based on at least one entity involved in the original write data; obtaining edge data for characterizing the association relationship based on the original write data; and generating write data corresponding to the graph database type based on the node data and the edge data.

[0122] According to an embodiment of the present disclosure, the coordinator can obtain the write request sent by the client. When generating write data, the coordinator can also generate a write identifier corresponding to the write request. When sending the write data to the storage node, the write identifier can also be sent to the storage node to facilitate the storage node to distinguish the write data under different write requests.

[0123] According to an embodiment of the present disclosure, in the process of generating write data, because the data types corresponding to the graph database include node data and edge data, wherein the node data is used to characterize entities and the edge data is used to characterize the association relationship between entities, it is possible to extract information corresponding to the entity from the original write data to generate node data, and extract the association relationship between entities from the write data based on the information corresponding to the entity to obtain edge data.

[0124] According to an embodiment of the present disclosure, in an operation and maintenance scenario, the entity may be an entity operation and maintenance object, that is, a network device such as a server, a router, etc. The node data may be used to characterize the information of each entity operation and maintenance object, and the edge data may be used to characterize the communication link information between two entity operation and maintenance objects having a communication relationship.

[0125] According to an embodiment of the present disclosure, the database type includes a document database type, and data processing is performed on the original write data to obtain write data corresponding to each database type, including: determining a classification field for identifying the original write data based on attribute information of the original write data; and generating write data corresponding to the document database type based on the original write data and the classification field.

[0126] According to an embodiment of the present disclosure, since data in a document database is stored in the form of key-value pairs, that is, key-value pairs consisting of classification fields and field values, but the original write data sent by the client may not include the corresponding classification field, the coordinator can determine the classification field corresponding to the original write data from multiple preset classification fields according to the attribute information of the original write data as the classification field of the original write data, wherein the attribute information can be obtained by feature extraction of the write data.

[0127] According to an embodiment of the present disclosure, when the original write data is used to characterize the addition of a new entity operation and maintenance object, for the graph database type, node data for characterizing the entity operation and maintenance object can be created based on the information related to the new entity operation and maintenance object in the original write data, and edge data can be generated based on the entity operation and maintenance objects in the original write data that are associated with the new entity operation and maintenance object; for the document database type, multiple classification fields corresponding to the original write data can be determined, such as device type, device identification, etc.

[0128] According to the embodiments of the present disclosure, since the operation and maintenance data has the characteristics of dynamic partitioning and small scale, the storage node can also be divided into multiple partitions according to the device type of the physical operation and maintenance object. When sending write data to the storage node, the write sub-data can also be sent to the partition of the storage node in parallel according to the respective device types of multiple write sub-data in the write data, so as to improve data transmission efficiency.

[0129] According to an embodiment of the present disclosure, data consistency among multiple storage nodes is achieved by using a coordinator to convert original write data into write data corresponding to a database type.

[0130] According to an embodiment of the present disclosure, the data writing method also includes: determining the writing result of writing the write data to the original storage data set, and sending the writing result to the coordinator; and in response to receiving a rollback instruction sent by the coordinator, using the first mirror data set to roll back the original storage data set.

[0131] According to an embodiment of the present disclosure, a rollback instruction is generated by the coordinator when it is determined that any one of a write result and at least one other write result represents a write failure.

[0132] According to an embodiment of the present disclosure, even if the pre-write simulation result indicates that the write data can be written to the original storage data set, the storage node may fail to write the write data to the original storage data set due to network reasons, thereby causing data inconsistency among multiple storage nodes. Therefore, after writing the write data to the original data set, the storage node should also determine the write result and return the write result to the coordinator.

[0133] According to an embodiment of the present disclosure, when any write result indicates a write failure, it indicates that there is a storage node that has not written the write data to the original storage data set. In order to ensure data consistency between multiple storage nodes, the original storage data set in all storage nodes needs to be rolled back.

[0134] According to an embodiment of the present disclosure, since the first mirror data set has the same data storage state as the original storage data set before writing the write data, the first mirror data set can be used to restore the original storage data set to the data storage state before writing the write data. Specifically, the first mirror data set can be copied to the disk through the connection between the local cache and the disk.

[0135] According to an embodiment of the present disclosure, the original storage data set is rolled back using the first mirror data set to ensure data consistency among multiple storage nodes.

[0136] Figure 6 The flowchart of the submission phase according to another embodiment of the present disclosure is schematically shown.

[0137] like Figure 6 As shown, the submission phase includes operations S610 to S640.

[0138] In operation S610, the coordinator sends a commit instruction to the first storage node and the second storage node respectively.

[0139] In operation S620, the first storage node sends a write result indicating a write failure to the coordinator, and the second storage node sends a write result indicating a write success to the coordinator.

[0140] According to an embodiment of the present disclosure, in response to receiving a commit instruction, the first storage node and the second storage node write the write data into their respective original storage data sets.

[0141] According to an embodiment of the present disclosure, the first storage node fails to successfully write the write data to the original storage data set, and sends a write result to the coordinator to indicate the write failure; the second storage node successfully writes the write data to the original storage data set, and sends a write result to the coordinator to indicate the write success.

[0142] In operation S630 , the coordinator sends a rollback instruction to the first storage node and the second storage node respectively.

[0143] According to an embodiment of the present disclosure, when the coordinator determines that there is a write result representing a write failure among multiple write results, the coordinator sends a rollback instruction to the first storage node and the second storage node. Specifically, the first storage node and the second storage node can use their respective first mirror data sets to roll back their respective original storage data sets.

[0144] In operation S640 , the coordinator sends a submission result indicating a submission failure to the client.

[0145] According to an embodiment of the present disclosure, when the coordinator determines that there is a write result indicating a write failure among multiple write results, the coordinator returns a commit result indicating a commit failure to the client.

[0146] Figure 7 The flowchart of a data writing method according to a specific embodiment of the present disclosure is schematically shown.

[0147] like Figure 7 As shown, the data writing method includes operations S701 to S712.

[0148] In operation S701 , the client sends a write request.

[0149] In operation S702 , the coordinator processes the original write data in the write request to obtain the write data of the graph database and the write data of the document database, and sends them to the graph database and the document database respectively.

[0150] In operation S703, the graph database is pre-written.

[0151] According to an embodiment of the present disclosure, the graph database performs a pre-write operation on the write data in response to receiving the write data.

[0152] According to an embodiment of the present disclosure, the write data can be written into the first mirror data set, and the write data can be simulated to be written into the graph database to obtain the second mirror data set, and the validity of the second mirror data set is verified to obtain the validity verification result. If the second mirror data set is valid, it is determined that the pre-write is successful; if the second mirror data is not valid, it is determined that the pre-write fails.

[0153] In operation S704, the document database is pre-written.

[0154] According to an embodiment of the present disclosure, in response to receiving the write data, the document database performs a pre-write operation on the write data. Specifically, the write data may be written into the first mirror data set to simulate writing the write data into the document database.

[0155] According to an embodiment of the present disclosure, the write data can be written into the first mirror data set, and the write data can be simulated to be written into the document database to obtain the second mirror data set, and the validity of the second mirror data set is verified to obtain the validity verification result. If the second mirror data set is valid, it is determined that the pre-write is successful; if the second mirror data is not valid, it is determined that the pre-write fails.

[0156] In operation S705 , it is determined whether all pre-writing is successful.

[0157] According to an embodiment of the present disclosure, a coordinator may be used to determine whether both the graph database and the document database are pre-written successfully. If at least one pre-write fails, operation S706 is performed, and if both pre-writes are successful, operations S707 and S708 are performed.

[0158] According to an embodiment of the present disclosure, when the coordinator determines that both the graph database and the document database are written successfully, a commit instruction is sent to the graph database and the document database.

[0159] In operation S706 , the coordinator determines to cancel the submission.

[0160] According to an embodiment of the present disclosure, if there is a pre-write simulation result indicating a pre-write failure among the plurality of pre-write simulation results, the write data is not submitted in order to ensure data consistency.

[0161] In operation S707 , the graph database is written.

[0162] According to an embodiment of the present disclosure, in response to receiving a commit instruction, the graph database writes the write data into the original storage data set and sends the write result to the coordinator.

[0163] In operation S708, the document database is written.

[0164] According to an embodiment of the present disclosure, in response to receiving a commit instruction, the document database writes the write data into the original storage data set and sends the write result to the coordinator.

[0165] In operation S709 , it is determined whether all the data are written successfully.

[0166] According to an embodiment of the present disclosure, a coordinator may be used to determine whether both the graph database and the document database are written successfully. If both are determined to be written successfully, operation S710 is performed. If at least one write fails, operations S711 and S712 are performed.

[0167] According to an embodiment of the present disclosure, when the coordinator determines that at least one of the graph database and the document database fails to be written, a rollback instruction is sent to the graph database and the document database.

[0168] In operation S710 , the coordinator determines that writing is successful.

[0169] In operation S711, the graph database is rolled back.

[0170] According to an embodiment of the present disclosure, in response to receiving a rollback instruction, the graph database performs a rollback using the first mirror data set.

[0171] In operation S712, the document database is rolled back.

[0172] According to an embodiment of the present disclosure, in response to receiving a rollback instruction, the document database performs a rollback using the first mirror data set.

[0173] Based on the above data writing method, the present disclosure also provides a data writing device. Figure 8 The device is described in detail.

[0174] Figure 8 The structural block diagram of a data writing device according to an embodiment of the present disclosure is schematically shown.

[0175] like Figure 8 As shown, the data writing device 800 of this embodiment includes a first writing module 810 , a sending module 820 and a second writing module 830 .

[0176] The first writing module 810 is used to simulate writing the write data to the storage node in response to receiving the write data sent by the coordinator, based on the write data and the first mirror data set, to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node. In one embodiment, the first writing module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0177] The sending module 820 is used to send the pre-written simulation result to the coordinator. In one embodiment, the sending module 820 can be used to perform the operation S220 described above, which will not be described in detail here.

[0178] The second writing module 830 is used to receive the commit instruction sent by the coordinator, and write the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node. In one embodiment, the second writing module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0179] According to an embodiment of the present disclosure, the first writing module includes a writing submodule, a verification submodule, a first determination submodule and a second determination submodule.

[0180] The writing submodule is used to write the write data into the first mirror data set to obtain the second mirror data set.

[0181] The verification submodule is used to perform validity verification on the second mirror data set to obtain a validity verification result.

[0182] The first determining submodule is used to determine that the pre-writing simulation result indicates that the pre-writing is successful when it is determined that the validity verification result indicates that the second mirror data set is valid.

[0183] The second determining submodule is used to determine that the pre-writing simulation result indicates that the pre-writing fails when it is determined that the validity verification result indicates that the second mirror data set is not valid.

[0184] According to an embodiment of the present disclosure, the data writing device 800 further includes a submission cancellation module.

[0185] The unsubmit module is used to delete the second mirror data set in response to receiving a unsubmit instruction sent by the coordinator, wherein the unsubmit instruction is generated by the coordinator when it is determined that any pre-write simulation result among the pre-write simulation result and at least one other pre-write simulation result indicates a pre-write failure.

[0186] According to an embodiment of the present disclosure, the verification submodule includes a first verification unit and a second verification unit.

[0187] The first verification unit is used to perform the same preprocessing on the written data in the second mirror data set and the first mirror data set, and then perform validity verification to obtain a first validity verification result, wherein the same preprocessing includes at least one of the following: semantic extraction and node data identification.

[0188] The second verification unit is used to obtain a validity verification result based on the first validity verification result and the second validity verification result, wherein the second validity verification result is obtained based on data logic verification of the written data.

[0189] According to an embodiment of the present disclosure, write data is obtained through the following operations: in response to receiving the original write data sent by the client, the coordinator processes the original write data based on the database types of the storage node and at least one other storage node different from the storage node to obtain write data corresponding to each database type.

[0190] According to an embodiment of the present disclosure, the database type includes a graph database type, and data processing is performed on the original write data to obtain write data corresponding to each database type, including: based on at least one entity involved in the original write data, generating node data corresponding to each entity; based on the original write data, obtaining edge data for characterizing the association relationship; and based on the node data and the edge data, generating write data corresponding to the graph database type; the database type includes a document database type, and data processing is performed on the original write data to obtain write data corresponding to each database type, including: based on the attribute information of the original write data, determining a classification field for identifying the original write data; and generating write data corresponding to the document database type based on the original write data and the classification field.

[0191] According to an embodiment of the present disclosure, the data writing device 800 further includes a result determination module and a rollback module.

[0192] The result determination module is used to determine the writing result of writing the write data into the original storage data set, and send the writing result to the coordinator.

[0193] The rollback module is used to roll back the original storage data set using the first mirror data set in response to receiving a rollback instruction sent by the coordinator, wherein the rollback instruction is generated by the coordinator when it is determined that any one of the write result and at least one other write result represents a write failure.

[0194] According to an embodiment of the present disclosure, any multiple modules of the first writing module 810, the sending module 820, and the second writing module 830 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first writing module 810, the sending module 820, and the second writing module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first writing module 810, the sending module 820, and the second writing module 830 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0195] Fig. 9 A block diagram of an electronic device suitable for implementing a data writing method according to an embodiment of the present disclosure is schematically shown.

[0196] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0197] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0198] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0199] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0200] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0201] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data writing method provided by the embodiment of the present disclosure.

[0202] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0203] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0204] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0205] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0206] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0207] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0208] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data writing method, characterized in that: Applied to a storage node, the method comprises: In response to receiving the write data sent by the coordinator, based on the write data and the first mirror data set, simulating writing the write data into the storage node to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; sending the pre-written simulation results to the coordinator; and Receive a commit instruction sent by the coordinator, and write the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node that is different from the storage node.

2. The method according to claim 1, characterized in that The step of simulating writing the write data into the storage node based on the write data and the first mirror data set to obtain a pre-write simulation result includes: Writing the write data into the first mirror data set to obtain a second mirror data set; Performing validity verification on the second mirror data set to obtain a validity verification result; In a case where it is determined that the validity verification result indicates that the second mirror data set is valid, determining that the pre-writing simulation result indicates that the pre-writing is successful; and In a case where it is determined that the validity verification result indicates that the second mirror data set is not valid, it is determined that the pre-writing simulation result indicates that the pre-writing fails.

3. The method according to claim 2, characterized in that The method further comprises: In response to receiving a cancel commit instruction sent by the coordinator, deleting the second mirror data set, wherein the cancel commit instruction is generated by the coordinator when determining that any one of the pre-write simulation result and at least one other pre-write simulation result indicates a pre-write failure.

4. The method according to claim 2, characterized in that: The performing validity verification on the second mirror data set to obtain a validity verification result includes: After performing the same preprocessing on the written data in the second mirror data set and the first mirror data set, respectively, a validity verification is performed to obtain a first validity verification result, wherein the same preprocessing includes at least one of the following: semantic extraction, node data identification; and The validity verification result is obtained based on the first validity verification result and the second validity verification result, wherein the second validity verification result is obtained based on performing data logic verification on the write data.

5. The method according to claim 1, characterized in that The written data is obtained by the following operations: In response to receiving the original write data sent by the client, the coordinator processes the original write data based on the database types of the storage node and at least one other storage node different from the storage node to obtain write data corresponding to each of the database types.

6. The method according to claim 5, characterized in that The database type includes a graph database type, and the data processing of the original written data to obtain the written data corresponding to each of the database types includes: Based on at least one entity involved in the original written data, generating node data corresponding to each of the entities; Based on the original written data, acquiring edge data for representing the association relationship; and Based on the node data and the edge data, generate write data corresponding to the graph database type; The database type includes a document database type, and the processing of the original written data to obtain the written data corresponding to each of the database types includes: Based on the attribute information of the original written data, determining a classification field for identifying the original written data; and Based on the original written data and the classification field, written data corresponding to the document database type is generated.

7. The method according to claim 1, characterized in that The method further comprises: determining a write result of writing the write data to the original stored data set, and sending the write result to the coordinator; and In response to receiving a rollback instruction sent by the coordinator, the original storage data set is rolled back using the first mirror data set, wherein the rollback instruction is generated by the coordinator when it is determined that any one of the write result and at least one other write result represents a write failure.

8. A data writing device, characterized in that: The device comprises: A first writing module is used for, in response to receiving the write data sent by the coordinator, simulating writing the write data into the storage node based on the write data and a first mirror data set to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; a sending module, configured to send the pre-written simulation result to the coordinator; and A second write module is used to receive a commit instruction sent by the coordinator, and write the write data into the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both indicate that the pre-write is successful, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images