Data writing method, device, apparatus, medium and program product

Abstract

This disclosure provides a data writing method applicable to the field of big data technology. The data writing method includes: responding to receiving write data sent by a coordinator, simulating writing the write data to a storage node based on the write data and a first mirror data set to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; sending the pre-write simulation result to the coordinator; receiving a commit instruction sent by the coordinator, and writing the write data to the original storage data set, wherein the commit instruction is generated by the coordinator after determining that the pre-write simulation result and at least one other pre-write simulation result both characterize successful pre-write, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node. This disclosure also provides a data writing apparatus, device, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and more specifically to a data writing method, apparatus, device, medium, and program product. Background Technology

[0002] With the advent of the big data era, there is a need for diverse data formats to meet complex data usage requirements. Since a single database can typically only store one type of data, multiple storage nodes based on different types of databases are needed to store various data formats.

[0003] However, when writing data to multiple storage nodes simultaneously, if an exception causes one or more of the storage nodes to fail to write data, data consistency between the multiple storage nodes cannot be guaranteed. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a data writing method, apparatus, device, medium and program product.

[0005] According to a first aspect of this disclosure, a data writing method is provided, comprising: responding to receiving write data sent by a coordinator, simulating writing the write data to the storage node based on the write data and a first mirror data set to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying an original storage data set stored in the storage node; sending the pre-write simulation result to the coordinator; and receiving a commit instruction sent by the coordinator to write the write data to the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both characterize a successful pre-write, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0006] According to embodiments of this disclosure, the above-mentioned simulation of writing the written data to the storage node based on the written data and the first mirror data set to obtain a pre-write simulation result includes: writing the written data into the first mirror data set to obtain a second mirror data set; performing validity verification on the second mirror data set to obtain a validity verification result; if the validity verification result indicates that the second mirror data set is valid, determining that the pre-write simulation result indicates that the pre-write is successful; and if the validity verification result indicates that the second mirror data set is not valid, determining that the pre-write simulation result indicates that the pre-write is unsuccessful.

[0007] According to an embodiment of this disclosure, the method further includes: deleting the second mirror data set in response to receiving a cancel commit instruction sent by the coordinator, wherein the cancel commit instruction is generated by the coordinator when it is determined that any one of the pre-write simulation results and at least one other pre-write simulation result represents a pre-write failure.

[0008] According to embodiments of this disclosure, the above-mentioned validity verification of the second mirror data set to obtain a validity verification result includes: performing the same preprocessing on the written data in the second mirror data set and the first mirror data set respectively, and then performing validity verification to obtain a first validity verification result, wherein the same preprocessing includes at least one of the following: semantic extraction, node data identification; and obtaining the above-mentioned validity verification result based on the first validity verification result and the second validity verification result, wherein the second validity verification result is obtained based on data logic verification of the written data.

[0009] According to an embodiment of this disclosure, the written data is obtained through the following operation: in response to receiving the original written data sent by the client, the coordinator performs data processing on the original written data based on the respective database types of the storage node and at least one other storage node different from the storage node, to obtain written data corresponding to each of the aforementioned database types.

[0010] According to embodiments of this disclosure, the database type includes a graph database type. The above-mentioned data processing of the original written data to obtain written data corresponding to each of the above database types includes: generating node data corresponding to each of the above entities based on at least one entity involved in the original written data; obtaining edge data for representing the association relationship based on the original written data; and generating written data corresponding to the graph database type based on the node data and the edge data.

[0011] According to embodiments of this disclosure, the database type includes a document database type. The above-mentioned data processing of the original written data to obtain written data corresponding to each of the above database types includes: determining a classification field for identifying the original written data based on the attribute information of the original written data; and generating written data corresponding to the document database type based on the original written data and the classification field.

[0012] According to an embodiment of this disclosure, the method further includes: determining a write result of writing the written data to the original storage data set, and sending the write result to the coordinator; and in response to receiving a rollback instruction sent by the coordinator, rolling back the original storage data set using the first mirror data set, wherein the rollback instruction is generated by the coordinator when it is determined that either the write result or at least one other write result represents a write failure.

[0013] A second aspect of this disclosure provides a data writing apparatus, comprising: a first writing module, configured to, in response to receiving write data sent by a coordinator, simulate writing the write data to a storage node based on the write data and a first mirror data set to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying an original storage data set stored in the storage node; a sending module, configured to send the pre-write simulation result to the coordinator; and a second writing module, configured to receive a commit instruction sent by the coordinator and write the write data to the original storage data set, wherein the commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both represent successful pre-write, and the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0014] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this disclosure, a first mirror data set is obtained by copying the original data storage set. The first mirror data set is used to simulate the data storage state of the original data storage set in the storage node. The first mirror data set is also used to simulate writing data to the storage node, thus obtaining a pre-commit simulation effect that can characterize whether the data can be successfully written to the storage node. Only when the pre-write results of multiple storage nodes all indicate that the pre-write is successful will the data be officially written to the original storage set. This avoids data writing failure due to an anomaly of any storage node, which would lead to data inconsistency between multiple storage nodes. This ensures data consistency between multiple storage nodes while avoiding impact on the original storage data set. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration schematically depicts application scenarios of data writing methods, apparatus, devices, media, and program products according to embodiments of the present disclosure;

[0020] Figure 2 A flowchart illustrating a data writing method according to an embodiment of the present disclosure is shown schematically;

[0021] Figure 3 A flowchart illustrating the pre-submission phase according to an embodiment of this disclosure is shown schematically;

[0022] Figure 4 A flowchart illustrating the submission phase according to an embodiment of this disclosure is shown schematically;

[0023] Figure 5 A flowchart illustrating the pre-submission phase according to another embodiment of this disclosure is shown schematically;

[0024] Figure 6 A flowchart illustrating the submission phase according to another embodiment of this disclosure is shown schematically;

[0025] Figure 7 A flowchart illustrating a data writing method according to a specific embodiment of the present disclosure is shown schematically.

[0026] Figure 8 A schematic block diagram of a data writing apparatus according to an embodiment of the present disclosure is shown; and

[0027] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data writing method according to an embodiment of the present disclosure. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, they do not mean that the applicant has used or necessarily used such solutions.

[0033] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0035] When writing data to multiple storage nodes simultaneously, if one or more of the storage nodes fail to write data due to an anomaly, the data consistency between the multiple storage nodes cannot be guaranteed, which in turn affects the accuracy and reliability of the data.

[0036] To ensure data consistency, a common approach is to use message queue-based coordination, a two-phase commit protocol, and delayed synchronization. Message queue-based coordination requires a message queue as a coordinator, placing all write requests into the queue. The consumer retrieves the operation request from the queue and then uses a transactional consumption model to write data sequentially to multiple storage nodes. However, this approach divides transaction processing into two phases, introducing additional time overhead and throughput bottlenecks. Furthermore, it requires the consumer process to run continuously, which can lead to data loss or duplicate writes in case of failure.

[0037] The two-phase commit protocol, by introducing a transaction coordinator, unifies the scheduling of write operations across multiple storage units, achieving strict transaction consistency. However, because most NoSQL databases lack support for the distributed transaction processing interface specification protocol (XA protocol), they cannot implement the two-phase commit protocol, requiring modifications to the kernel of the NoSQL database storage nodes. Therefore, implementing the two-phase commit protocol in NoSQL databases is costly. Furthermore, even if the storage node supports the XA protocol, the two-phase commit protocol requires locking the original stored data set in the storage node, leading to congestion and low read / write efficiency.

[0038] Delayed synchronization requires writing to be completed on one of the storage nodes first, then committed, and asynchronously replicated to other storage nodes. While this approach avoids direct coordination among multiple storage nodes, it suffers from eventual consistency issues. Even with incremental synchronization, data inconsistencies within the synchronization time window cannot be avoided, failing to meet the demands of scenarios requiring high data real-time performance and consistency.

[0039] This disclosure provides a data writing method. In response to receiving write data from a coordinator, the method simulates writing the write data to a storage node based on the write data and a first mirrored data set, obtaining a pre-write simulation result. The first mirrored data set is obtained by copying the original storage data set stored in the storage node. The pre-write simulation result is sent to the coordinator. The method also receives a commit instruction from the coordinator and writes the write data to the original storage data set. The commit instruction is generated by the coordinator when it determines that the pre-write simulation result and at least one other pre-write simulation result both indicate successful pre-write. The at least one other pre-write simulation result comes from at least one other storage node different from the original storage node. This embodiment of the application simulates writing the write data to the storage node using a first mirrored data set obtained by copying the original storage data set. Only when the pre-write is confirmed to be successful is the write data formally written to the storage node. This not only ensures real-time data consistency among multiple storage nodes but also guarantees the performance of the storage nodes.

[0040] Figure 1 The illustration schematically depicts application scenarios of data writing methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0041] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a coordinator 105, a first storage node 106, and a second storage node 107. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the coordinator 105, and between the coordinator 105 and the first storage node 106 and the second storage node 107. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0042] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the coordinator 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0044] The coordinator 105 can be used to forward data processing requests issued by the first terminal device 101, the second terminal device 102, or the third terminal device 103 to the first storage node 106 or the second storage node 107, coordinate the first storage node 106 and the second storage node 107, and feed back the processing results of the first storage node 106 and the second storage node 107 (such as web pages, information, or data obtained or generated according to user requests) to the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0045] The first storage node 106 and the second storage node 107 are used to store data, process data processing requests sent by the coordinator 105, and return the storage results to the coordinator 105.

[0046] It should be noted that the data writing method provided in this embodiment can generally be executed by the first storage node 106 and the second storage node 107. Correspondingly, the data writing device provided in this embodiment can generally be located in the first storage node 106 and the second storage node 107. The data writing method provided in this embodiment can also be executed by a database or database cluster that is different from the first storage node 106 and the second storage node 107 but can communicate with the coordinator 105. Correspondingly, the data writing device provided in this embodiment can also be located in a database or database cluster that is different from the first storage node 106 and the second storage node 107 but can communicate with the coordinator 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, coordinators, and storage nodes shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, coordinators, and storage nodes can be included.

[0048] The following will be based on Figure 1 The described scene, through Figures 2-7 The data writing method of the disclosed embodiments will be described in detail.

[0049] Figure 2 A flowchart illustrating a data writing method according to an embodiment of the present disclosure is shown schematically.

[0050] like Figure 2 As shown, the data writing method of this embodiment includes operations S210 to S230.

[0051] In operation S210, in response to receiving write data sent by the coordinator, based on the write data and the first mirror data set, the write data is simulated to be written to the storage node to obtain the pre-write simulation result.

[0052] According to embodiments of this disclosure, the storage node can be any storage node in a heterogeneous database cluster. The heterogeneous database cluster can be implemented based on various types of databases, specifically including relational databases and non-relational databases such as graph databases and document databases.

[0053] According to embodiments of this disclosure, the first mirrored data set is obtained by copying the original stored data set stored in the storage node. Specifically, the original stored data set can be stored on the disk of the storage node, and the first mirrored data set can be stored in the local cache of the storage node. In the process of copying the original stored data set to obtain the first mirrored data set, an environment identical to the original stored data set can first be established in the local cache of the storage node. Then, a connection is established between the disk and the local cache, and the original stored data set is copied from the disk to the local cache through this connection to obtain the first mirrored data set.

[0054] According to embodiments of this disclosure, data is simulated to be written to a storage node based on a first mirrored data set. Since the first mirrored data set is obtained by copying the original storage data set, the first mirrored data set can simulate the data storage state of the original data storage set in the storage node.

[0055] According to embodiments of this disclosure, by using a first mirrored data set for pre-writing, the pre-write simulation result can be determined in advance whether the data can be written to the storage node before it is actually written. Furthermore, since the pre-write operations are all performed based on the first mirrored data set, there is no need to modify the storage node to support the XA protocol, nor is it necessary to perform operations such as locking on the original storage data set. Users can still access the original storage data set normally, reducing the impact on the storage node and the original storage data set.

[0056] In operation S220, the pre-written simulation results are sent to the coordinator.

[0057] According to embodiments of this disclosure, since the coordinator is used for coordination among multiple storage nodes, the coordinator can be used to statistically analyze the pre-write simulation results of each of the multiple storage nodes, and further coordination among the multiple storage nodes can be performed based on the pre-write simulation results to ensure consistency among the multiple storage nodes.

[0058] In operation S230, a commit instruction sent by the coordinator is received, and the data to be written is written to the original storage data set.

[0059] According to embodiments of this disclosure, the commit instruction is generated by the coordinator when it is determined that both the pre-write simulation result and at least one other pre-write simulation result characterize the pre-write as successful, wherein the at least one other pre-write simulation result comes from at least one other storage node different from the storage node.

[0060] According to embodiments of this disclosure, if the pre-write simulation results of multiple storage nodes are all successfully written, it indicates that the write data can be successfully written to the storage nodes. At this time, the coordinator can send a commit instruction to each of the multiple storage nodes to submit the write data.

[0061] According to embodiments of this disclosure, in response to receiving a commit instruction, the storage node confirms that the write data can be formally persisted and can be written to the original storage data set.

[0062] According to embodiments of this disclosure, a first mirror data set is obtained by copying the original data storage set. The first mirror data set is used to simulate the data storage state of the original data storage set in the storage node. The first mirror data set is also used to simulate writing data to the storage node, thus obtaining a pre-commit simulation effect that can characterize whether the data can be successfully written to the storage node. Only when the pre-write results of multiple storage nodes all indicate that the pre-write is successful will the data be officially written to the original storage set. This avoids data writing failure due to an anomaly of any storage node, which would lead to data inconsistency between multiple storage nodes. This ensures data consistency between multiple storage nodes while avoiding impact on the original storage data set.

[0063] According to embodiments of this disclosure, based on the written data and a first mirrored data set, a pre-write simulation result is obtained by simulating the writing of the written data to a storage node. This includes: writing the written data into the first mirrored data set to obtain a second mirrored data set; validating the second mirrored data set to obtain a validity verification result; if the validity verification result indicates that the second mirrored data set is valid, determining that the pre-write simulation result indicates that the pre-write was successful; and if the validity verification result indicates that the second mirrored data set is not valid, determining that the pre-write simulation result indicates that the pre-write failed.

[0064] According to embodiments of this disclosure, since the second mirrored data set is the result of writing data to the first mirrored data set, the second mirrored data set can simulate the data storage state of writing data to the storage node. The process of validating the second mirrored data set can also simulate the process of validating the original data storage set after writing data to it. The validity of the second mirrored data set is equivalent to the validity of the original storage data set after the data was written; conversely, the invalidity of the second mirrored data set is equivalent to the invalidity of the original storage data set after the data was written.

[0065] According to embodiments of this disclosure, when validating the second mirror data, the validity of the written data itself can be verified, or the validity of the written data can be verified based on the first mirror data set in the second mirror data set, so as to avoid the second mirror data set being invalid due to a conflict between the written data and the first mirror data set.

[0066] According to the embodiments of this disclosure, if it is determined that the second mirrored data set is valid, it indicates that the write data can be written normally to the storage node, and the pre-write simulation result is determined to be a successful pre-write; if it is determined that the second mirrored data set is invalid, it indicates that the write data cannot be written normally to the storage node, and the pre-write simulation result is determined to be a failed pre-write.

[0067] According to the embodiments of this disclosure, by validating the second mirror data set, the validity of the original storage data set after writing the data is simulated. Before the data is officially written to the storage node, it is determined whether the writing data will cause the storage node to write abnormally, thus ensuring the consistency of data between multiple storage nodes.

[0068] According to embodiments of this disclosure, the write data is obtained by the following operation: in response to receiving the raw write data sent by the client, the coordinator performs data processing on the raw write data based on the respective database types of the storage node and at least one other storage node different from the storage node, to obtain write data corresponding to each database type.

[0069] According to embodiments of this disclosure, the coordinator can obtain raw write data from write requests sent by clients.

[0070] According to embodiments of this disclosure, since multiple storage nodes correspond to different database types and have different data storage formats, it is necessary to process the raw write data sent by the client and convert it into write data corresponding to the database type. Specifically, the coordinator can be used to process the raw write data.

[0071] According to embodiments of this disclosure, a coordinator is used to process the original write data to obtain write data corresponding to the database type, thereby decoupling multiple storage nodes from the client and enabling unified coordination of data storage processes across storage nodes.

[0072] Figure 3 A flowchart illustrating the pre-submission phase according to an embodiment of this disclosure is shown schematically.

[0073] like Figure 3 As shown, the pre-submission phase includes operations S310 to S340.

[0074] When operating S310, the client sends a write request to the coordinator.

[0075] During operation S320, the coordinator sends write data to the first storage node and the second storage node respectively.

[0076] According to embodiments of this disclosure, the write data is obtained by the coordinator processing the original write data in the write request.

[0077] During operation of S330, the first storage node and the second storage node perform validity verification respectively.

[0078] According to embodiments of this disclosure, the first storage node and the second storage node can respectively write the write data into their respective first mirror data, simulate writing the write data to the storage node to obtain the second mirror data, and verify the validity of the second mirror data.

[0079] During operation S340, the first and second storage nodes respectively return pre-write simulation results indicating successful pre-write to the coordinator.

[0080] According to embodiments of this disclosure, the first storage node and the second storage node determine that the pre-write is successful if they determine that their respective second mirror data is valid.

[0081] Figure 4 A flowchart illustrating the submission phase according to an embodiment of this disclosure is shown schematically.

[0082] like Figure 4 As shown, the submission phase includes operations S410 to S430.

[0083] During operation S410, the coordinator sends commit commands to the first storage node and the second storage node, respectively.

[0084] During operation S420, the first storage node and the second storage node respectively send a write result indicating that the write was successful to the coordinator.

[0085] According to embodiments of this disclosure, the first storage node and the second storage node, in response to receiving a commit instruction, write the data to their respective original storage data sets.

[0086] According to embodiments of this disclosure, both the first storage node and the second storage node successfully write the data to their respective original storage data sets and send a write result to the coordinator to indicate that the write was successful.

[0087] In operation S430, the coordinator sends a commit result to the client indicating a successful commit.

[0088] According to embodiments of this disclosure, when the coordinator determines that multiple write results all indicate successful writes, it returns a commit result indicating successful commit to the client.

[0089] According to embodiments of this disclosure, in response to receiving a cancel commit instruction sent by the coordinator, the second mirror data set is deleted.

[0090] According to embodiments of this disclosure, the cancel commit instruction is generated by the coordinator upon determining that any one of the prewrite simulation results and at least one other prewrite simulation result characterizes a prewrite failure.

[0091] According to embodiments of this disclosure, if the pre-write simulation result indicates a pre-write failure, it means that the storage node cannot write the data normally. Therefore, to ensure data consistency, the commit of the write data needs to be abandoned. When the coordinator determines that any pre-write simulation result indicates a pre-write failure, it generates a cancel commit instruction, that is, it does not write the data to the original storage data set.

[0092] According to the embodiments of this disclosure, since the pre-write process writes the write data to the first mirror data set and does not affect the original storage data set, there is no need to operate on the original storage data set. It is only necessary to delete the second mirror data set stored in the storage node where the pre-write was successfully completed.

[0093] According to embodiments of this disclosure, in the event of a storage node failing to write data, the write data submission is cancelled to ensure data consistency among multiple storage nodes.

[0094] Figure 5 A flowchart illustrating the pre-submission phase according to another embodiment of this disclosure is shown.

[0095] like Figure 5 The pre-submission stage includes operations S510 to S560.

[0096] When operating S510, the client sends a write request to the coordinator.

[0097] During operation of S520, the coordinator sends write data to the first storage node and the second storage node respectively.

[0098] According to embodiments of this disclosure, the write data is obtained by the coordinator processing the original write data in the write request.

[0099] During the operation of S530, the first storage node and the second storage node respectively perform validity verification.

[0100] According to embodiments of this disclosure, the first storage node and the second storage node can write the write data into their respective first mirror data, simulate writing the write data into the storage node, obtain their respective second mirror data, and verify the validity of the second mirror data.

[0101] During operation S540, the first storage node returns a pre-write simulation result indicating a pre-write failure to the coordinator, and the second storage node returns a pre-write simulation result indicating a pre-write success to the coordinator.

[0102] According to an embodiment of this disclosure, the first storage node determines that the second image data is invalid and determines that the pre-write has failed; the second storage node determines that the second image data is valid and determines that the pre-write has succeeded.

[0103] During S550 operation, the coordinator sends cancellation commit commands to the first storage node and the second storage node, respectively.

[0104] According to embodiments of this disclosure, the coordinator determines that among a plurality of pre-write simulation results therein is a pre-write simulation result characterizing a pre-write failure, and generates and sends a cancel commit instruction.

[0105] When operating S560, the coordinator sends a commit result to the client indicating that the commit failed.

[0106] According to embodiments of this disclosure, in order to avoid the storage node not returning the pre-write simulation result to the coordinator in a timely manner due to the storage node not receiving the write data, the coordinator can also send the write data to the storage node again if the pre-write simulation result is not received within a preset time, so as to avoid blocking.

[0107] According to embodiments of this disclosure, validating the second mirrored data set to obtain a validity verification result includes: performing the same preprocessing on the data written to the second mirrored data set and the first mirrored data set respectively, then performing validity verification to obtain a first validity verification result; and obtaining a validity verification result based on the first validity verification result and the second validity verification result.

[0108] According to embodiments of this disclosure, to avoid conflicts between the written data and the first mirror data set, it is necessary to match the written data based on the first mirror data set. However, since the storage node can be implemented based on a non-relational database, there is a problem of inconsistent formats between the written data and multiple first mirror data in the first mirror data set. Therefore, it is necessary to preprocess the written data and multiple first mirror data in the first mirror data set for subsequent matching operations.

[0109] According to embodiments of this disclosure, the same preprocessing includes at least one of the following: semantic extraction and node data identification. Semantic extraction can be used to extract the semantic features of each of the written data and multiple first mirror data in the first mirror data set. Specifically, the entity objects involved in the written data and the first mirror data can be extracted as semantic features.

[0110] According to embodiments of this disclosure, the first validity verification result can be used to characterize whether the written data conflicts with the first mirror data set. Specifically, feature matching can be performed based on the semantic features of the written data and the semantic features of multiple first mirror data to obtain multiple target first mirror data that match the semantic features of the written data, and to determine whether the written data conflicts with the target first mirror data. Since the written data and the target first mirror data have the same or similar semantic features, such as both being related to the same entity object, the target first mirror data can be used to determine whether the written data conflicts with the first mirror data set, and thus determine whether the written data conflicts with the original stored data set.

[0111] According to embodiments of this disclosure, if there is a conflict between the written data and the first mirror data set, the first validity verification result indicates that the written data is not valid; if there is no conflict between the written data and the first mirror data set, the first validity verification result indicates that the written data is valid.

[0112] According to embodiments of this disclosure, when the storage node is implemented based on a graph database, the first mirror data set includes multiple node data, each node data representing an entity object. Therefore, preprocessing of the written data and the first mirror data set can also be achieved through node data identification. Specifically, the node data in the written data and the node data involved in each first mirror data in the first mirror data set can be identified.

[0113] According to the embodiments of this disclosure, for node data in the written data, it can be determined whether node data representing the same entity object as the written data already exists in the first mirror data set. If node data representing the same entity object as the written data already exists in the first mirror data set, it is determined that node data representing the entity object already exists in the first mirror data set. In this case, there is a conflict between the written data and the first mirror data set, and the pre-write simulation result is a write failure.

[0114] According to the embodiments of this disclosure, for edge data in the written data, two entity objects corresponding to the edge data can be identified, and then it can be determined whether edge data representing the association between the two entity objects represented by the written data already exists in the first mirror data set. If edge data representing the association between the two entity objects represented by the written data already exists in the first mirror data set, and it is determined that edge data representing the association between the two entity objects already exists in the first mirror data set, then there is a conflict between the written data and the first mirror data set, and the pre-write simulation result is pre-write failure.

[0115] According to embodiments of this disclosure, the second validity verification result is obtained based on data logic verification of the written data. Specifically, it can be determined whether there are cases where necessary information is empty in the written data. For example, in an operation and maintenance scenario, if there is no entity operation and maintenance object information in the written data, then the entity operation and maintenance object to which the written data is targeted cannot be determined, and the written data is invalid.

[0116] According to embodiments of this disclosure, the validity of the written data can be verified before writing the written data to the first image data. Specifically, it can be verified whether the written data meets a preset range or a predetermined data format.

[0117] According to embodiments of this disclosure, it is not necessary to store the verification rules and verification code used to validate the second mirrored data set in the storage node; instead, the storage node can be extended in a pluggable manner. Specifically, in an operation and maintenance scenario, the verification rules and verification code can be stored in the operation and maintenance server. The storage node uses an interface to call the verification rules and verification code stored in the operation and maintenance server to validate the second mirrored data set. Furthermore, since the verification rules and verification code are stored in the operation and maintenance server, operation and maintenance personnel can customize the verification rules and verification code according to business needs, facilitating adjustments and providing high flexibility.

[0118] According to embodiments of this disclosure, if either the first validity verification result or the second validity verification result indicates that the second mirror data set is not valid, then the validity verification result indicates that the second mirror data set is not valid.

[0119] According to embodiments of this disclosure, by determining whether there is a conflict between the written data and the first mirror data set and by performing data logic verification on the written data, the validity of the written data is fully verified, thereby further ensuring data consistency.

[0120] According to embodiments of this disclosure, in operation and maintenance scenarios, it is typically necessary to use both graph databases and search engines to store and retrieve relational data and document data of entity operation and maintenance objects. Graph databases excel at expressing and querying complex entity relationships, while document databases can efficiently index and search full-text content. Combining the two allows for a more complete description and management of operation and maintenance objects, providing accurate and timely data support for operation and maintenance decisions.

[0121] According to embodiments of this disclosure, the database type includes a graph database type. Data processing is performed on the original written data to obtain written data corresponding to each database type, including: generating node data corresponding to each entity based on at least one entity involved in the original written data; obtaining edge data for representing the association relationship based on the original written data; and generating written data corresponding to the graph database type based on the node data and the edge data.

[0122] According to embodiments of this disclosure, the coordinator can obtain the write request from the client. When generating write data, the coordinator can also generate a write identifier corresponding to the write request. When sending the write data to the storage node, the write identifier can also be sent to the storage node together, so that the storage node can distinguish the write data under different write requests.

[0123] According to embodiments of this disclosure, during the process of generating write data, since the data types corresponding to the graph database include node data and edge data, where node data is used to represent entities and edge data is used to represent the relationships between entities, information corresponding to entities can be extracted from the original write data to generate node data, and the relationships between entities can be extracted from the write data based on the information corresponding to entities to obtain edge data.

[0124] According to embodiments of this disclosure, in an operation and maintenance scenario, an entity can be an entity operation and maintenance object, i.e., a network device such as a server or router. Node data can be used to characterize the information of each entity operation and maintenance object, and edge data can be used to characterize the communication link information between two entity operation and maintenance objects that have a communication relationship.

[0125] According to embodiments of this disclosure, the database type includes a document database type. Data processing is performed on the original written data to obtain written data corresponding to each database type, including: determining a classification field for identifying the original written data based on the attribute information of the original written data; and generating written data corresponding to the document database type based on the original written data and the classification field.

[0126] According to embodiments of this disclosure, since data in the document database is stored in the form of key-value pairs, i.e. key-value pairs consisting of a category field and a field value, but the original write data sent by the client may not include the corresponding category field, the coordinator can determine the category field corresponding to the original write data from multiple preset category fields based on the attribute information of the original write data, and use it as the category field of the original write data. The attribute information can be obtained by feature extraction of the write data.

[0127] According to embodiments of this disclosure, when the original written data is used to represent the addition of a new entity maintenance object, for graph database types, node data representing the entity maintenance object can be created based on the information related to the new entity maintenance object in the original written data, and edge data can be generated based on the entity maintenance objects in the original written data that are associated with the new entity maintenance object; for document database types, multiple classification fields corresponding to the original written data can be determined, such as device type, device identifier, etc.

[0128] According to the embodiments of this disclosure, since the operation and maintenance data has the characteristics of dynamic partitioning and small scale, the storage node can be divided into multiple partitions according to the device type of the physical operation and maintenance object. When sending write data to the storage node, the write sub-data can be sent to the partition of the storage node in parallel according to the device type of each of the multiple write sub-data in the write data, so as to improve the data sending efficiency.

[0129] According to embodiments of this disclosure, data consistency among multiple storage nodes is achieved by using a coordinator to transform the original write data into write data corresponding to the database type.

[0130] According to embodiments of this disclosure, the data writing method further includes: determining a write result of writing the data to the original storage data set, and sending the write result to the coordinator; and in response to receiving a rollback instruction sent by the coordinator, rolling back the original storage data set using the first mirrored data set.

[0131] According to embodiments of this disclosure, the rollback instruction is generated by the coordinator when it is determined that either the write result or at least one other write result characterizes a write failure.

[0132] According to embodiments of this disclosure, even if the pre-write simulation results indicate that the write data can be written to the original storage data set, write failures may still occur due to network issues or other reasons, leading to data inconsistencies among multiple storage nodes. Therefore, after writing the write data to the original data set, the storage node should also determine the write result and return it to the coordinator.

[0133] According to embodiments of this disclosure, if any write result indicates a write failure, it indicates that there are storage nodes that have not written the data to the original storage data set. In order to ensure data consistency among multiple storage nodes, it is necessary to roll back the original storage data set in all storage nodes.

[0134] According to embodiments of this disclosure, since the first mirrored data set has the same data storage state as the original stored data set before the write data was written, the original stored data set can be restored to the data storage state before the write data was written using the first mirrored data set. Specifically, the first mirrored data set can be copied to the disk through the connection between the local cache and the disk.

[0135] According to embodiments of this disclosure, the original storage data set is rolled back using a first mirrored data set to ensure data consistency among multiple storage nodes.

[0136] Figure 6 A flowchart illustrating the submission phase according to another embodiment of this disclosure is shown.

[0137] like Figure 6 As shown, the submission phase includes operations S610 to S640.

[0138] During operation S610, the coordinator sends commit commands to the first storage node and the second storage node, respectively.

[0139] In operation S620, the first storage node sends a write result to the coordinator to indicate a write failure, and the second storage node sends a write result to the coordinator to indicate a write success.

[0140] According to embodiments of this disclosure, the first storage node and the second storage node, in response to receiving a commit instruction, write the data to their respective original storage data sets.

[0141] According to an embodiment of this disclosure, if the first storage node fails to write the data to the original storage data set, it sends a write result to the coordinator to indicate that the write failed; if the second storage node successfully writes the data to the original storage data set, it sends a write result to the coordinator to indicate that the write succeeded.

[0142] During operation S630, the coordinator sends rollback commands to the first storage node and the second storage node, respectively.

[0143] According to embodiments of this disclosure, when the coordinator determines that there is a write result indicating a write failure among multiple write results, it sends a rollback instruction to the first storage node and the second storage node. Specifically, the first storage node and the second storage node can use their respective first mirror data sets to roll back their respective original storage data sets.

[0144] In operation S640, the coordinator sends a commit result to the client to indicate that the commit failed.

[0145] According to embodiments of this disclosure, if the coordinator determines that there is a write result indicating a write failure among multiple write results, it returns a commit result indicating a commit failure to the client.

[0146] Figure 7 A flowchart illustrating a data writing method according to a specific embodiment of the present disclosure is shown.

[0147] like Figure 7 As shown, the data writing method includes operations S701 to S712.

[0148] When operating the S701, the client sends a write request.

[0149] During operation S702, the coordinator processes the raw write data in the write request to obtain the write data for the graph database and the write data for the document database, and sends them to the graph database and the document database respectively.

[0150] When operating S703, the graph database is pre-written.

[0151] According to embodiments of this disclosure, the graph database performs a pre-write operation on the written data in response to receiving the written data.

[0152] According to embodiments of this disclosure, write data can be written to a first mirrored data set, simulating the writing of write data to a graph database to obtain a second mirrored data set, and the validity of the second mirrored data set is verified to obtain a validity verification result. If the second mirrored data set is valid, the pre-write is determined to be successful; if the second mirrored data set is invalid, the pre-write is determined to be unsuccessful.

[0153] When operating S704, the document database is pre-written.

[0154] According to embodiments of this disclosure, in response to receiving write data, the document database performs a pre-write operation on the write data. Specifically, the write data can be written to a first mirror data set to simulate writing the write data to the document database.

[0155] According to embodiments of this disclosure, write data can be written to a first mirrored data set, simulating the writing of write data to a document database to obtain a second mirrored data set, and the validity of the second mirrored data set is verified to obtain a validity verification result. If the second mirrored data set is valid, the pre-write is determined to be successful; if the second mirrored data set is invalid, the pre-write is determined to be unsuccessful.

[0156] When operating the S705, check whether the pre-write was successful.

[0157] According to embodiments of this disclosure, a coordinator can be used to determine whether both the graph database and the document database have been successfully pre-written. If at least one pre-write fails, operation S706 is executed; if both pre-writes are successfully pre-written, operations S707 and S708 are executed.

[0158] According to embodiments of this disclosure, if the coordinator determines that both the graph database and the document database have been successfully written, it sends a commit instruction to both the graph database and the document database.

[0159] In operation S706, the coordinator confirms the cancellation of the commit.

[0160] According to embodiments of this disclosure, if there is a pre-write simulation result that indicates a pre-write failure among multiple pre-write simulation results, the written data will not be committed in order to ensure data consistency.

[0161] When operating S707, the graph database is written.

[0162] According to embodiments of this disclosure, in response to receiving a commit instruction, the graph database writes the data to the original stored data set and sends the write result to the coordinator.

[0163] When operating S708, the document database is written.

[0164] According to embodiments of this disclosure, in response to receiving a commit instruction, the document database writes the data to the original stored data set and sends the write result to the coordinator.

[0165] When operating the S709, check if all writes were successful.

[0166] According to embodiments of this disclosure, the coordinator can be used to determine that both the graph database and the document database have been successfully written. If both have been successfully written, operation S710 is executed. If at least one has been determined to have failed to write, operations S711 and S712 are executed.

[0167] According to embodiments of this disclosure, if the coordinator determines that at least one write operation to the graph database and the document database has failed, it sends a rollback instruction to the graph database and the document database.

[0168] When operating the S710, the coordinator confirmed that the write was successful.

[0169] During S711 operation, the graph database is rolled back.

[0170] According to embodiments of this disclosure, the graph database, in response to receiving a rollback instruction, performs a rollback using a first mirrored data set.

[0171] During S712 operation, the document database is rolled back.

[0172] According to embodiments of this disclosure, the document database, in response to receiving a rollback instruction, performs a rollback using a first mirrored data set.

[0173] Based on the above data writing method, this disclosure also provides a data writing device. The following will be combined with... Figure 8 The device is described in detail.

[0174] Figure 8 A schematic block diagram of a data writing apparatus according to an embodiment of the present disclosure is shown.

[0175] like Figure 8 As shown, the data writing device 800 of this embodiment includes a first writing module 810, a sending module 820, and a second writing module 830.

[0176] The first write module 810 is used to respond to the write data sent by the coordinator, and simulate writing the write data to the storage node based on the write data and the first mirror data set to obtain a pre-write simulation result. The first mirror data set is obtained by copying the original storage data set stored in the storage node. In one embodiment, the first write module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0177] The sending module 820 is used to send the pre-written simulation results to the coordinator. In one embodiment, the sending module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0178] The second write module 830 is used to receive a commit instruction sent by the coordinator and write the write data to the original storage data set. The commit instruction is generated by the coordinator after determining that both the pre-write simulation result and at least one other pre-write simulation result indicate successful pre-write. The at least one other pre-write simulation result comes from at least one other storage node different from the storage node. In one embodiment, the second write module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0179] According to embodiments of this disclosure, the first writing module includes a writing submodule, a verification submodule, a first determining submodule, and a second determining submodule.

[0180] The write submodule is used to write the data into the first mirror data set to obtain the second mirror data set.

[0181] The validation submodule is used to validate the second mirrored data set and obtain the validation results.

[0182] The first determination submodule is used to determine the pre-write simulation result to indicate that the pre-write was successful, provided that the validity verification result indicates that the second mirror data set is valid.

[0183] The second determination submodule is used to determine the pre-write simulation result as indicating pre-write failure if the determination validity verification result indicates that the second mirror data set is not valid.

[0184] According to embodiments of this disclosure, the data writing device 800 further includes a cancellation module.

[0185] The cancel commit module is used to delete the second mirror data set in response to receiving a cancel commit instruction sent by the coordinator. The cancel commit instruction is generated by the coordinator when it is determined that either the pre-write simulation result or at least one other pre-write simulation result characterizes a pre-write failure.

[0186] According to embodiments of this disclosure, the verification submodule includes a first verification unit and a second verification unit.

[0187] The first verification unit is used to perform validity verification on the data written in the second mirror data set and the first mirror data set respectively after performing the same preprocessing to obtain the first validity verification result. The same preprocessing includes at least one of the following: semantic extraction and node data recognition.

[0188] The second verification unit is used to obtain a validity verification result based on the first validity verification result and the second validity verification result, wherein the second validity verification result is obtained by performing data logic verification on the written data.

[0189] According to embodiments of this disclosure, the write data is obtained by the following operation: in response to receiving the raw write data sent by the client, the coordinator performs data processing on the raw write data based on the respective database types of the storage node and at least one other storage node different from the storage node, to obtain write data corresponding to each database type.

[0190] According to embodiments of this disclosure, the database type includes a graph database type. Data processing is performed on the original written data to obtain written data corresponding to each database type, including: generating node data corresponding to each entity based on at least one entity involved in the original written data; obtaining edge data for representing relationships based on the original written data; and generating written data corresponding to the graph database type based on the node data and edge data. The database type also includes a document database type. Data processing is performed on the original written data to obtain written data corresponding to each database type, including: determining a classification field for identifying the original written data based on attribute information of the original written data; and generating written data corresponding to the document database type based on the original written data and the classification field.

[0191] According to embodiments of this disclosure, the data writing device 800 further includes a result determination module and a rollback module.

[0192] The result determination module is used to determine the write result of writing the data to the original storage data set and send the write result to the coordinator.

[0193] The rollback module is used to roll back the original stored data set in response to receiving a rollback command from the coordinator, using the first mirrored data set. The rollback command is generated by the coordinator when it determines that either the write result or at least one other write result indicates a write failure.

[0194] According to embodiments of this disclosure, any plurality of modules among the first write module 810, the transmit module 820, and the second write module 830 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first write module 810, the transmit module 820, and the second write module 830 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first write module 810, the transmit module 820, and the second write module 830 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0195] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data writing method according to an embodiment of the present disclosure.

[0196] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0197] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0198] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0199] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0200] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0201] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the data writing method provided in the embodiments of this disclosure.

[0202] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0203] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0204] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0205] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0207] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0208] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data writing method, characterized in that, Applied to storage nodes, the method includes: In response to receiving write data sent by the coordinator, based on the write data and the first mirror data set, the write data is simulated to be written to the storage node to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; Send the pre-written simulation results to the coordinator; and The coordinator sends a commit instruction to write the write data into the original storage data set. The commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both represent successful pre-write. The at least one other pre-write simulation result comes from at least one other storage node that is different from the storage node. The step of simulating writing the write data to the storage node based on the write data and the first mirror data set to obtain pre-write simulation results includes: The written data is written into the first mirror data set to obtain the second mirror data set; The validity of the second mirrored data set is verified, and the validity verification result is obtained; If the validity verification result indicates that the second mirrored data set is valid, then the pre-write simulation result indicates that the pre-write was successful; and If the validity verification result indicates that the second mirrored data set is not valid, the pre-write simulation result indicates that the pre-write has failed.

2. The method according to claim 1, characterized in that, The method further includes: In response to receiving a cancel commit instruction from the coordinator, the second mirror data set is deleted, wherein the cancel commit instruction is generated by the coordinator when it is determined that any one of the pre-write simulation results and at least one other pre-write simulation result characterizes a pre-write failure.

3. The method according to claim 1, characterized in that, The validity verification of the second mirrored data set, to obtain the validity verification result, includes: After performing the same preprocessing on the written data in the second mirrored data set and the first mirrored data set, a validity verification is performed to obtain a first validity verification result. The same preprocessing includes at least one of the following: semantic extraction, node data identification; and The validity verification result is obtained based on the first validity verification result and the second validity verification result, wherein the second validity verification result is obtained based on data logic verification of the written data.

4. The method according to claim 1, characterized in that, The written data is obtained through the following operations: In response to receiving raw write data sent by the client, the coordinator processes the raw write data based on the database type of the storage node and at least one other storage node that is different from the storage node, to obtain write data corresponding to each of the database types.

5. The method according to claim 4, characterized in that, The database type includes a graph database type. The data processing of the original written data to obtain written data corresponding to each database type includes: Based on at least one entity involved in the original written data, node data corresponding to each entity is generated; Based on the original written data, edge data used to represent the association relationship is obtained; and Based on the node data and the edge data, write data corresponding to the graph database type is generated; The database type includes a document database type. The data processing of the original written data to obtain written data corresponding to each database type includes: Based on the attribute information of the original written data, a classification field for identifying the original written data is determined; and Based on the original write data and the classification field, write data corresponding to the document database type is generated.

6. The method according to claim 1, characterized in that, The method further includes: Determine the write result of writing the data to the original storage data set, and send the write result to the coordinator; and In response to receiving a rollback instruction from the coordinator, the original storage data set is rolled back using the first mirrored data set, wherein the rollback instruction is generated by the coordinator when it is determined that either the write result or at least one other write result indicates a write failure.

7. A data writing device, characterized in that, The device includes: The first write module is configured to, in response to receiving write data sent by the coordinator, simulate writing the write data to the storage node based on the write data and the first mirror data set, to obtain a pre-write simulation result, wherein the first mirror data set is obtained by copying the original storage data set stored in the storage node; A sending module is configured to send the pre-written simulation results to the coordinator; and The second write module is used to receive the commit instruction sent by the coordinator and write the write data into the original storage data set. The commit instruction is generated by the coordinator when it is determined that the pre-write simulation result and at least one other pre-write simulation result both represent the success of the pre-write. The at least one other pre-write simulation result comes from at least one other storage node that is different from the storage node. The first writing module includes: The write submodule is used to write the write data into the first mirror data set to obtain the second mirror data set; The verification submodule is used to perform validity verification on the second mirrored data set and obtain the validity verification result. The first determining submodule is configured to, if the validity verification result indicates that the second mirrored data set is valid, determine that the pre-write simulation result indicates that the pre-write was successful; and The second determining submodule is used to determine that the pre-write simulation result indicates pre-write failure if the validity verification result indicates that the second mirrored data set is not valid.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6.

Images