Data synchronization methods, distributed systems, storage media and electronic devices

Abstract

The present disclosure relates to the technical field of data processing, and relates to a data synchronization method, a distributed system, a computer readable storage medium and an electronic device. The data synchronization method is applied to a subsystem in a distributed system, the subsystem comprising a plurality of sub-processes and a shared memory between the plurality of sub-processes, and the data synchronization method comprises: in response to a data synchronization request, a first sub-process determining a data processing strategy according to a request type; and the first sub-process performing data synchronization on the shared memory according to the data processing strategy; wherein the first sub-process monitors a read state of old data corresponding to the data synchronization based on a timing mechanism, so as to perform invalidation processing on the old data according to the read state. The data synchronization method of the present disclosure avoids resource competition between multi-core systems, realizes data consistency of each core in the multi-core system, and guarantees the performance of the distributed system.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data synchronization method, a distributed system, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the rise of distributed microservice architecture, in practical applications, inconsistencies in data between multiple applications within a distributed system can arise due to complex network conditions and dependencies. This can lead to system malfunctions and reduced performance. Therefore, achieving data synchronization in distributed systems has become particularly important.

[0003] However, when these technologies are used for data synchronization in distributed systems, there is a problem of inconsistent data synchronization among the cores in a multi-core system, which affects the performance of the multi-core system to some extent.

[0004] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a data synchronization method, a distributed system, a computer-readable storage medium, and an electronic device, thereby avoiding resource contention between multi-core systems, achieving data consistency in multi-core systems, and ensuring the stability of the distributed system.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0007] According to one aspect of this disclosure, a data synchronization method is provided, applied to a subsystem in a distributed system, the subsystem including multiple subprocesses and shared memory among the multiple subprocesses, the data synchronization method comprising: responding to a data synchronization request; a first subprocess determining a data processing strategy based on a request type; the first subprocess performing data synchronization on the shared memory according to the data processing strategy; wherein the first subprocess monitors the read status of old data corresponding to the data synchronization based on a timed mechanism, so as to invalidate the old data according to the read status.

[0008] In one exemplary embodiment of this disclosure, the first subprocess performs data synchronization on the shared memory according to the data processing strategy, including: after the first subprocess acquires a security lock on the shared memory, it performs data synchronization on the shared memory according to the data processing strategy; monitors the read lock count of the old data corresponding to the data synchronization based on a timed mechanism, the read lock count being determined based on the number of second subprocesses that have read the old data; if the read lock count is zero, the old data is invalidated.

[0009] In an exemplary embodiment of this disclosure, if the request type is data modification, the first subprocess performs data synchronization on the shared memory according to the data processing strategy, including: the first subprocess modifies the old data in the shared memory to obtain target data, the target data being used for subsequent data reading; an expiration time is added to the old data, and when the expiration time expires, the old data is invalidated according to the read status.

[0010] In one exemplary embodiment of this disclosure, the method further includes: configuring a first identifier for the target data and configuring a second identifier for the old data, wherein the first identifier has a higher priority than the second identifier; and prioritizing reading the target data with the first identifier before obtaining the target data and before the expiration time expires.

[0011] In one exemplary embodiment of this disclosure, if the request type is data deletion, the first subprocess performs data synchronization on the shared memory according to the data processing strategy, including: adding an expiration time to the old data, and deleting the old data according to the read status when the expiration time expires.

[0012] In one exemplary embodiment of this disclosure, the process of determining the failure time includes: obtaining the number of sub-processes that have read the old data and the service type corresponding to each sub-process based on the read status of the old data; and determining the failure time based on the number of sub-processes and the service type.

[0013] In one exemplary embodiment of this disclosure, if the request type is to add data, the first subprocess performs data synchronization on the shared memory according to the data processing strategy, including: adding the data to be added to the shared memory.

[0014] According to one aspect of this disclosure, a distributed system is provided, the distributed system including a data processing module and a plurality of subsystems; the data processing module is configured to process data to be synchronized and update a database according to the obtained standard data; the data processing module is further configured to send data synchronization requests to the subsystems sequentially; any of the subsystems is configured to perform data synchronization processing according to the data synchronization method described in any of the exemplary embodiments above, so as to obtain subsystem data consistent with the database.

[0015] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0016] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0017] The data synchronization method in the exemplary embodiments of this disclosure, within a subsystem of a distributed system, involves a first subprocess receiving and responding to data synchronization requests, determining a data processing strategy based on the request type, and synchronizing data in shared memory according to the data processing strategy. Specifically, the first subprocess monitors the read status of old data corresponding to the data synchronization based on a timed mechanism and invalidates the old data based on the read status. On one hand, the first subprocess synchronizes data in shared memory according to the data processing strategy, ensuring data consistency between the subsystem and the distributed system, avoiding system malfunctions or performance degradation due to data inconsistencies between systems, thus fully leveraging the advantages of the distributed system. On the other hand, while synchronizing data in shared memory, the first subprocess monitors the read status of old data corresponding to the data synchronization based on a timed mechanism. Since other subprocesses may have already read the old data during data synchronization, to prevent affecting the processing progress and results of other subprocesses, the first subprocess invalidates the old data based on its read status, avoiding premature invalidation of old data that could cause system malfunctions or performance problems. In other words, data synchronization is achieved while ensuring the performance of the distributed system.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0020] Figure 1 A schematic diagram of a process model of an Nginx subsystem according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 2 A flowchart of a data synchronization method according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 3 A flowchart illustrating an implementation of a first subprocess synchronizing data in shared memory according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 4 A flowchart illustrating an implementation of determining failure time according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 5 A complete flowchart of a data synchronization according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic diagram of a distributed system according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 7 A schematic diagram of another distributed system according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0028] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0032] In distributed system applications, complex network conditions and dependencies may lead to data inconsistencies between multiple applications in each subsystem, which may result in abnormal system function or reduced system performance.

[0033] During data synchronization, if the subsystem is a single-core or single-process system, the data in the subsystem is consistent with the data in the main system. However, if the subsystem is a multi-process (or multi-core) subsystem, the data synchronization request will be received and processed by one of the subprocesses, while other subprocesses that have not received the data synchronization request will continue to use the old data. This leads to the problem of data inconsistency between multiple subprocesses in the subsystem.

[0034] For example, such as Figure 1 The diagram illustrates a process model of an Nginx subsystem according to an exemplary embodiment of this disclosure. The Nginx subsystem includes one master core and multiple worker cores, which need to maintain data consistency. However, currently only one worker core receives and processes data synchronization requests, while the user data on other worker cores remains outdated.

[0035] Based on this, in the exemplary embodiments of this disclosure, a data synchronization method is first provided. This method can be applied to any subsystem in a distributed system, which includes multiple child processes and shared memory among the child processes. (See reference...) Figure 2 A flowchart of a data synchronization method according to an exemplary embodiment of the present disclosure is shown, including steps S210 and S220:

[0036] Step S210: In response to the data synchronization request, the first child process determines the data processing strategy based on the request type;

[0037] Step S220: The first subprocess performs data synchronization on the shared memory according to the data processing strategy; wherein, the first subprocess monitors the read status of the old data corresponding to the data synchronization based on a timed mechanism, so as to invalidate the old data according to the read status.

[0038] Based on the data synchronization method in the exemplary embodiments of this disclosure, on the one hand, the first subprocess synchronizes the data in the shared memory according to the data processing strategy, so that the data in the subsystem is consistent with the data in the distributed system, avoiding the problem of system malfunction or performance degradation caused by data inconsistency between systems, so as to give full play to the advantages of the distributed system; on the other hand, when the first subprocess synchronizes the data in the shared memory, it monitors the read status of the old data corresponding to the data synchronization based on a timed mechanism. Since there may be other subprocesses that have read the old data during data synchronization, in order to prevent affecting the processing progress and results of other subprocesses, the first subprocess performs invalidation processing on the old data according to the read status of the old data, avoiding system malfunction or performance problems caused by premature invalidation of old data, that is, while realizing data synchronization, the performance of the distributed system is guaranteed.

[0039] The following is a detailed explanation of each of the above steps.

[0040] In step S210, in response to a data synchronization request, the first subprocess determines a data processing strategy based on the request type.

[0041] In the exemplary embodiments of this disclosure, the data synchronization request is issued by the data processing module of the distributed system. This data processing module can process the data to be synchronized, obtain standard data, and save the standard data to the database. The data to be synchronized is typically the user's configuration data, which the data processing module processes and stores in the database.

[0042] The data processing module can preprocess the data to be synchronized, such as format conversion and data filtering, to obtain the data required by the subsystem and adapted to the subsystem. The specific method of preprocessing is not specifically limited in the embodiments of this disclosure.

[0043] The first child process can be any child process among multiple child processes in the subsystem. For example, the currently idle child process can be determined as the first child process; or a rotation mechanism can be used to rotate one process from multiple child processes to become the first child process each time. Of course, the embodiments of this disclosure can also use other methods to determine the first child process from multiple child processes, and there is no special limitation on this.

[0044] The request types in this embodiment include modifying data, deleting data, and adding data. The first subprocess determines the corresponding data processing strategy based on the request type. A pre-defined correspondence between different request types and data processing strategies can be established so that the first subprocess determines the data processing strategy based on this correspondence.

[0045] In step S220, the first subprocess performs data synchronization on the shared memory according to the data processing strategy; wherein, the first subprocess monitors the read status of the old data corresponding to the data synchronization based on a timer mechanism, so as to invalidate the old data according to the read status.

[0046] In the exemplary embodiments of this disclosure, shared memory can be accessed by multiple child processes in the subsystem, and each child process can read or process the data in the shared memory. Data synchronization of the shared memory can ensure that the data accessed by the multiple child processes is consistent with the data in the database of the distributed system.

[0047] Data synchronization of shared memory refers to adding, deleting, and modifying data in shared memory according to data processing strategies, so that the data in shared memory is the data required by the subsystem and is consistent with the data in the database of the distributed system.

[0048] In some possible embodiments, when adding data to shared memory according to the data processing strategy, there is no corresponding old data for data synchronization, that is, there is no need to monitor the read status of old data based on a timed mechanism.

[0049] In some possible embodiments, when modifying or deleting data in shared memory according to the data processing strategy, since the modification or deletion of old data is involved, in order to avoid affecting the data processing of other subprocesses that have read the old data, the first subprocess needs to monitor the read status of the old data corresponding to the data synchronization based on a timed mechanism, so as to invalidate the old data according to the read status.

[0050] It should be noted that after the data in the shared memory is modified according to the data processing strategy, subsequent child processes will access the modified data when accessing the shared memory, in order to ensure the timeliness of data consistency between the child processes and the distributed system.

[0051] In this context, "old data in the read state" means that the old data has been accessed by a process and used for data processing, such as message processing. If the old data is no longer in the read state, it means that the process that accessed the old data has finished processing the message.

[0052] Failure handling of old data can be achieved by deleting the old data in shared memory and releasing the corresponding memory space, thereby making the shared memory of the subsystem consistent with the database of the distributed system.

[0053] In this embodiment of the disclosure, when synchronizing data in shared memory, the first subprocess monitors the read status of the old data corresponding to the data synchronization and performs invalidation processing on the old data according to the read status. This ensures data consistency between the subsystem and the database of the distributed system while avoiding the impact on the performance of the subsystem. In other words, new data can be accessed in a timely manner, and old data is retained until the subprocess completes data processing, thus maintaining the stability of the system.

[0054] In one exemplary embodiment, a method is provided for a first child process to synchronize data on shared memory. For example... Figure 3 As shown, the first child process performs data synchronization of shared memory according to the data processing strategy, including steps S310 to S330:

[0055] Step S310: After the first child process acquires the security lock of the shared memory, it performs data synchronization on the shared memory according to the data processing strategy.

[0056] The first child process can acquire the security lock of the shared memory. In this embodiment of the disclosure, after the first child process responds to the data synchronization request and determines the data processing strategy, it acquires the security lock of the shared memory and can read or process the data in the shared memory (such as deleting, adjusting, etc.). During this data synchronization process, other child processes can read the data in the shared memory, but cannot modify it.

[0057] Step S320: The first child process monitors the read lock count of the old data corresponding to the data synchronization based on a timer mechanism. The read lock count is determined based on the number of second child processes that have read the old data.

[0058] While the first child process synchronizes data in shared memory, the read lock count of the old data corresponding to the data synchronization can be continuously monitored. In this embodiment, the read lock count of the subsystem is determined based on the number of second child processes that have already read the old data.

[0059] For example, in an Nginx system, the read lock count is determined based on the number of worker cores that acquire the old data block in shared memory. When a child process acquires the read lock of an old data block, the read lock count of that old data block is incremented by 1, and when the child process releases the read lock of an old data block, the read lock count of that old data block is decremented by 1.

[0060] The read status of old data blocks can be determined by the read lock count of the old data. A read lock count of zero indicates that the old data block is no longer in a read state, while a non-zero read lock count indicates that a child process is still reading the old data block for corresponding data processing.

[0061] Step S330: If the read lock count is zero, invalidate the old data.

[0062] When the read lock count of the old data corresponding to the data synchronization is zero, it indicates that the failure of the old data will not affect the system performance or cause system errors, and the old data can be invalidated.

[0063] The embodiments of this disclosure can determine whether a read lock has been completely released by a child process based on the read lock count of old data, so as to perform invalidation processing of old data and ensure the stability of system performance.

[0064] In an exemplary embodiment, the timing mechanism may be based on determining an expiration time. Before the expiration time expires, the old data is not processed for expiration. The expiration time refers to the time required to ensure that the child process that acquired the old data read lock after the data synchronization process has finished processing the message and released the read lock.

[0065] In one possible embodiment, if the request type is data modification, the first child process synchronizing data in shared memory according to the data processing strategy may include:

[0066] The first child process modifies the old data in shared memory to obtain the target data, which is used for subsequent data reading. Then, the first child process adds an expiration time to the old data, and when the expiration time expires, the old data is invalidated according to the read status.

[0067] Optionally, if the old data is no longer in a read state after the expiration time, it can be invalidated to avoid accidental operation. Optionally, if the old data is detected to have been out of a read state for more than a preset time before the expiration time expires, it can be invalidated in advance to reduce the occupation of shared memory.

[0068] Furthermore, in an exemplary embodiment, before the expiration time expires, old data and target data will exist in the shared memory. Based on this, an implementation method for determining the data to be read is also provided. Specifically, a first identifier can be configured for the target data, and a second identifier can be configured for the old data, with the first identifier having a higher priority than the second identifier. Before the target data is obtained and the expiration time expires, the target data with the first identifier is read first.

[0069] For example, in an Nginx system, before the target data and its expiration time expire, there are two copies of the same index in shared memory—one for the old data and one for the target data. A first identifier can be configured for the target data, such as an expiration flag of "0," and a second identifier can be configured for the old data, such as an expiration flag of "1." When worker cores subsequently retrieve data from shared memory, they will prioritize querying the data with an expiration flag of "0" based on the index, thus using the consistent data for message processing.

[0070] It should be noted that the first and second identifiers mentioned above are merely exemplary and can be set according to actual needs. This disclosure does not impose any special limitations on them.

[0071] In practice, adding time to old data can be done by adding a timer to the old data. The timer's time pointer is stored in the old data to precisely control the expiration time of the old data.

[0072] In an exemplary embodiment, if the request type is to delete data, the first subprocess performs data synchronization on the shared memory according to the data processing strategy. This may involve adding an expiration time to the old data and deleting the old data based on the read status when the expiration time expires.

[0073] In this exemplary embodiment, the old data can also be invalidated by adding a timer to the old data. The timer's pointer is stored in the old data, and the old data is deleted based on its read status when the expiration time expires. For example, if the last worker core in the Nginx system releases the read lock on the shared memory and the timer expires, the old data will be invalidated.

[0074] Based on this, it can be ensured that the child process that has read the old data completes the data processing, avoiding the impact on system stability due to the deletion of data in shared memory.

[0075] In one exemplary embodiment, an implementation method for determining the failure time is also provided. For example... Figure 4 The process of determining the failure time may include steps S410 and S420:

[0076] Step S410: Based on the read status of the old data, obtain the number of subprocesses that have read the old data and the business type corresponding to each subprocess.

[0077] The read status of old data can include the number of child processes reading the old data and the business type that each child process is handling. Typically, the duration for which old data remains in the read state is related to the number of child processes reading it; a larger number of child processes necessitates a longer expiration time. Similarly, if the business type corresponding to a child process is time-consuming, the expiration time also needs to be delayed.

[0078] Step S420: Determine the expiration time based on the number of subprocesses and the service type.

[0079] In this embodiment of the disclosure, the failure time can be determined by combining the number of subprocesses that have read old data and the corresponding service type.

[0080] Specifically, a first correspondence between the number of subprocesses and time can be pre-defined, so that a first time can be determined based on the number of subprocesses according to the first correspondence. Furthermore, a second correspondence between business type and time can be pre-defined, so that a second time can be determined based on the business type of the subprocess that has read old data, and the expiration time can be determined based on both the first and second times.

[0081] The first and second correspondences can be determined based on the historical information of each subprocess of the subsystem. This disclosure does not impose any special limitations on the method of determining the first and second correspondences.

[0082] Optionally, when determining the failure time based on the first time and the second time, weight information can be assigned to the first time and the second time respectively, so that the first time and the second time are weighted and summed based on the weight information to obtain the failure time. The second time is the second time with the largest value among the second times determined according to the business type among the multiple sub-processes that have read the old data, thus ensuring that the old data is fully utilized.

[0083] Optionally, when determining the failure time based on the first time and the second time, the time with the larger value between the first time and the second time can be used as the failure time to fully guarantee the access time of the old data.

[0084] Of course, the embodiments of this disclosure may also combine the first time and the second time in other ways to obtain the failure time. The embodiments of this disclosure include, but are not limited to, the combination methods described above.

[0085] In one exemplary embodiment, if the request type is to add data, the first subprocess synchronizes data in the shared memory according to the data processing strategy, which may involve adding the data to be added to the shared memory. That is, if there is data to be added, the data to be added can be directly added to the shared memory without involving the monitoring process of the old data.

[0086] Figure 5 A complete flowchart of a data synchronization according to an exemplary embodiment of this disclosure is shown below, in conjunction with... Figure 5 Taking the Nginx system as an example, the data synchronization method of this disclosure embodiment will be described.

[0087] Step S510: The data processing module in the distributed system receives the user's configuration data, processes the configuration data to obtain standard data, and stores it in the database.

[0088] Step S520: The data processing module sends a data synchronization request to the Nginx subsystem.

[0089] In this system, the distributed system can send data synchronization requests to the Nginx system via HTTP messages.

[0090] Step S530: Worker A in the Nginx system receives and responds to the data synchronization request.

[0091] In this process, worker A determines the data processing strategy based on the request type of the data synchronization request. If the request type is to add data, the process proceeds to step S540; if the request type is to modify data, the process proceeds to step S550; and if the request type is to delete data, the process proceeds to step S560.

[0092] Step S540: Add the data to be added to the shared memory.

[0093] This involves integrating the data to be added to obtain data adapted to the Nginx system and saving it to shared memory.

[0094] Step S550: Modify the old data in the shared memory to obtain the target data, add a timer to the old data, configure a first identifier for the target data and a second identifier for the old data.

[0095] Specifically, before the target data is obtained and the timer expires, the target data with the first identifier is read first. After the timer expires, the old data is deleted.

[0096] Step S560: Add an expiration time to the old data, and delete the old data according to the read status when the timer expires.

[0097] Before the expiration time expires, the subprocess that reads the data to be deleted (old data) continues to use the old data for message processing.

[0098] Thus, while synchronizing new data to shared memory, system problems caused by the modification or deletion of data in shared memory are avoided. By adopting a timed mechanism to delay the deletion of old data, the contention between read and write locks is reduced, ensuring that new data is accessed in a timely manner and old data is fully utilized during data synchronization in the multi-core subsystem, thereby ensuring system stability.

[0099] The data synchronization method in the exemplary embodiments of this disclosure, within a subsystem of a distributed system, involves a first subprocess receiving and responding to data synchronization requests, determining a data processing strategy based on the request type, and synchronizing data in shared memory according to the data processing strategy. Specifically, the first subprocess monitors the read status of the old data corresponding to the data synchronization based on a timed mechanism and invalidates the old data according to the read status. On the one hand, the first subprocess synchronizes data in shared memory according to the data processing strategy, ensuring data consistency between the subsystem and the distributed system. This avoids system malfunctions or performance degradation due to data inconsistencies, thus fully leveraging the advantages of the distributed system. On the other hand, while synchronizing data in shared memory, the first subprocess monitors the read status of the old data corresponding to the data synchronization based on a timed mechanism. Since other subprocesses may have already read the old data during data synchronization, the first subprocess invalidates the old data based on its read status to prevent premature invalidation and related system malfunctions or performance issues. This achieves data synchronization while ensuring the performance of the distributed system. Furthermore, for currently popular distributed systems, this solution enables consistency management of multi-core subsystems. Without affecting existing forwarding performance and functionality, it achieves data consistency among subprocesses or cores in a multi-core subsystem, providing a data foundation for fully utilizing the advantages of the distributed system.

[0100] Furthermore, according to exemplary embodiments of this disclosure, a distributed system is also provided, such as... Figure 6 As shown, the distributed system 600 includes a data processing module 610 and multiple subsystems 620.

[0101] The data processing module 610 is configured to process the data to be synchronized and update the database based on the obtained standard data;

[0102] The data processing module 610 is also configured to send data synchronization requests to the subsystem 620 in sequence.

[0103] Any subsystem 620 is configured to perform data synchronization processing according to any of the data synchronization methods in the exemplary embodiments described above, so as to obtain subsystem data consistent with the database of the distributed system.

[0104] Specifically, the data processing module 610 processes the data to be synchronized to obtain standard data, and saves the standard data to the database. The data to be synchronized is usually user-configured data; the data processing module processes the user-configured data and stores it in the database.

[0105] The data processing module 610 can preprocess the data to be synchronized, such as format conversion and data filtering, to obtain data that is compatible with the subsystem. The specific method of preprocessing is not limited in this embodiment.

[0106] The data processing module 610 sends a data synchronization request to the subsystem 620 via HTTP (Hypertext Transfer Protocol) messages, requesting the subsystem 620 to complete a consistency update with the database. The subsystem 620 may include subsystems of distributed systems such as Nginx systems and AI systems, without special limitations.

[0107] It should be noted that if a subsystem 620 is a single-core or single-process system, the data of that subsystem will be consistent with the database of the distributed system. This will not be described in detail.

[0108] Each subsystem 620 may have a processing priority, and the data processing module 610 sends data synchronization requests to each subsystem 620 in sequence according to the processing priority.

[0109] In one exemplary embodiment, the distributed system may further include a data forwarding module, such as Figure 7 A schematic diagram of another distributed system according to an exemplary embodiment of the present disclosure is shown.

[0110] like Figure 7 As shown, users can configure data through the web configuration frontend, obtain configuration data, and send it to the data processing module 610. The data processing module 610 processes the data to obtain standard data which is then stored in the database. That is, the standard data in the database must be consistent with the data in each subsystem.

[0111] The data processing module 610 can call the interface function of the data forwarding module 630 to complete the synchronization of forwarded data, and determine whether to continue data synchronization for each subsystem based on the response result of the data forwarding module 630.

[0112] Specifically, if the response result of the data forwarding module 630 is failure, it may be due to reasons such as communication between systems or illegal user data that cause the forwarding data synchronization failure. In this case, the data processing module 610 receives the failure response message, rolls back the user data in the database, and returns the reason and result of the data synchronization failure to the user's web front end.

[0113] If the data forwarding module 630 responds successfully, the data processing module 610 uses an HTTP message to send a data synchronization request to a subsystem (such as an Nginx system, an AI system, etc.) so that a subprocess of the subsystem can determine and execute a data processing strategy based on the request type, thereby ensuring that the data in the subsystem's shared memory remains consistent with the database.

[0114] Furthermore, after the first subsystem completes data synchronization, the data processing module 610 will continue to send data synchronization requests to the second subsystem, requesting the second subsystem to perform data synchronization, until all subprocesses complete data synchronization.

[0115] In an exemplary embodiment, subsystem 620 is configured to perform: after acquiring a security lock on the shared memory through a first subprocess, perform data synchronization on the shared memory according to the data processing strategy; monitor the read lock count of the old data corresponding to the data synchronization based on a timed mechanism, wherein the read lock count is determined based on the number of second subprocesses that have read the old data; and if the read lock count is zero, invalidate the old data.

[0116] In an exemplary embodiment, if the request type is data modification, the subsystem 620 is configured to perform: modifying the old data in the shared memory through a first subprocess to obtain target data, the target data being used for subsequent data reading; adding an expiration time to the old data, and performing expiration processing on the old data according to the read status when the expiration time expires.

[0117] In an exemplary embodiment, subsystem 620 is configured to perform: configuring a first identifier for target data and configuring a second identifier for the old data through a first subprocess, wherein the first identifier has a higher priority than the second identifier; and prioritizing reading the target data with the first identifier before obtaining the target data and before the expiration time expires.

[0118] In an exemplary embodiment, if the request type is data deletion, the subsystem 620 is configured to perform: adding an expiration time to the old data through a first subprocess, and deleting the old data according to the read status when the expiration time expires.

[0119] In an exemplary embodiment, subsystem 620 is configured to perform: obtaining, through a first subprocess, the number of subprocesses that have read the old data and the service type corresponding to each subprocess based on the read status of the old data; and determining the failure time based on the number of subprocesses and the service type.

[0120] In an exemplary embodiment, if the request type is to add data, the subsystem 620 is configured to perform the following: adding the data to be added to the shared memory via a first subprocess.

[0121] In this embodiment of the disclosure, the data processing module sequentially sends data synchronization requests to the subsystems to enable data synchronization among the subsystems, ensuring data consistency across the subsystems in the distributed system. This improves the reliability and scalability of the distributed system, prevents system problems caused by data inconsistency, and fully demonstrates the high-performance advantages of the distributed system. The data synchronization method of this embodiment can achieve fast read and write speeds, facilitate sharing, and enable large-scale, highly reliable, high-performance, and hyper-converged storage applications, making it suitable for application scenarios with ever-increasing requirements for scalability and storage performance.

[0122] Since the specific details of the various functional modules (subsystems) of the distributed system in the exemplary embodiments of this disclosure have been described in detail in the inventive embodiments of the data synchronization method described above, they will not be repeated here.

[0123] It should be noted that although several modules or units of a distributed system have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0124] Furthermore, in exemplary embodiments of this disclosure, a computer storage medium capable of implementing the above-described methods is also provided. A program product capable of implementing the methods described in this specification is stored thereon. In some possible embodiments, various aspects of this disclosure can also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0125] This disclosure also provides a program product for implementing the above methods, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0129] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0130] Furthermore, in exemplary embodiments of this disclosure, an electronic device capable of implementing the above-described methods is also provided. Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as "circuit," "module," or "system."

[0131] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0132] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0133] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure.

[0134] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 821 and / or cache memory 822, and may further include a read-only memory (ROM) 823.

[0135] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0137] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0139] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0140] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A data synchronization method, characterized by, The application is applied to a subsystem in a distributed system, the subsystem comprising a plurality of sub-processes and shared memory between the plurality of sub-processes, the data synchronization method comprising: In response to a data synchronization request, a first sub-process determines a data processing strategy according to a request type; The first sub-process performs data synchronization on the shared memory according to the data processing strategy; wherein the first sub-process monitors a read state of old data corresponding to the data synchronization based on a timing mechanism, to perform invalidation processing on the old data according to the read state; If the request type is modifying data, the first sub-process modifies the old data in the shared memory to obtain target data, the target data being used for subsequent data reading; an invalidation time is added to the old data, and the old data is invalidated according to the read state when the invalidation time expires; If the request type is deleting data, an invalidation time is added to the old data, and the old data is deleted according to the read state when the invalidation time expires; the invalidation time is determined based on the number of sub-processes that have read the old data and the business types corresponding to each sub-process.

2. The method of claim 1, wherein, The first sub-process performs data synchronization on the shared memory according to the data processing strategy, comprising: After the first sub-process acquires a safety lock of the shared memory, the first sub-process performs data synchronization on the shared memory according to the data processing strategy; The first sub-process monitors a read lock count of old data corresponding to the data synchronization based on a timing mechanism, the read lock count being determined according to the number of second sub-processes that have read the old data; If the read lock count is zero, the old data is invalidated.

3. The method of claim 1, wherein, The method further comprises: A first identifier is configured to the target data, and a second identifier is configured to the old data, the priority of the first identifier being higher than that of the second identifier; Before the target data is obtained and the invalidation time expires, the target data with the first identifier is read preferentially.

4. The method of claim 1, wherein, The process of determining the invalidation time comprises: Based on the read state of the old data, the number of sub-processes that have read the old data and the business types corresponding to each sub-process are obtained; The invalidation time is determined according to the number of sub-processes and the business types.

5. The method of claim 1, wherein, If the request type is adding data, the first sub-process performs data synchronization on the shared memory according to the data processing strategy, comprising: The data to be added is added to the shared memory.

6. A distributed system, characterized by The distributed system comprises a data processing module and a plurality of subsystems; The data processing module is configured to process data to be synchronized, and update a database according to obtained standard data; The data processing module is further configured to send data synchronization requests to the subsystems in sequence; Any of the subsystems is configured to perform data synchronization processing according to the data synchronization method of any of claims 1 to 5, to obtain subsystem data consistent with the database.

7. A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the method of any of claims 1 to 5.

8. An electronic device, comprising: comprising: a processor; and ​ a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any one of claims 1 to 5 via execution of the executable instructions.

Images