Micro-service data consistency processing method and device, equipment and storage medium

Abstract

The invention relates to a micro-service data consistency processing method and device, equipment and a storage medium, and the method comprises the steps: executing a first operation which comprises the business logic execution of a first service and a corresponding data modification operation; after the first operation is executed, writing a to-be-sent message related to the first operation into a local message table; and sending the message to the message-oriented middleware, so that the second service can receive a notification that the first operation is successfully executed by subscribing the message sent to the message-oriented middleware, and execute a second operation of the second service based on an operation result of the first operation. According to the method, after the first service is successfully executed, the second service can perform subsequent operation based on the execution result of the first service, and the purpose of service consistency is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a microservice data consistency processing method, device, equipment and storage medium. Background Art

[0002] In the prior art, microservice architecture usually splits a single service into multiple microservices for decoupling and flexibility. Each microservice communicates through an interface, but because they are in different contexts, data consistency becomes a difficult problem in the microservice architecture. For example, after performing its own operation, service A may need to trigger service B to perform an operation. If the operation of service A succeeds but the operation of service B fails, the overall data of the system will be in an inconsistent state, which will cause serious problems for some businesses that require high consistency, such as transactions and inventory management.

[0003] To solve this problem, some companies use transaction management, but in distributed systems, traditional database transaction mechanisms are not applicable to microservice architectures. There are still many challenges in ensuring the eventual consistency of system services. Summary of the invention

[0004] The present application provides a microservice data consistency processing method, device and storage medium, aiming to solve the problem of data consistency between multiple services under the microservice architecture, especially when one service succeeds, another service must continue to perform operations after receiving the success event. In the present application, by introducing a local message table, message middleware and background task retry mechanism, it is ensured that after the first service is successfully executed, the second service can perform subsequent operations based on the execution result of the first service, thereby achieving final consistency.

[0005] In a first aspect, the present application provides a microservice data consistency processing method, the method is applied to a first service, and the method includes:

[0006] Execute a first operation, where the first operation includes executing the business logic of the first service and a corresponding data modification operation;

[0007] After the first operation is completed, writing a message to be sent related to the first operation into the local message table, where the message includes the execution result of the first operation and relevant information that needs to be notified to the second service;

[0008] After writing the message to the local message table, sending the message to the message middleware, so that the second service can receive a notification of successful execution of the first operation by subscribing to the message sent to the message middleware, and execute a second operation of the second service based on the operation result of the first operation;

[0009] Determine whether the message is successfully sent to the message middleware;

[0010] If not, the unsent messages in the local message table are scanned periodically by a background task, and the unsent messages are resent to the message middleware until the messages are sent successfully.

[0011] Furthermore, the method further comprises:

[0012] receiving an execution result of the second operation from the second service;

[0013] Based on the execution result of the second operation, determining whether the first service and the second service meet a preset data consistency requirement;

[0014] If not, check the atomicity of the business logic execution and the data modification operation, and re-execute the first operation.

[0015] Furthermore, the detecting the atomicity of the business logic execution and the data modification operation includes:

[0016] Obtaining the basic business logic of the first service;

[0017] Based on the basic business logic, creating a simulation environment for simulating execution of the first operation;

[0018] In the simulation environment, database transactions are used to detect the atomicity of the business logic execution and the data modification operation.

[0019] Furthermore, in the simulation environment, using database transactions to detect the atomicity of the business logic execution and the data modification operation includes:

[0020] Sequentially executing the business logic execution and data modification operation of the first operation in the simulation environment;

[0021] In response to the business logic of the first operation and the data modification operation being successfully executed, prompting for the operation to be re-executed;

[0022] In response to the failure of the business logic execution and / or data modification operation of the first operation, an operation abnormality is prompted.

[0023] Furthermore, when an abnormal operation is prompted, the method further includes:

[0024] Obtaining log data of the business logic execution and data modification operation of the first operation;

[0025] Performing feature extraction on the log data to obtain a first feature vector;

[0026] Inputting the first feature vector into an abnormality analysis model to determine abnormal factors;

[0027] Based on the abnormal factor, the business logic execution and / or data modification operation of the first operation is adjusted.

[0028] Furthermore, the method further comprises:

[0029] Before writing the message to be sent related to the first operation into the local message table,

[0030] Determine, based at least on the business logic execution of the first service and the corresponding data modification operation, a first dynamic parameter for configuring the message middleware and a second dynamic parameter for configuring the local message table; wherein the first dynamic parameter includes a connection parameter, a message queue configuration parameter, a message sending strategy, and a message persistence parameter, and the second dynamic parameter includes a table structure parameter, a message cleanup strategy, a scanning strategy, and a message status management strategy;

[0031] The configurations of the message middleware and the local message table are adjusted based on the first dynamic parameter and the second dynamic parameter.

[0032] Further, the determining, based at least on the business logic execution of the first service and the corresponding data modification operation, a first dynamic parameter for configuring the message middleware and a second dynamic parameter for configuring the local message table includes:

[0033] Obtain historical abnormal factors that caused operation abnormalities during historical service execution;

[0034] Determine the first dynamic parameter and the second dynamic parameter by using a parameter configuration model based on the business logic execution of the first service and the corresponding data modification operation and the abnormal factor;

[0035] The parameter configuration model is a deep network based on reinforcement learning, and the deep network based on reinforcement learning is trained in the following way:

[0036] Determine a state space and an action space for training a deep network; wherein the state space includes historical abnormal factors, business logic execution, data modification operations, and current configuration states, and the action space includes multiple parameter configuration strategies;

[0037] estimating system performance corresponding to the plurality of parameter configuration strategies, and providing reward or penalty feedback based on the estimated system performance;

[0038] Based on the reward or penalty, the parameters of the parameter configuration model to be trained are adjusted with the goal of minimizing the loss function.

[0039] In a second aspect, the present application provides a microservice data consistency processing device, through which a first service can be run, and the device includes:

[0040] A first operation execution module, configured to execute a first operation, wherein the first operation includes a business logic execution of a first service and a corresponding data modification operation;

[0041] a message writing module, configured to write a message to be sent related to the first operation into a local message table after the first operation is completed, wherein the message includes an execution result of the first operation and relevant information that needs to be notified to the second service;

[0042] A message sending module, configured to send the message to a message middleware after writing the message to the local message table, so that the second service can receive a notification of successful execution of the first operation by subscribing to the message sent to the message middleware, and execute a second operation of the second service based on the operation result of the first operation;

[0043] A judgment module, used to judge whether the message is successfully sent to the message middleware;

[0044] The scanning module is used to periodically scan the unsent messages in the local message table through a background task when the message is not sent to the message middleware, and resend the unsent messages to the message middleware until the message is sent successfully.

[0045] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement the steps of the microservice data consistency processing method described in any one of the embodiments of the first aspect when executing the program stored in the memory.

[0048] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the microservice data consistency processing method as described in any embodiment of the first aspect are implemented.

[0049] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the above technical solution records the execution result of the first operation in the local message table and sends the relevant message to the message middleware, thereby ensuring that the message will not be lost or delayed after the first operation is successfully executed. At the same time, if the message is not successfully sent to the message middleware, the unsent messages in the local message table will be scanned regularly through the background task to ensure that the message will eventually arrive, thereby improving the reliability of message delivery. For example, in the case of network jitter or temporary system failure, it can also ensure that the message will not be lost, thereby improving the robustness of the system. It can effectively avoid the loss of messages in abnormal situations such as network interruption and service restart, and maintain the consistency of cross-service operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 A flowchart of a microservice data consistency processing method provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of another microservice data consistency processing method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the process of configuring service parameters provided in the embodiment of the present application;

[0055] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0057] Figure 1A flowchart of a microservice data consistency processing method provided in an embodiment of the present application. In some embodiments, Figure 1 The process shown may be executed by the electronic device (eg, processor 111 ) or a first service running on the electronic device.

[0058] Step 101: execute a first operation, where the first operation includes executing the business logic of a first service and a corresponding data modification operation.

[0059] The first operation refers to the core business logic and related data operations performed by the microservice in the microservice architecture. For example, the microservice may be a first service, specifically, the first service may be a user-related service, and the first operation may be creating a user.

[0060] Business logic execution refers to the core calculation or business processing steps in the microservice. In some embodiments, business logic execution may include reading data, modifying data, deleting data, processing orders, updating inventory, etc.

[0061] Data modification operations refer to operations such as adding, deleting, and modifying data on the database after the business logic is executed. For example, updating the status of an order or creating a new record.

[0062] Step 102: after the first operation is completed, a message to be sent related to the first operation is written into a local message table, where the message includes an execution result of the first operation and relevant information that needs to be notified to the second service.

[0063] The local message table is a special table in the database, which is used to record and store messages related to the current first service execution operation. The local message table can ensure that after the first service successfully executes the first operation, the related messages will not be lost.

[0064] In some embodiments, the local message table may include the following information: message content, sending status, creation time, and update time, etc.

[0065] The message content includes the execution result of the first operation and detailed information to be notified to other services.

[0066] The sending status is used to indicate whether the message has been successfully sent to the message middleware.

[0067] The creation time and update time are used to record the generation time of the message, which is convenient for subsequent message management and regular scanning.

[0068] For example, the relevant message that needs to be notified to the second service may include the execution result of the first operation, the creation time and update time of the execution content corresponding to the first operation, such as the user creation time and update time.

[0069] Step 103, after writing the message to the local message table, sends the message to the message middleware, so that the second service can receive the notification of the successful execution of the first operation by subscribing to the message sent to the message middleware, and execute the second operation of the second service based on the operation result of the first operation.

[0070] Message middleware is a tool that can be used for asynchronous message delivery and is responsible for delivering messages in a microservice architecture. In some embodiments, message middleware may include Kafka, RabbitMQ, ActiveMQ, etc. Message middleware can help decouple services and use asynchronous message delivery to enable services to communicate without direct calls. Message middleware is responsible for reliably sending messages from producers (such as the first service) to consumers (such as the second service) and can ensure the order and persistence of messages.

[0071] In some embodiments, writing a message to the local message table is to insert the execution result of the first operation and the related information that needs to be notified to the second service into the local message table after the first operation is executed. The message content may include the success or failure result of the operation, and the detailed information to be notified to the second service, such as specific data changes, updated status, etc.

[0072] In some embodiments, after writing the message to be sent into the local message table, the first service may asynchronously send the message to the message middleware.

[0073] The second service subscribes to the message means that the second service receives the message sent by the first service by pre-subscribing to a specific message queue or topic in the message middleware.

[0074] When the second service receives the message from the message middleware, it means that the second service has received the notification that the first operation is successfully executed, and can perform corresponding subsequent business processing accordingly.

[0075] The second operation refers to the execution of the second service's own business logic and operations based on the result of the first operation after receiving the notification message from the first service. For example, if the first service updates the order status, the second service may handle the shipment or notify the user. For another example, if the first service creates a user, the second service may send a coupon to the user based on the result of the first service creating the user.

[0076] Step 104: determine whether the message is successfully sent to the message middleware.

[0077] Determining whether the message is sent successfully means that after sending the message to the message middleware, the first service will perform a status check to confirm whether the message is successfully received. If the message middleware does not confirm that the message has been received, the first service needs to do corresponding processing.

[0078] In some embodiments, whether the message is successfully sent to the message middleware can be confirmed through the response of the message middleware or the local log or monitoring system.

[0079] If it is determined that the message is not sent to the message middleware successfully, step 105 may be executed. If it is successful, the first service may end the current operation.

[0080] Step 105: Periodically scan the unsent messages in the local message table through a background task, and resend the unsent messages to the message middleware until the messages are successfully sent.

[0081] The background task is a scheduled task running inside the first service, which is used to scan the local message table for messages that have not been successfully sent to the message middleware. This task can be executed regularly, for example, every few seconds or minutes, to check the message table for messages that have not been sent or failed to be sent.

[0082] Unsent messages refer to messages that are not sent to the message middleware in a timely manner due to network problems or temporary unavailability of the middleware.

[0083] Resending means that when the background task scans unsent messages, it attempts to send them to the message middleware again. This process can continue until the message is successfully sent.

[0084] In this embodiment, through the mechanism of periodic background scanning and resending of messages, repeated attempts are made to ensure that the messages will eventually be sent successfully, thereby solving the problem of message loss that may be caused by network failures or temporary system errors, that is, ensuring message consistency between different services.

[0085] Successful message sending means that the message has been reliably delivered to the message middleware, and the second service can perform subsequent operations based on the message.

[0086] When the message is successfully sent, the status in the local message table can be updated to indicate that the message has been sent so that the background task does not need to process it repeatedly.

[0087] In this embodiment, by recording the execution result of the first operation in the local message table and sending the relevant message to the message middleware, it is ensured that the message will not be lost or delayed after the first operation is successfully executed. At the same time, if the message is not successfully sent to the message middleware, the unsent messages in the local message table will be scanned regularly through the background task to ensure that the message will eventually arrive. This mechanism improves the reliability of message delivery, especially in the case of network jitter or temporary system failure, it can ensure that the message will not be lost, thereby improving the robustness of the system. It can effectively avoid the loss of messages in abnormal situations such as network interruption and service restart, and maintain the consistency of cross-service operations.

[0088] In some embodiments of the present invention, based on the same inventive concept, the first service can also perform the following steps: Figure 2 The following operations are shown to better ensure the message consistency between microservices. Specifically, the following operations may be included.

[0089] Step 106: Receive the execution result of the second operation from the second service.

[0090] The execution result of the second operation refers to the feedback on whether the operation executed by the second service is successful or not. The result can be used to determine whether the data consistency between the two services meets expectations.

[0091] After the first service notifies the second service, the second service performs related operations based on the result of the first operation (such as updating status, processing orders, sending coupons, etc.), and feeds back its execution results to the first service so that the first service can make further judgments.

[0092] Step 107: Based on the execution result of the second operation, determine whether the first service and the second service meet a preset data consistency requirement.

[0093] Data consistency refers to whether the data state between the first service and the second service corresponds to the same preset service state. For example, if the first service updates the order status, the second service should also update it accordingly. For another example, the first service creates a user, and the second service sends a coupon to the created user account accordingly. If there is an inconsistency, it may mean that the operation failed or there is a logical error.

[0094] The preset data consistency requirement refers to the data consistency standard defined during design. Data consistency requirements can be defined based on business requirements, data models, or database constraints to ensure that data between services in a distributed environment remains consistent. It should be noted that data consistency does not mean that the data of two services must be the same.

[0095] In some embodiments, the first service determines whether the data between the two services meets the consistency requirement based on the execution result of the second operation. If not, it means that the data consistency requirement is not met, and further processing or re-execution of the operation may be required. For example, step 108 may be executed at this time.

[0096] Step 108: Detect the atomicity of the business logic execution and the data modification operation, and re-execute the first operation.

[0097] Atomicity is a basic property of transactions, ensuring that operations are either fully successful or completely rolled back, without partial success or failure. In this method, atomicity detection is used to ensure that there are no inconsistencies between the execution of business logic and data modification operations.

[0098] When the first service finds that the data with the second service is inconsistent, it needs to check the atomicity of its own business logic execution and data modification operations. In some embodiments, this process can be performed through a simulation environment to ensure that the system can accurately reproduce the operation and perform atomicity detection.

[0099] Exemplarily, detecting the atomicity of the business logic execution and the data modification operation may include the following operations.

[0100] S10: Obtain the basic business logic of the first service.

[0101] S11, based on the basic business logic, creating a simulation environment for simulating the execution of the first operation.

[0102] S12: In the simulation environment, using database transactions to detect the atomicity of the business logic execution and the data modification operation.

[0103] The basic business logic refers to the basic way and method of executing the first service.

[0104] The simulation environment is an isolated environment used to test and reproduce the operation of the first service. It usually simulates the database and service behavior in the real environment, but does not affect the production data. The simulation environment is part of the atomic detection and can help troubleshoot potential problems.

[0105] In the simulation environment, the system can reproduce the business logic and data operations of the first service to ensure the atomicity of the entire process. In addition, the simulation environment can effectively help developers find problems without affecting the actual production environment, by simulating the database and service behaviors in the real environment without affecting the production data.

[0106] In some embodiments, in the simulation environment, the atomicity of the business logic execution and the data modification operation is detected using database transactions, and the following operations may be performed.

[0107] S20, sequentially executing the business logic execution and data modification operation of the first operation in the simulation environment;

[0108] S21, in response to the business logic of the first operation and the data modification operation being successfully executed, prompting for the operation to be re-executed;

[0109] S22, in response to the failure of the business logic execution and / or data modification operation of the first operation, prompting an operation abnormality.

[0110] In this embodiment, by using database transactions, the system can detect whether the business logic and data modification operations of the first service can be executed as an atomic whole. If the execution is successful, it means that the operation is atomic; if it fails, it means that the operation is not atomic, and an operation abnormality may be prompted.

[0111] When business logic execution and data modification operations are successful in the simulation environment, the system can prompt the developer or operation and maintenance personnel to re-execute the operation to fix the data inconsistency problem.

[0112] If the operation in the simulation environment fails, the system may prompt that there is an exception, which means that there is a problem with the business logic or data modification operation of the first service, which requires further debugging or repair.

[0113] Exemplarily, when an abnormal operation is prompted, the system may perform the following operations.

[0114] S30, obtaining log data of the business logic execution and data modification operation of the first operation;

[0115] S31, extracting features from the log data to obtain a first feature vector;

[0116] S32, inputting the first feature vector into an abnormality analysis model to determine abnormal factors;

[0117] S33: Based on the abnormal factor, adjust the business logic execution and / or data modification operation of the first operation.

[0118] Log data refers to the detailed information of business logic execution and data modification operations. The log may contain operation steps, input and output data, execution time, error information, etc. Log data can be used for troubleshooting and system debugging.

[0119] When an operation is abnormal, the system will extract relevant data from the log, which can help developers or the system automatically analyze the problem.

[0120] Feature extraction is the process of extracting key information from log data, such as the time point when certain operations failed, the steps where abnormalities occurred, etc. This information can be used as input features for further analysis.

[0121] The anomaly analysis model is a model used to automatically analyze the causes of anomalies. It can be built based on machine learning, rule engines, or pattern recognition technologies. The model can determine the specific factors that may cause anomalies based on the input feature vector.

[0122] For example, the system can extract features from log data and input the extracted features into an anomaly analysis model to determine the cause of the anomaly. The anomaly analysis model can quickly locate and solve problems, avoid repeated manual troubleshooting, and improve problem solving efficiency.

[0123] When the anomaly analysis model is a machine learning model, the model can be trained based on a large amount of historical data to obtain the anomaly analysis model.

[0124] When the abnormal factors that cause the problem are analyzed and determined, the system can adjust the corresponding business logic execution or data modification operation. The abnormal factors are the reasons that cause the abnormality, which can be one or more reasons.

[0125] Adjusting business logic execution means optimizing or correcting business logic based on the results of exception analysis. For example, if a step is prone to errors in concurrent situations, you may need to add a lock or adjust the logic order.

[0126] Similarly, data modification operations may also need to be optimized, for example, by adding data validation, adjusting transaction isolation levels, or changing the way data is updated.

[0127] In this embodiment, based on the results of the exception analysis, the system can automatically optimize the business logic or data operations based on the obtained exception factors to ensure that the same problem does not occur during the next execution, while reducing manual participation and saving labor costs.

[0128] Based on the same inventive concept, in order to better solve the problem of consistency of microservice data processing, the microservice data processing method provided in some embodiments of this specification may also include the following: Figure 3 The following operations are shown.

[0129] Specifically, before writing the to-be-sent message related to the first operation into the local message table, the first service may perform the following steps 301 and 302 .

[0130] Step 301 : determining a first dynamic parameter for configuring the message middleware and a second dynamic parameter for configuring the local message table based at least on the business logic execution of the first service and a corresponding data modification operation.

[0131] The first dynamic parameters include connection parameters, message queue configuration parameters, message sending strategies and message persistence parameters, and the second dynamic parameters include table structure parameters, message cleanup strategies, scanning strategies and message status management strategies.

[0132] The first dynamic parameters refer to the configuration parameters related to the message middleware, including connection parameters, message queue configuration parameters, message sending strategy and message persistence parameters. These parameters can affect the transmission mode, speed and reliability of messages in the message middleware.

[0133] Connection parameters refer to the connection settings between the microservice and the message middleware, such as connection timeout, number of retries, etc.

[0134] Message queue configuration parameters include queue size, priority, partition strategy, etc., which are used to manage how messages are queued and processed in the middleware.

[0135] Message sending policy refers to the priority, sequence, batch processing mode, etc. of message sending, which determines when and how the message is sent to the recipient.

[0136] The message persistence parameters can be used to configure whether messages need to be persistently stored to ensure that messages are not lost after a middleware crash or network interruption.

[0137] The second dynamic parameters refer to configuration parameters for the local message table, including table structure, message cleanup strategy, scanning strategy and message status management strategy.

[0138] Table structure parameters include table fields, index settings, etc., which determine how to store messages to be sent.

[0139] The message cleanup policy determines when messages are cleaned up from the local message table after being sent to avoid excessive table expansion.

[0140] The scanning policy controls when the system scans the unsent messages in the local message table, which can be based on a time interval or an event trigger.

[0141] The message status management strategy determines how to record and update the message status, such as whether the message has been sent, whether the resend is successful, etc.

[0142] Step 302: Adjust the configuration of the message middleware and the local message table based on the first dynamic parameter and the second dynamic parameter.

[0143] In this embodiment, when the first service executes business logic and data modification operations, the system can ensure the efficiency and reliability of the operations by dynamically adjusting the parameter configurations of the message middleware and the local message table.

[0144] In some embodiments, determining the first dynamic parameter for configuring the message middleware and the second dynamic parameter for configuring the local message table based at least on the business logic execution of the first service and the corresponding data modification operation may include the following operations.

[0145] S40, obtaining historical abnormal factors that caused abnormal operations during historical service execution;

[0146] S41, based on the business logic execution of the first service and the corresponding data modification operation and the abnormal factor, determine the first dynamic parameter and the second dynamic parameter using a parameter configuration model.

[0147] The parameter configuration model refers to a model for determining the first dynamic parameter and the second dynamic parameter, which can optimize the configuration by using historical abnormal data, business logic and data modification operations. In some embodiments, the parameter configuration can be trained using a deep network method of reinforcement learning.

[0148] Deep network based on reinforcement learning means that through continuous training and optimization, the model can make the best decision according to different states. In the present invention, the parameter configuration model dynamically selects the best parameter configuration strategy according to historical anomalies and current business status. Through the reinforcement learning model, the system can select appropriate parameter configuration according to real-time business logic and historical anomalies to ensure the efficiency and stability of the system.

[0149] In some embodiments, the parameter configuration model is a deep network based on reinforcement learning, and the deep network based on reinforcement learning is trained in the following manner:

[0150] S50, determining the state space and action space for training the deep network; wherein the state space includes historical abnormal factors, business logic execution, data modification operations and current configuration status, and the action space includes multiple parameter configuration strategies.

[0151] S51, estimating system performance corresponding to the multiple parameter configuration strategies, and feeding back rewards or penalties based on the estimated system performance.

[0152] S52: Based on the reward or penalty, adjust the parameters of the parameter configuration model to be trained with the goal of minimizing the loss function.

[0153] The state space is a data set used to train the model, which represents the current state of the system. In the present invention, the state space includes historical abnormal factors, current business logic execution status, data modification operation status, and current system configuration status.

[0154] The action space refers to the set of actions that the system can take. In this scenario, the action space includes multiple possible parameter configuration strategies. The model affects the system performance by selecting different actions (i.e. different configuration strategies).

[0155] Reinforcement learning evaluates the quality of a given strategy by the results of its execution. If the strategy improves the performance of the system (such as reducing message delays or avoiding anomalies), a positive reward is given; if it causes performance degradation, a penalty is given. Based on these rewards or penalties, the system can continuously optimize the parameter configuration model.

[0156] The loss function is the goal of model optimization, and minimizing the loss function refers to the error or loss that the system hopes to minimize. In reinforcement learning, the loss function is related to rewards or penalties, and unnecessary losses are reduced by adjusting the parameters of the model.

[0157] In this embodiment, the system continuously optimizes the parameter configuration model based on the feedback of the state space and action space by training the reinforcement learning model to adapt to the dynamic changes in different business scenarios. This further expands the dynamic parameter configuration method of the message middleware and local message table in the microservice system.

[0158] like Figure 4 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0159] Memory 113, used for storing computer programs;

[0160] In one embodiment of the present application, the processor 111 is used to implement the microservice data consistency processing method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113, including:

[0161] Execute a first operation, where the first operation includes executing the business logic of the first service and a corresponding data modification operation;

[0162] After the first operation is completed, writing a message to be sent related to the first operation into the local message table, where the message includes the execution result of the first operation and relevant information that needs to be notified to the second service;

[0163] After writing the message to the local message table, sending the message to the message middleware, so that the second service can receive a notification of successful execution of the first operation by subscribing to the message sent to the message middleware, and execute a second operation of the second service based on the operation result of the first operation;

[0164] Determine whether the message is successfully sent to the message middleware;

[0165] If not, the unsent messages in the local message table are scanned periodically by a background task, and the unsent messages are resent to the message middleware until the messages are sent successfully.

[0166] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the microservice data consistency processing method provided in any of the aforementioned method embodiments are implemented.

[0167] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0168] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A microservice data consistency processing method, characterized in that: The method comprises: Execute a first operation, where the first operation includes executing the business logic of the first service and a corresponding data modification operation; After the first operation is completed, writing a message to be sent related to the first operation into the local message table, where the message includes the execution result of the first operation and relevant information that needs to be notified to the second service; After writing the message to the local message table, sending the message to the message middleware, so that the second service can receive a notification of successful execution of the first operation by subscribing to the message sent to the message middleware, and execute a second operation of the second service based on the operation result of the first operation; Determine whether the message is successfully sent to the message middleware; If not, the unsent messages in the local message table are scanned periodically by a background task, and the unsent messages are resent to the message middleware until the messages are sent successfully.

2. The method according to claim 1, characterized in that The method further comprises: receiving an execution result of the second operation from the second service; Based on the execution result of the second operation, determining whether the first service and the second service meet a preset data consistency requirement; If not, check the atomicity of the business logic execution and the data modification operation, and re-execute the first operation.

3. The method according to claim 2, characterized in that The detecting the atomicity of the business logic execution and the data modification operation includes: Obtaining the basic business logic of the first service; Based on the basic business logic, creating a simulation environment for simulating execution of the first operation; In the simulation environment, database transactions are used to detect the atomicity of the business logic execution and the data modification operation.

4. The method according to claim 3, characterized in that The step of using a database transaction to detect the atomicity of the business logic execution and the data modification operation in the simulation environment includes: Sequentially executing the business logic execution and data modification operation of the first operation in the simulation environment; In response to the business logic of the first operation and the data modification operation being successfully executed, prompting for the operation to be re-executed; In response to the failure of the business logic execution and / or data modification operation execution of the first operation, an operation abnormality is prompted.

5. The method according to claim 4, characterized in that When the operation is abnormal, the method further includes: Obtaining log data of the business logic execution and data modification operation of the first operation; Performing feature extraction on the log data to obtain a first feature vector; Inputting the first feature vector into an abnormality analysis model to determine abnormal factors; Based on the abnormal factor, the business logic execution and / or data modification operation of the first operation is adjusted.

6. The method according to claim 1, characterized in that The method further comprises: Before writing the message to be sent related to the first operation into the local message table, Determine, based at least on the business logic execution of the first service and the corresponding data modification operation, a first dynamic parameter for configuring the message middleware and a second dynamic parameter for configuring the local message table; wherein the first dynamic parameter includes a connection parameter, a message queue configuration parameter, a message sending strategy, and a message persistence parameter, and the second dynamic parameter includes a table structure parameter, a message cleanup strategy, a scanning strategy, and a message status management strategy; The configurations of the message middleware and the local message table are adjusted based on the first dynamic parameter and the second dynamic parameter.

7. The method according to claim 6, characterized in that The determining, based at least on the business logic execution of the first service and the corresponding data modification operation, a first dynamic parameter for configuring the message middleware and a second dynamic parameter for configuring the local message table, comprises: Obtain historical abnormal factors that caused operation abnormalities during historical service execution; Determine the first dynamic parameter and the second dynamic parameter by using a parameter configuration model based on the business logic execution of the first service and the corresponding data modification operation and the abnormal factor; The parameter configuration model is a deep network based on reinforcement learning, and the deep network based on reinforcement learning is trained in the following way: Determine a state space and an action space for training a deep network; wherein the state space includes historical abnormal factors, business logic execution, data modification operations, and current configuration states, and the action space includes multiple parameter configuration strategies; estimating system performance corresponding to the plurality of parameter configuration strategies, and providing reward or penalty feedback based on the estimated system performance; Based on the reward or penalty, the parameters of the parameter configuration model to be trained are adjusted with the goal of minimizing the loss function.

8. A microservice data consistency processing device, characterized in that: The device comprises: A first operation execution module, configured to execute a first operation, wherein the first operation includes a business logic execution of a first service and a corresponding data modification operation; a message writing module, configured to write a message to be sent related to the first operation into a local message table after the first operation is completed, wherein the message includes an execution result of the first operation and relevant information that needs to be notified to the second service; A message sending module, configured to send the message to a message middleware after writing the message to the local message table, so that the second service can receive a notification of successful execution of the first operation by subscribing to the message sent to the message middleware, and execute a second operation of the second service based on the operation result of the first operation; A judgment module, used to judge whether the message is successfully sent to the message middleware; The scanning module is used to periodically scan the unsent messages in the local message table through a background task when the message is not sent to the message middleware, and resend the unsent messages to the message middleware until the message is sent successfully.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the microservice data processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the microservice data processing method according to any one of claims 1 to 7 are implemented.

Images