Data processing method and device, electronic equipment and storage medium
Patent Information
- Application Number
- CN202310528679.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
The version number mechanism of existing storage systems is prone to expand the scope of write conflicts, affecting the stability of the microservice system.
By grouping management of each field in the business data entry, parallel writing is used to process update data of different target fields, and data writing is performed based on their respective version information, avoiding locking in the business data entry unit and refinement of the data Write granularity.
Reduce the scope of write conflicts, improve the stability and data writing efficiency of the microservice system.
Smart Images

Figure CN118939173A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] Microservice is a software development architecture and organizational method. Compared with the overall architecture, the business execution unit under the microservice architecture focuses on a single function and exists in the form of providing a single service to the outside world. Compared with the overall architecture, the execution unit under the microservice architecture is fine-grained, small in size, but with many categories. Microservice architecture usually has multiple writes. The prior art generally adopts the version number mechanism of the storage system, that is, the storage system performs version management on each data entry. Each data entry has a version number. When the business process reads data, the read data will carry the version number. When a data modification write-back operation occurs, the data write request will carry the version number. When the storage system performs a write-back operation, it compares the version number in the write request and the version number in the storage system. When the written version number is lower than the storage system version number, the write is rejected, otherwise the write is accepted and the version number is upgraded. The version number mechanism of the existing storage system is easy to expand the scope of write conflicts, thereby affecting the stability of the service system. Summary of the invention
[0003] The technical problem to be solved by the present application is to provide a data processing method, device, electronic device and storage medium, which can realize group management of each field in the business data entry, reduce the scope of write conflicts, and improve the stability of the microservice system.
[0004] In order to solve the above technical problems, on the one hand, the present application provides a data processing method, including:
[0005] In response to a concurrent processing request for a target business data entry; the concurrent processing request includes a plurality of data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries update data of the at least one target field and first version information of the at least one target field;
[0006] When the target fields targeted by the multiple data write requests are different, based on the first version information corresponding to the different target fields, the update data of the different target fields are written in parallel to obtain the data writing results corresponding to the different target fields;
[0007] Based on the data writing results corresponding to the different target fields, the data writing result for the target business data entry is determined.
[0008] On the other hand, the present application provides a data processing device, comprising:
[0009] A concurrent request response module, configured to respond to concurrent processing requests for target business data entries; the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries update data of the at least one target field and first version information of the at least one target field;
[0010] A parallel writing module, configured to, when the target fields targeted by the multiple data writing requests are different, perform parallel writing processing on the update data of the different target fields based on the first version information corresponding to the different target fields, so as to obtain data writing results corresponding to the different target fields;
[0011] The first determination module is used to determine the data writing result of the target business data entry based on the data writing results corresponding to the different target fields.
[0012] On the other hand, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method as described above.
[0013] On the other hand, the present application provides a computer storage medium, wherein the storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by a processor and executes the data processing method as described above.
[0014] Implementing the embodiments of the present application has the following beneficial effects:
[0015] The present application can respond to concurrent processing requests for target business data entries, and the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; when the target fields for the multiple data write requests are different, that is, when the target fields for different data write requests are different, the different target fields are processed in parallel, so that the data write results corresponding to the different target fields can be obtained, that is, the different target fields are managed in groups, so that the write operations for the different target fields are independent of each other, avoiding the problem of locking the business data entry as a unit when writing data based on the version number mechanism of the storage system in the prior art, which easily expands the scope of write conflicts. The present application can write data in units of fields, refine the data write granularity, reduce the scope of write conflicts, and thus improve the stability of the service system. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 It is a schematic diagram of the implementation environment provided by the embodiment of the present application;
[0018] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application;
[0019] Figure 3 This is a flow chart of a method for writing field data provided by an embodiment of the present application;
[0020] Figure 4 It is a flow chart of a data processing method based on a full write request provided in an embodiment of the present application;
[0021] Figure 5 is a schematic diagram of the data structure corresponding to the business data entry provided in the embodiment of the present application;
[0022] Figure 6 is a flow chart of a node replacement method provided in an embodiment of the present application;
[0023] Figure 7 This is a schematic diagram of the data migration process during the node replacement process provided by an embodiment of the present application;
[0024] Figure 8 is a schematic diagram of a data processing device provided in an embodiment of the present application;
[0025] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] To make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] First, the relevant terms involved in the embodiments of this specification are explained as follows:
[0029] RPC: remote process call, remote procedure call, refers to the ability, through architectural design, to allow a business unit to call functions of other local machines or processes just like calling local functions.
[0030] Coroutine: An execution unit with an independent stack and context. Unlike processes and threads, coroutines switch contexts in user mode.
[0031] Version number: The so-called version number refers to the number of a data, which is used to identify the version information of the current data. It is generally single-incremental and can be used to identify the newness of the data.
[0032] See also Figure 1 , which shows a schematic diagram of the implementation environment provided by an embodiment of the present application. The implementation environment may include at least a microservice cluster 110 and a request terminal 120. The microservice cluster 110 may include a control node, a data processing node, a storage node, etc. The microservice cluster 110 and the request terminal 120 can communicate data through a network.
[0033] Specifically, the control node can be used to manage each node in the microservice cluster 110; the data processing node can be used to process the business service request sent by the request terminal and return the processing result to the request terminal 120. The business service request can specifically include a data write request, a data read request, etc.; the storage node can be used to store the business data generated during the operation of the microservice cluster. The request terminal 120 can send a business service request to the microservice cluster and receive the processing result returned by the microservice cluster 110.
[0034] The request terminal 120 can communicate with the microservice cluster 110 based on the browser / server mode (B / S) or the client / server mode (C / S). The request terminal 120 may include: physical devices such as smart phones, tablet computers, laptops, digital assistants, smart wearable devices, vehicle terminals, servers, etc., and may also include software running in physical devices, such as applications, etc. The operating system running on the request terminal 120 in the embodiment of the present application may include but is not limited to Android system, IOS system, Linux, Windows, etc.
[0035] The microservice cluster 110 and the request terminal 120 may establish a communication connection via wired or wireless communication. Each node in the microservice cluster 110 may be a physical server or a cloud server.
[0036] In order to solve the problem that the version number mechanism of the storage system in the prior art easily expands the scope of write conflicts, thereby affecting the stability of the service system, the embodiment of the present application provides a data processing method, the execution subject of which can be the data processing node in the above-mentioned microservice cluster. Figure 2 , the method may include:
[0037] S210. Respond to concurrent processing requests for target business data entries; the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries updated data of the at least one target field and first version information of the at least one target field.
[0038] The business data in this embodiment may be data in a game business scenario, or data in a live broadcast business scenario, or data in an e-commerce business scenario, etc. In these business scenarios, the business data structure is complex, and a single business data entry may include data of multiple field contents; and in these business scenarios, the concurrency and interaction logic are complex, and multi-point writing is prone to occur, which may lead to write conflict problems.
[0039] In an example, the multiple data write requests included in the concurrent processing request may be data write requests that arrive at the data processing node at the same time, and the data processing node may respond to the multiple concurrent data write requests at the same time.
[0040] In another example, in view of the fact that there are usually multiple writes in the microservice architecture, the data write path can be converged to converge the data operation permissions for the same business data entry to the same data processing node. The data processing node that undertakes the data operation can generate a corresponding message queue for each business data entry. The message queue can cache data write requests for the business data entry. After the current data write request is completed, the next data write request can be executed to achieve the purpose of serial execution of the write operation. Therefore, for the data processing node that executes the data processing method in this embodiment, any data processing node can be used to process business service requests for at least one business data entry, and the business service request may include a data write request, a data read request, etc. For example, data processing node A has data operation permissions for business data item a. Therefore, when the microservice cluster receives a business service request for business data item a, the business service request for business data item a can be routed to data processing node A to converge the business service request for business data item a to data processing node A. In addition, data processing node A can cache the business service request for business data item a to message queue a' to implement serial processing of the business service requests in the message queue.
[0041] Further, the business service request for the same business data item may come from different types of request nodes, and the business service requests sent by different types of request nodes may be cached in different message queues; specifically, for multiple data write requests included in the concurrent processing request, the multiple data write requests may come from multiple message queues, and the multiple message queues are message queues corresponding to different types of request nodes; thus, in the present application, before responding to the concurrent processing request for the target business data item, the method further includes:
[0042] receiving data writing requests for the target business data entry sent by multiple types of request nodes;
[0043] The data write request sent by any type of request node is cached in a message queue corresponding to the request node of any type.
[0044] In this embodiment, multiple types of request nodes can be the above-mentioned request terminals, and different request terminals can be used as different types of request nodes. Different types of request nodes can specifically be request terminals of different user levels, so that message queues corresponding to request terminals of different user levels can exist in the data processing node. Multiple types of request nodes can include the above-mentioned request terminals and control terminals, wherein the request terminal is a terminal corresponding to a user who needs to operate business data, the control terminal can be a terminal independent of the microservice cluster and used to operate business data, and the control terminal can also be a control node in the microservice cluster, so that the data processing node can have a message queue corresponding to the request terminal and a message queue corresponding to the control terminal.
[0045] Specifically, when a data write request is received, the data write request may be cached in a corresponding message queue according to the type of the requesting node that sends the data write request.
[0046] Therefore, by caching the data write requests sent by different types of request nodes into the corresponding message queues, it is possible to achieve classified caching of data write requests, thereby facilitating the data processing nodes to process the data write requests in different message queues in parallel, thereby improving data writing efficiency.
[0047] For example, data processing node A can cache a business service request for business data item a to message queue a', and message queue a' may include message queue a1' corresponding to a first type of request node, message queue a2' corresponding to a second type of request node... When data processing node A receives a business service request from a first type of request node, it can cache the business service request to message queue a1'; when data processing node A receives a business service request from a second type of request node, it can cache the business service request to message queue a2'...
[0048] When multiple data write requests come from multiple message queues, that is, one data write request is taken from the head of each of the multiple message queues to form multiple concurrent data write requests. The time when these multiple data write requests enter their respective message queues can be the same or different.
[0049] Each data write request may be directed to one target field or multiple target fields in the target business data entry, and the data write request directed to multiple target fields in the target business data entry may include a data write request directed to all target fields in the target business data entry.
[0050] Each data write request carries the updated data of the target field and the first version information of the target field. The updated data of the target field is the data that needs to be written into the target field in the target business data entry. The first version information of the target field is the current storage version information of the target field when the field data of the target field was obtained last time. Therefore, when the field data of the target field is rewritten in the current request, the last storage version information can be carried to facilitate the determination of the data writing result based on the version information comparison result. The storage version information of the target field can be stored together with the field data of the target field, that is, each time the field data of the target field is updated, the storage version information of the target field can be updated at the same time, and the updated field data of the target field and the updated storage version information can be stored.
[0051] S220. When the target fields targeted by the multiple data write requests are different, based on the first version information corresponding to the different target fields, the update data of the different target fields are written in parallel to obtain data writing results corresponding to the different target fields.
[0052] The existence of different target fields targeted by multiple data write requests may mean that the target fields targeted by at least two data write requests are different, for example, data write request 1 is targeted at target field 1, and data write request 2 is targeted at target field 2, so that target field 1 is different from target field 2, that is, different target fields include target field 1 and target field 2. In this embodiment, since different target fields are managed in groups, the write operations on different target fields are independent of each other, and the update data of different target fields can be written in parallel, that is, the field data of different target fields of the target business data item are updated in parallel based on the first version information of different target fields, and the data write results corresponding to the different target fields are obtained.
[0053] In one example, when the target fields targeted by the multiple data write requests are the same, the target fields targeted by the multiple data write requests are the same, which means that the target fields targeted by at least two data write requests are the same, for example, data write request 3 is targeted at target field 3, and data write request 4 is targeted at target field 3, so that target field 3 is the same target field. If multiple data write requests arrive at the data processing node at the same time, the processing order of the data write requests for the same target field can be randomly determined to achieve field data writing for the same target field. If multiple data write requests are from multiple message queues, and the multiple message queues are message queues corresponding to different types of request nodes, the processing order of the corresponding data write requests can be determined based on the request node type corresponding to the message queue where the data write request for the same target field is located; for example, for a message queue with a high priority of the request node type, the priority of the data write request therein is high, and the corresponding processing order is determined to be ahead; for a message queue with a low priority of the request node type, the priority of the data write request therein is low, and the corresponding processing order is determined to be behind. In the case where multiple data write requests for the same target field are determined, at least two data write requests can be processed based on the processing order to obtain data write results corresponding to the same target field.
[0054] S230. Determine the data writing result for the target business data entry based on the data writing results corresponding to the different target fields.
[0055] The data writing results corresponding to different target fields may include write success and write failure. Write success may refer to a successful write once or after multiple retries. Write failure may refer to request timeout or network anomaly, etc. Then, based on the determination of the data writing results for different target fields, the data writing results for the target business data item may be determined. The data writing results for the target business data item may represent the target field that is written successfully, the field data of the target field that is written successfully, and the target field that is written failed.
[0056] Furthermore, the data writing results for the target business data entries can be determined based on the data writing results corresponding to different target fields and the data writing results corresponding to the same target fields. The data writing results corresponding to the same target fields are similar to the data writing results of different target fields and will not be repeated here.
[0057] The present application can respond to concurrent processing requests for target business data entries, and the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; when the target fields for the multiple data write requests are different, that is, when the target fields for different data write requests are different, the different target fields are processed in parallel, so that the data write results corresponding to the different target fields can be obtained, that is, the different target fields are managed in groups, so that the write operations for the different target fields are independent of each other, avoiding the problem of locking the business data entry as a unit when writing data based on the version number mechanism of the storage system in the prior art, which easily expands the scope of write conflicts. The present application can write data in units of fields, refine the data write granularity, reduce the scope of write conflicts, and thus improve the stability of the service system.
[0058] Further, according to the above content, it can be known that the data writing request carries the first version information of the target field, and the first version information can be used for version comparison when writing field data; please refer to Figure 3 , which shows a method for writing field data, which may include:
[0059] S310. Perform parallel matching on the first version information of each target field in the different target fields and the currently stored version information of each target field to obtain a first matching result corresponding to each target field.
[0060] When processing each data write request, the first version information of the target field targeted in the data write request is matched with the currently stored version information. The currently stored version information may refer to the version information determined when the field information of the target field was last updated most recently to the current time. The first version information of the target field targeted in the data write request may be consistent with the currently stored version information, or the first version information may be lower than the currently stored version information.
[0061] When processing data write requests for different target fields, the first version information of each target field in the different target fields can first be matched in parallel with the corresponding currently stored version information, that is, the version information matching operations corresponding to different target fields are implemented in parallel, and then the first matching result corresponding to each target field can be obtained.
[0062] For each target field, the first matching result may be that the first version information of the target field is consistent with the currently stored version information of the target field, or that the first version information of the target field is lower than the currently stored version information of the target field.
[0063] S320. Write the updated data of the first field into the target business data entry to obtain a data writing result corresponding to the first field; the first field is at least one target field among the different target fields, and the first matching result of the first field indicates that the first version information of the first field is consistent with the currently stored version information of the first field.
[0064] If the first version information of the target field targeted by any data write request is consistent with the current storage version information of the target field, it means that during the initiation of the data write request, the field data of the target field has not been updated, that is, there is no data write operation on the target field. In this case, the updated data of the target field can be directly written into the target field. It should be noted that when writing the updated data of the target field into the target field, the updated field data of the target field can be determined based on the current field data and the updated data of the target field, that is, based on the current field data, the current field data is updated based on the updated data of the target field to obtain the updated field data of the target field; the updated data of the target field can represent the data of the field data change amount information, so that the field data update operation based on the updated data can be an increase, decrease or reset of the field data.
[0065] S330. Retry the data write request for the second field, and determine the data write result corresponding to the second field based on the retry result; the second field is at least one target field among the different target fields, and the first matching result of the second field indicates that the first version information of the second field is lower than the current storage version information of the second field.
[0066] If the first version information of the target field targeted by any data write request is lower than the current stored version information of the target field, it means that the field data of the target field has been updated during the initiation of the data write request. In this case, the data write operation needs to be based on the latest field data. If the first version information is lower than the current stored version information of the target field, it means that the field data on which the data write request is based has expired, so that the data write request can be retried, that is, the field data of the target field and the current stored version information are reread. The retry request can carry the reread field data of the target field and the current stored version information to perform subsequent data write operations.
[0067] The retry result may include a retry success or a retry failure. A retry success may be a retry success within a preset number of times, and a retry failure may be a write failure after retrying the preset number of times, or a network failure, etc. Therefore, the data write result may be further determined based on the retry result. If the retry result is a retry success, the corresponding data write result may be a data write success to the second field; if the retry result is a retry failure, the corresponding data write result may be a data write failure to the second field.
[0068] When the field data of the target field is written successfully, the current storage version information of each pair of target fields is upgraded to obtain the upgraded storage version information.
[0069] Therefore, when updating the field data of each target field, it can be implemented based on the field version number information, so that when there are concurrent write requests for the target field, the field data of the target field can be written in order, reducing the probability of write conflicts.
[0070] The above-mentioned concurrent processing requests may also specifically include full write requests, which are for the full target fields in the target business data entry, that is, the full write request can be used to rewrite the field data of the full target fields in the target business data entry; please refer to Figure 4 , which shows a data processing method based on a full write request, the method may include:
[0071] S410. Respond to concurrent processing requests for target business data entries; the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries updated data of the at least one target field and first version information of the at least one target field.
[0072] The multiple data write requests include a full write request, because the full target field in the target business data entry targeted by the full write request, that is, the target field targeted by the full write request is the same as the target field targeted by other data write requests in the multiple data write requests. Therefore, in the case where the multiple data write requests include the full write request, the specific processing method for the multiple data write requests in the case where the target fields targeted by the multiple data write requests are the same as those in the above step S220 can be referred to.
[0073] For other implementation details of this step, please refer to the above step S210, which will not be repeated here.
[0074] S420. When writing data to the full target field in the target business data entry based on the full write request, update the current storage version information of the target business data entry to obtain the updated current storage version information of the target business data entry.
[0075] In this embodiment, the version information of the business data entry and the version information of the target field in the business data entry can be hierarchically managed. Specifically, the business data entry can correspond to the parent version number information, and the target field can correspond to the child version number information. The parent version number information is used to manage the version of the business data entry, and the child version number information is used to manage the version of the target field. The target fields in the business data entry are grouped and managed, and each target field has independent version number information.
[0076] When writing the field data of all target fields in the target business data entry, the currently stored version information of the target business data entry may be updated to obtain updated parent version number information.
[0077] S430. Update the current storage version information of each target field in the target business data entry to obtain the updated current storage version information of each target field.
[0078] Accordingly, since the full write request realizes the rewriting of the field data of each target field in the target business data entry, the current storage version information of each target field can be updated to obtain the updated storage version information of each target field.
[0079] Furthermore, when the field data of any target field in the target business data entry is updated, the corresponding target business data entry is updated, and the parent version number corresponding to the corresponding target business data entry may also be updated accordingly.
[0080] Please refer to Figure 5 , which shows a schematic diagram of the data structure corresponding to the business data entry, wherein the parent version number information of the business data entry is stored in the header of the business data entry, followed by the version number information of target field 1, the field data of target field 1, the version number information of target field 2, the field data of target field 2...the version number information of target field n, the field data of target field n.
[0081] When all target fields in the target business data entry are updated based on a full write request, the storage version number information of the target business data entry and the storage version number information of each target field can be updated synchronously to achieve consistent update of the version number information, thereby facilitating subsequent data write requests to perform field data write operations based on the storage version number information.
[0082] In this embodiment, when specifically processing a data write request, if the data write request is for a specified target field, it can be determined based on the current storage version information of the specified target field whether the data write condition is met, that is, whether the version number information of the specified target field carried in the data write request is consistent with the current storage version information of the specified target field. If they are consistent, it is determined that the data write condition is met, and a data write operation is performed on the specified target field, and the current storage version information of the specified target field is updated. When specifically processing a data write request, if the data write request is for a target business data entry, that is, a full write request, it can be determined based on the current storage version information of the target business data entry whether the data write condition is met, that is, whether the version number information of the target business data entry carried in the data write request is consistent with the current storage version number information of the target business data entry. If they are consistent, it is determined that the data write condition is met, and a data write operation is performed on each target field in the target business data entry, and the current storage version information of the target business data entry and the current storage version information of each target field are updated. That is, when processing a full write request, when judging whether the data writing conditions are met, there is no need to match the version information of all target fields separately. The match can be directly based on the parent version information, thereby improving the matching efficiency of the version information and further improving the data writing efficiency.
[0083] According to the above content of this embodiment, it can be known that data write requests for the same business data entry can be converged to the same data processing node for processing, so that the data processing node can have the management authority for the data related to the business data entry. When replacing the node, it can actually be the migration of management authority between different data processing nodes; please refer to Figure 6 , which shows a node replacement method, which may include:
[0084] S610. In response to a node registration request sent by the newly added node, determine target data with management authority; the target data includes preset business data items.
[0085] The executor of the node replacement method may be a data processing node, specifically a data processing node to be replaced. The replaced data processing node may cache target data for which it has management authority. The target data may specifically include target business data entries that converge to the data processing node.
[0086] When it is necessary to replace a data processing node in a microservice cluster, the newly added node can send a node registration request to the replaced data processing node; the node registration request can carry the node identifier of the replaced data processing node and the item identifier of the target business data item, so that when the replaced data processing node receives the node registration request, it can confirm whether the node identifier it carries is consistent with its own node identifier based on the node identifier it carries. When the node identifiers are consistent, it can be determined whether the business data item corresponding to the item identifier is a data item for which the replaced data processing node has management authority. If so, the relevant data of the business data item corresponding to the item identifier can be determined as the target data for which management authority has been granted.
[0087] In this embodiment, the business data entries can be specifically cached in the data processing node that converges the business data entries, so that the data that the data processing node has management authority over may include the business data entries, and including the business data entries can specifically refer to the storage version information of the business data entries and the field data of each target field of the business data entries.
[0088] S620. Write the preset business data entry back to the storage node.
[0089] When the replaced data processing node determines the target data with management authority, it can write the preset business data entries back to the storage node. The storage node is a node that can be accessed by all data processing nodes, making it easier for the newly added node to load relevant data from the storage node and continue data processing.
[0090] S630. Send a registration success message to the newly added node; so that the newly added node reports the registration success message to the control node and loads the preset business data entry from the storage node; the control node can synchronize the updated routing information to each node in the microservice cluster based on the registration success message.
[0091] When the replaced data processing node completes writing back the target data, a registration success message can be sent to the newly added node; when the newly added node receives the registration success message, it can report the registration success message to the control node and load the business data entry from the storage node to continue processing the data write request for the target business data entry. At the same time, when the control node receives the registration success message reported by the newly added node, it can update the routing information between the nodes in the microservice cluster and synchronize the updated routing information to the nodes in the microservice cluster. Among them, when the control node updates the routing information, it can add the newly added node as a routing node in the routing information to the routing information to update the routing information.
[0092] S640. In response to the cache pull request sent by the newly added node, return to the newly added node various business service requests cached in the message queue corresponding to the preset business data entry.
[0093] The data node to be replaced caches a message queue corresponding to the preset business data entry, and the message queue caches business service requests for the preset business data entry. The business service requests include data write requests, data read requests, etc. The newly added node can pull the cached business service requests to continue processing the business service requests.
[0094] Therefore, in this embodiment, when it is necessary to replace the data processing node, the new node can directly register with the replaced node, and the replaced data processing node can write the target data with management authority back to the storage node, so that the new node can directly load the target data from the storage node for processing, thereby achieving the successful launch of the new node without waiting for the control node to send a confirmation message. This avoids the technical solution that requires the routing perspectives of each node to be in a consistent state before data routing can be performed after the node is replaced, thereby improving the efficiency of the new node launch and data processing efficiency; at the same time, it can improve the stability of node replacement, thereby improving the stability of the microservice system.
[0095] In this embodiment, the replaced data processing node may need to process requests related to multiple business data entries, that is, multiple business data entries are converged into the same data processing node, so that the node registration request sent by the newly added node can be used to request the transfer of management authority for some business data entries, and can also be used to request management authority for all business data entries.
[0096] In one example, since the replaced data processing node carries the processing work of multiple business data items, in order to reduce the data processing pressure of the replaced data node, the processing work of some business data items can be shared by the newly added node, that is, the replaced data processing node can realize the transfer of management authority of some data; thus, when the newly added node sends a node registration request to the replaced data processing node, it can carry the item identifier of the specified business data item. When updating the routing information, the corresponding control node can add the routing node corresponding to the newly added node based on the existing routing information.
[0097] In another example, since the replaced data processing node needs to be troubleshooted or repaired, the management authority of the full amount of data in the replaced data processing node can be transferred, that is, the new node completely replaces the replaced data processing node; thus, when the new node sends a node registration request to the replaced data processing node, it can carry full amount replacement identification information, and the full amount replacement identification information can specifically be "all" or "1", etc. When updating the routing information, the corresponding control node can replace the routing node corresponding to the replaced data processing node based on the routing node corresponding to the new node.
[0098] In the process of the replaced data processing node writing the target data back to the storage node, the replaced data processing node may also receive a data write request for the target business data item, so that before responding to the cache pull request sent by the newly added node and returning various business service requests cached in the message queue corresponding to the preset business data item to the newly added node, the method further includes:
[0099] In the process of writing the preset business data entry back to the storage node, if a new business service request is received, the new business service request is cached in the message queue corresponding to the preset business data entry.
[0100] That is, when the node switching is not completed, the data processing node to be replaced can still receive business service requests. At this time, the data processing node to be replaced can directly cache the business service requests to the corresponding message queue, so that when the subsequent new node sends a cache pull request, the business service requests in the message queue are returned to the new queue in units of message queues, thereby avoiding the omission of new business service requests during the node switching process, and then avoiding the problem of business service request failure, thereby improving the availability of microservices during the node switching process.
[0101] Furthermore, due to the lag in synchronization of routing information, after the newly added node is successfully online, some business service requests may still be routed to the replaced data processing node, so that after responding to the cache pull request sent by the newly added node and returning the various business service requests cached in the message queue corresponding to the preset business data entry to the newly added node, the method further includes:
[0102] In the process of the control node synchronizing the updated routing information to each node in the microservice cluster, if a new business service request is received, the new business service request is forwarded to the newly added node; so that the newly added node caches the new business service request.
[0103] That is, when the newly added node has already performed a cache pull operation on the replaced data processing node, due to the lag in routing information synchronization, the business service request that should have been routed to the newly added node is routed to the replaced data processing node. At this time, the replaced data processing node can directly forward the received business service request that should have been routed to the newly added node to the newly added node, so that the newly added node caches the business service request. Specifically, when the newly added node caches the business service request forwarded by the replaced data processing node, the cache order of each business service request can be determined based on the order in which each business service request arrives at the replaced data processing node.
[0104] Therefore, due to the lag in routing information synchronization, although the newly added node has been successfully launched, some business service requests will still be routed to the replaced data processing node. The replaced data processing node will forward the business service request to the newly added node, which can avoid the problem that the business service request cannot be successfully routed to the newly added node due to the lag in routing information synchronization, avoid the omission of business service requests, and improve the availability of microservices.
[0105] Specifically, the replaced data processing node can send the latest message sequence number to the newly added node while sending a registration success message to the newly added node. The message sequence number is the number of each message, which is used as the basis for the order of message sending and receiving. This message sequence number can be a combination of node identifier + time, or a node identifier + a single-increment sequence number, etc. The latest message sequence number is also the message sequence number of the message currently to be processed. By sending the latest message sequence number to the newly added node, the newly added node can determine the starting message to be processed. Accordingly, before the newly added node sends a cache pull request to the replaced data processing node, the replaced data processing node can still process the messages in the message queue in sequence normally and update the latest message sequence number accordingly. Therefore, when the newly added node sends a cache pull request, if the latest message sequence number is updated, the cached message and the updated latest message sequence number can be sent to the newly added node synchronously, so that the newly added node can determine the starting message to be processed based on the updated latest message sequence number. The message in this embodiment may specifically include the above-mentioned business service request.
[0106] Therefore, according to the above content, during the node switching process, by caching the business service requests in the message queue, the newly added node can pull the cached message queue; and the newly added node sequentially caches the business service requests forwarded by the replaced data processing node, and the newly added node can also receive the latest message sequence number, so that the processing timing of the business service requests will not be affected by the node switching and data migration, thereby ensuring the processing timing of the business service requests.
[0107] See also Figure 7 , which shows a schematic diagram of the data migration process during node replacement, which may specifically include:
[0108] 1. The newly added node registers with the original node (the replaced data processing node) and sends a node registration request to the original node;
[0109] 2. The original node writes back the data with management permissions to the storage cluster;
[0110] 3. The original node sends a registration success message and the latest message sequence number to the new node; the new node pulls the cached message queue; at this point, it can be determined that the new node has been successfully online;
[0111] 4. The newly added node reports the successful registration message to the control cluster;
[0112] 5. The control cluster synchronizes updated routing information to each node in the microservice cluster;
[0113] 6. Both the original nodes and the newly added nodes can receive business service requests;
[0114] 7. The original node forwards the received business service request to the newly added node;
[0115] 8. The newly added node loads the data written back by the original node from the storage cluster into the memory.
[0116] When the newly added node has processed the existing messages and the routing updates are consistent, the newly added node can be fully engaged in the corresponding message processing process. During the data migration process, availability is not affected, that is, there will be no message failure during the migration; timing is not affected, that is, the write operation of a single node will not be out of order due to migration.
[0117] The following is an example of the implementation process of the present application using a game business scenario as an example. In the game business scenario, game data entries may include player data entries. Business service requests for the same player data entry may be converged to the same data processing node, and the same data processing node may be used to process business service requests corresponding to at least one player data entry; for example, a business service request for player data entry a of player 1 may be converged to data processing node A, and data processing node A may cache player data entry a and business service requests for player data entry a. For another example, a business service request for player data entry a of player 1 and a business service request for player data entry b of player 2 may be converged to data processing node A, and data processing node A may cache player data entry a and business service requests for player data entry a, as well as cache player data entry b and business service requests for player data entry b.
[0118] The business service request for player data entry a may come from the request terminal and the control terminal respectively. The request terminal may be a client corresponding to the player, and the control terminal may be a game service control terminal. The business service request from the client may be a request generated by the player based on the game operation performed by the client, and the business service request from the control terminal may be a request generated by the game service control terminal based on game rewards or game penalties, etc. The business service request for player data entry a may include a data write request for player data entry a, and a data read request for player data entry a, etc. The data processing node A may cache a message queue a1' corresponding to the request terminal, and a message queue a2' corresponding to the control terminal, respectively, so that the business service request from the request terminal may be cached in the message queue a1', and the business service request from the control terminal may be cached in the message queue a2'.
[0119] The player data entry may specifically include a game character name field, a game character experience field, an item purchase field, etc. When concurrent data write requests for player data entry a are received from message queue a1' and message queue a2', it can be determined whether the target fields targeted by the concurrent data write requests are different. When the target fields targeted by the concurrent data write requests are different, the different target fields can be written in parallel; for example, the data write request from message queue a1' is targeted at the game character experience field, and the data write request from message queue a2' is targeted at the item purchase field. The targeted fields are different, so the game character experience field and the item purchase field can be written in parallel to achieve synchronous updating of field data.
[0120] When data processing node A needs to be replaced, a newly added node A' can be used to replace data processing node A. The newly added node A' can send a node registration request to data processing node A. Based on the node registration request, data processing node A writes the target data with management authority into the storage node. Specifically, player data entry a can be written into the storage node. Player data entry a can include entry storage version information, field data of the game character name field, field data of the game character experience field, field data of the item purchase field, etc. Data processing node A sends a registration success message to the newly added node A'. The newly added node A' can report the registration success message to the control node and load the player data entry a from the storage node. The control node can update the routing information based on the registration success message and synchronize the updated routing information to each node in the microservice cluster. Data processing node A can respond to the cache pull request sent by the newly added node A', and data processing node A can return a message queue to the newly added node A', which may specifically include the above-mentioned message queue a1' and message queue a2'; so that the newly added node A' can continue to process the messages in the message queue a1' and the message queue a2' based on the player data entry a.
[0121] This application can reduce the probability of system data write conflicts and improve system efficiency. When the cluster scale changes, the addition and deletion of system nodes will be smoother, ensuring that system availability is not affected. There is no need to cache too many messages for data migration.
[0122] It should be noted that any of the above methods in this embodiment can be combined based on actual implementation conditions and have corresponding beneficial effects, which will not be described in detail here.
[0123] See also Figure 8 This embodiment further provides a data processing device, including:
[0124] A concurrent request response module 810 is used to respond to concurrent processing requests for target business data entries; the concurrent processing requests include multiple data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries update data of the at least one target field and first version information of the at least one target field;
[0125] A parallel writing module 820 is used to, when the target fields targeted by the multiple data writing requests are different, perform parallel writing processing on the update data of the different target fields based on the first version information corresponding to the different target fields, so as to obtain data writing results corresponding to the different target fields;
[0126] The first determination module 830 is used to determine the data writing result of the target business data entry based on the data writing results corresponding to the different target fields.
[0127] Further, the parallel writing module 820 includes:
[0128] A storage information matching module, used for matching the first version information of each target field in the different target fields with the current storage version information of each target field in parallel to obtain a first matching result corresponding to each target field;
[0129] a first writing module, configured to write the updated data of a first field into the target business data entry, and obtain a data writing result corresponding to the first field; the first field is at least one target field among the different target fields, and a first matching result of the first field indicates that the first version information of the first field is consistent with the currently stored version information of the first field;
[0130] A second writing module is used to retry a data writing request for a second field, and determine a data writing result corresponding to the second field based on the retry result; the second field is at least one target field among the different target fields, and the first matching result of the second field indicates that the first version information of the second field is lower than the currently stored version information of the second field.
[0131] Further, the concurrent processing request includes a full write request, and the full write request is for a full target field in the target business data entry;
[0132] The device also includes:
[0133] A first update module is used to update the current storage version information of the target business data entry when writing data to the full target field in the target business data entry based on the full write request, so as to obtain the updated current storage version information of the target business data entry;
[0134] The second updating module is used to update the current storage version information of each target field in the target business data entry to obtain the updated current storage version information of each target field.
[0135] Further, the multiple data write requests come from multiple message queues;
[0136] The device also includes:
[0137] A request receiving module, used for receiving data writing requests for the target business data entry sent by multiple types of request nodes;
[0138] The first cache module is used to cache the data written into the request by any type of request node into the message queue corresponding to the request node of any type.
[0139] Furthermore, the device also includes:
[0140] A registration request response module, configured to respond to a node registration request sent by a newly added node and determine target data with management authority; the target data includes preset business data items;
[0141] A data write-back module, used to write the preset business data entry back to the storage node;
[0142] A registration success message sending module is used to send a registration success message to the newly added node; so that the newly added node reports the registration success message to the control node and loads the preset business data entry from the storage node; the control node can synchronize the updated routing information to each node in the microservice cluster based on the registration success message;
[0143] The cache pull module is used to respond to the cache pull request sent by the newly added node and return various business service requests cached in the message queue corresponding to the preset business data entry to the newly added node.
[0144] Furthermore, the device also includes:
[0145] The second cache module is used to cache a new business service request into a message queue corresponding to the preset business data entry if a new business service request is received during the process of writing the preset business data entry back to the storage node.
[0146] Furthermore, the device also includes:
[0147] The request forwarding module is used to forward the new business service request to the newly added node if a new business service request is received during the process of the control node synchronizing the updated routing information to each node in the microservice cluster; so that the newly added node caches the new business service request.
[0148] The device provided in the above embodiment can execute the method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiment, please refer to the method provided in any embodiment of the present application.
[0149] This embodiment further provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. The at least one instruction or at least one program is loaded by a processor and executed as any of the above methods of this embodiment.
[0150] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the above methods.
[0151] Fig. 9 is a block diagram of an electronic device for data processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.
[0152] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0153] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on conventional or non-creative labor. The steps and sequence listed in the embodiments are only one way of executing the sequence of many steps and do not represent the only execution sequence. When the actual system or interrupt product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment).
[0154] The structure shown in this embodiment is only a partial structure related to the scheme of the present application, and does not constitute a limitation on the device to which the scheme of the present application is applied. The specific device may include more or fewer components than shown, or combine certain components, or have different arrangements of components. It should be understood that the method, device, etc. disclosed in this embodiment can be implemented in other ways. For example, the device embodiment described above is only schematic. For example, the division of the modules is only a division of a logical function. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or unit modules.
[0155] Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.
[0156] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0157] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data processing method, characterized in that: include: Responding to concurrent processing requests for target business data items; The concurrent processing request includes a plurality of data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries update data of the at least one target field and first version information of the at least one target field; When the target fields targeted by the multiple data write requests are different, based on the first version information corresponding to the different target fields, the update data of the different target fields are written in parallel to obtain the data writing results corresponding to the different target fields; Based on the data writing results corresponding to the different target fields, the data writing result for the target business data entry is determined.
2. The method according to claim 1, characterized in that The step of performing parallel writing processing on the update data of the different target fields based on the first version information respectively corresponding to the different target fields to obtain the data writing results corresponding to the different target fields includes: Parallel matching is performed on the first version information of each target field in the different target fields and the currently stored version information of each target field to obtain a first matching result corresponding to each target field; Writing the updated data of the first field into the target business data entry to obtain a data writing result corresponding to the first field; the first field is at least one target field among the different target fields, and the first matching result of the first field indicates that the first version information of the first field is consistent with the currently stored version information of the first field; A data write request for a second field is retried, and a data write result corresponding to the second field is determined based on the retry result; the second field is at least one target field among the different target fields, and a first match result of the second field indicates that the first version information of the second field is lower than the currently stored version information of the second field.
3. The method according to claim 1, characterized in that The concurrent processing request includes a full write request, and the full write request is for the full target field in the target business data entry; The method further comprises: In a case where data is written to the full target field in the target business data entry based on the full write request, the current storage version information of the target business data entry is updated to obtain the updated current storage version information of the target business data entry; The current storage version information of each target field in the target business data entry is updated to obtain the updated current storage version information of each target field.
4. The method according to claim 1, characterized in that: The multiple data writing requests come from multiple message queues; Before responding to the concurrent processing request for the target business data entry, the method further includes: receiving data writing requests for the target business data entry sent by multiple types of request nodes; The data write request sent by any type of request node is cached in a message queue corresponding to the request node of any type.
5. The method according to claim 1 or 4, characterized in that: The method further comprises: In response to a node registration request sent by a newly added node, determining target data with management authority; the target data includes preset business data items; Writing the preset business data entry back to the storage node; Sending a registration success message to the newly added node; so that the newly added node reports the registration success message to the control node and enables the newly added node to load the preset business data entry from the storage node; the control node can synchronize the updated routing information to each node in the microservice cluster based on the registration success message; In response to the cache pull request sent by the newly added node, each business service request cached in the message queue corresponding to the preset business data entry is returned to the newly added node.
6. The method according to claim 5, characterized in that Before returning each business service request cached in the message queue corresponding to the preset business data entry to the newly added node in response to the cache pull request sent by the newly added node, the method further includes: In the process of writing the preset business data entry back to the storage node, if a new business service request is received, the new business service request is cached in the message queue corresponding to the preset business data entry.
7. The method according to claim 6, characterized in that After responding to the cache pull request sent by the newly added node and returning to the newly added node various business service requests cached in the message queue corresponding to the preset business data entry, the method further includes: In the process of the control node synchronizing the updated routing information to each node in the microservice cluster, if a new business service request is received, the new business service request is forwarded to the newly added node; so that the newly added node caches the new business service request.
8. A data processing device, characterized in that: include: A concurrent request response module, used to respond to concurrent processing requests for target business data entries; The concurrent processing request includes a plurality of data write requests, each data write request is for at least one target field in the target business data entry; each data write request carries update data of the at least one target field and first version information of the at least one target field; A parallel writing module, configured to, when the target fields targeted by the multiple data writing requests are different, perform parallel writing processing on the update data of the different target fields based on the first version information corresponding to the different target fields, so as to obtain data writing results corresponding to the different target fields; The first determination module is used to determine the data writing result of the target business data entry based on the data writing results corresponding to the different target fields.
9. An electronic device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by a processor and executes the data processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Data processing method and device, electronic equipment and storage medium
CN112817995A
Business data processing method and device, equipment and storage medium
CN114153862A
Database data migration method and device, equipment and storage medium
CN115525631A
Configuration Control
US20080120479A1
Managing Concurrent Access to Data in Database Sytem
US20120323873A1
Cited By
Configuration backup method, device, equipment and system
CN121283862A
