Data processing method and data processing system of distributed system

Abstract

The present invention relates to a data processing method and system for a distributed system. The method includes: receiving a data request, parsing the data request and distinguishing it into a write request or a read request; for a write request, generating an inverted index based on all identification fields in the write request; for a read request, querying the inverted index based on any identification field in the read request to obtain an inverted index result; and for a write request, forwarding the write request to a processing center located by performing a specified operation on a certain identification field as a primary key, and for a read request, receiving a query result for the read request from the located processing center by performing a specified operation on the inverted index result as a primary key. According to the present invention, a data processing method for a distributed system and a data processing system for a distributed system can be provided that can support query methods of different dimensions in a read scenario.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method for a distributed system and a data processing system for a distributed system. Background Art

[0002] Figure 1 It is a diagram showing the architecture of a general distributed system.

[0003] exist Figure 1 In the distributed system shown in the figure, after receiving the request, the communication access layer distributes the transaction to different processing centers according to certain rules ( Figure 1 The figure shows that processing center 1 to processing center N) perform processing, and databases of different processing centers use data replication to synchronize data.

[0004] Typically, the communication access layer performs hash grouping based on certain fields in the request message (for example, each transaction request contains a unique serial number, which is modulo the number of the current processing centers, and the remainder is the number of the transaction processing center), and distributes different requests to different processing centers to ensure that the transaction load of different processing centers is balanced.

[0005] Consider the following scenario: an application system stores user registration data, which includes mobile phone numbers, usernames, card numbers, user IDs, and so on. This system can be divided into two scenarios: read and write. In write scenarios, requests contain the entire user registration data. In read scenarios, requests contain a specific user data set or a combination of specific user data. When this system is reconfigured to multiple centers, if the write scenario uses the user ID as the primary key for hash grouping and forwards the write request to a processing center, all read scenarios will only support queries based on the user ID. Otherwise, due to the lack of the user ID in the query request, the query request may not be processed by the same center as the previous write request. If the interval between the write and read requests is short, data synchronization delays will cause the read request to fail. This significantly limits the system's scalability, requiring users to log in using only their user IDs and not using information such as mobile phone numbers or usernames. Summary of the Invention

[0006] In view of the above problems, the present invention aims to provide a data processing method for a distributed system and a data processing system for a distributed system that can support query methods of different dimensions in a read scenario.

[0007] A data processing method for a distributed system according to one aspect of the present invention is used for transmitting and receiving data between various processing centers of the distributed system, and is characterized in that the data processing method comprises:

[0008] A data parsing step of receiving a data request, parsing the data request and distinguishing it into a write request or a read request;

[0009] an inverted index generating step, for the write request, generating an inverted index according to all identification fields in the write request;

[0010] an inverted index query step, for the read request, querying the inverted index according to any identification field in the read request to obtain an inverted index result; and

[0011] The data distribution step includes forwarding the write request to a processing center located by performing a prescribed operation on an arbitrary part of all identification fields in the write request as a primary key, and receiving a query result for the read request from the located processing center by performing a prescribed operation on an arbitrary part of all identification fields in the write request as a primary key.

[0012] Optionally, the inverted index is a second-order "primary key-value" dictionary.

[0013] Optionally, in the second-order "primary key-value" dictionary, the first-order "primary key" is the identifier-related information, and the first-order "value" is the corresponding value of the identifier-related information; the second-order "primary key" is the first-order "value", that is, the corresponding value of the identifier-related information, and the second-order "value" is the identifier itself.

[0014] Optionally, the prescribed operation is a hash modulo operation.

[0015] Optionally, in the inverted index query step, for the read request, when the inverted index is queried based on multiple identification fields in all identification fields in the read request, the inverted index is queried separately for each of the multiple identification fields and multiple search results are obtained respectively, and the intersection of the multiple search results is taken to obtain the inverted index result.

[0016] Optionally, in the data distribution step, for the read request, when multiple processing centers are located by performing a prescribed operation using the inverted index result as the primary key, query results for the read request from the multiple processing centers are received and merged.

[0017] Optionally, all of the inverted indexes are stored in memory.

[0018] Optionally, the inverted index is stored in memory and files according to a specified ratio.

[0019] Optionally, the inverted index is stored in memory and files in a prescribed ratio of 2:8.

[0020] Optionally, in the inverted index query step, a B+ tree method is used to query the inverted index.

[0021] A data processing system for a distributed system according to one aspect of the present invention is configured to transmit and receive data between processing centers of the distributed system, and is characterized in that the data processing system comprises:

[0022] A data parsing module, configured to receive a data request, parse the data request, and distinguish between a write request and a read request;

[0023] an inverted index generating module, configured to generate an inverted index for the write request according to all identification fields in the write request;

[0024] an inverted index query module, configured to query the inverted index according to any identification field in the read request to obtain an inverted index result; and

[0025] The data distribution module forwards the write request to a processing center located by performing a specified operation on an arbitrary part of all identification fields in the write request as a primary key; and for the read request, locates a processing center by performing a specified operation on a result of the inverted index as a primary key, and receives a query result for the read request from the located processing center.

[0026] Optionally, the inverted index is a second-order "primary key-value" dictionary.

[0027] Optionally, in the second-order "primary key-value" dictionary, the first-order "primary key" is the identifier-related information, and the first-order "value" is the corresponding value of the identifier-related information; the second-order "primary key" is the first-order "value", that is, the corresponding value of the identifier-related information, and the second-order "value" is the identifier itself.

[0028] Optionally, the prescribed operation is a hash modulo operation.

[0029] Optionally, in the inverted index query module, for the read request, the inverted index is queried based on multiple identification fields in all identification fields in the read request, and the inverted index is queried for each of the multiple identification fields to obtain multiple search results, and the intersection of the multiple search results is taken to obtain the inverted index result.

[0030] Optionally, in the data distribution module, for the read request, when multiple processing centers are located by performing a prescribed operation using the inverted index result as the primary key, query results for the read request from the multiple processing centers are received and merged.

[0031] Optionally, all of the inverted indexes are stored in memory.

[0032] Optionally, the inverted index is stored in memory and files according to a specified ratio.

[0033] Optionally, the inverted index is stored in memory and files in a prescribed ratio of 2:8.

[0034] Optionally, in the inverted index query module, a B+ tree method is used to query the inverted index.

[0035] A computer-readable medium according to an aspect of the present invention stores a computer program thereon, wherein the computer program implements the data processing method of the distributed system when executed by a processor.

[0036] A computer device according to one aspect of the present invention comprises: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein:

[0037] The running of the computer program enables the processor to implement the data processing method of the distributed system described in the claim when executing the computer program.

[0038] The data processing method of the distributed system and the data processing system of the distributed system of the present invention can solve the problem that the fields used for hash modulus of read and write requests must be consistent in the traditional mode. By applying the data processing method of the distributed system of the present invention, read requests can support queries of any field or combined queries of any field. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a diagram showing the architecture of a general distributed system.

[0040] Figure 2 It is a flowchart showing the data processing method of the distributed system of the present invention.

[0041] Figure 3 Schematic diagram showing the inverted index of the present invention.

[0042] Figure 4 Schematic diagram of an inverted index showing an example of the present invention.

[0043] Figure 5 It is a schematic diagram showing an inverted index query process of an example of the present invention.

[0044] Figure 6 It is a block diagram showing the structure of the data processing system of the distributed system of the present invention. DETAILED DESCRIPTION

[0045] The following describes some of the various embodiments of the present invention, which are intended to provide a basic understanding of the present invention, but are not intended to identify the key or decisive elements of the present invention or to limit the scope of protection.

[0046] For the purpose of brevity and illustration, the principles of the present invention are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equally applicable to and can be implemented in all types of data processing methods and data processing systems for distributed systems, and that any such changes do not depart from the true spirit and scope of this patent application.

[0047] Moreover, in the following description, reference is made to the accompanying drawings, which illustrate specific exemplary embodiments. Electrical, mechanical, logical, and structural changes may be made to these embodiments without departing from the spirit and scope of the present invention. In addition, although a feature of the present invention is disclosed in conjunction with only one of several embodiments, it may be desirable and / or advantageous to combine this feature with one or more other features of other embodiments as may be desired and / or advantageous for any given or identifiable function. Therefore, the following description should not be regarded in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.

[0048] Terms such as “having” and “including” indicate that in addition to the units (modules) and steps directly and clearly stated in the specification and claims, the technical solution of the present invention does not exclude the situation where it has other units (modules) and steps that are not directly or clearly stated.

[0049] First, the data processing method of the distributed system of the present invention is described.

[0050] The data processing method of the distributed system of the present invention relates to Figure 1 The data processing between the access layer and the processing center shown in the figure, the data processing method of the distributed system of the present invention is used to forward the data request to each processing center in the distributed system.

[0051] Figure 2 It is a flowchart showing the data processing method of the distributed system of the present invention.

[0052] like Figure 2 As shown, the data processing method of the distributed system of the present invention includes the following steps:

[0053] Data parsing step S100: receiving a data request, parsing the data request and distinguishing it into a write request or a read request;

[0054] Inverted index generation step S200: for the write request, generating an inverted index according to all identification fields in the write request, wherein all identification fields refer to all identification fields that can identify the write request;

[0055] Inverted index query step S300: for the read request, query the inverted index according to any identification field in the read request to obtain an inverted index result; and

[0056] Data distribution step S400: For the write request, forward the write request to the processing center located by performing a specified operation on the identification fields of any part of all the identification fields in the write request as the primary key; for the read request, forward the write request to the processing center located by performing a specified operation on the inverted index result as the primary key, and receive the query result for the read request from the located processing center.

[0057] In this way, the data processing method of the distributed system of the present invention can solve the problem of read request processing failure in the load balancing solution that directly uses hash grouping. In the present invention, when processing a write request, an inverted index is generated based on all the information in the request, and when processing a read request, the inverted index can be queried based on any information in the request to obtain the inverted index result, and the query result returned by the processing center located by performing a specified operation based on the inverted index result as the primary key is received. According to the present invention, it can ensure that all read and write requests for the same user are forwarded to the same processing center. At the same time, the read scenario can also support query methods with different dimensional combinations.

[0058] Wherein, the prescribed operation is a hash modulus operation.

[0059] The query information may be any one field or a combination of any multiple fields among all the identification fields.

[0060] Among them, there are two ways to store the inverted index: one is the full memory method, and the other is the memory + file method. If the information is small and the inverted index does not take up much space, it can be stored entirely in the memory, and periodically refreshed to a file for storage. When the system restarts, the inverted index in the file is directly loaded into the memory. If the information is large and the memory cannot store the entire inverted index data, the memory + file method can be used. Since most of the system's transaction requests conform to the 2 / 8 principle, that is, 80% of the transaction requests are initiated by 20% of the hot accounts, only 20% of the inverted index needs to be stored in the memory, and the remaining 80% of the inverted index data is stored in the file. In addition, when querying files, methods such as B+ trees can be used to accelerate the process.

[0061] Next, a data processing method of a distributed system according to a specific embodiment of the present invention is described.

[0062] A data processing method for a distributed system according to a specific embodiment of the present invention includes: (1) a data parsing step, (2) an inverted index generation step, (3) an inverted index query step, and (4) a data distribution step.

[0063] (1) Data analysis steps

[0064] After receiving the data request from the access layer, the data request is parsed and classified into a write request or a read request.

[0065] (2) Inverted index generation steps

[0066] When a write request is obtained by parsing the data, an inverted index is generated for each write request based on all identification fields in the request.

[0067] Here we take the user information system as an example. Assuming that all the identification fields of the user information system include mobile phone number, user name, card number, and user ID, then the write request will include all the user information, that is, including mobile phone number, user name, card number, and user ID. Assuming that the user ID is used as the primary key for hash grouping, the structure of the inverted index is as follows: Figure 3 .

[0068] Figure 3 Schematic diagram showing the inverted index of the present invention. Figure 3 As shown, the structure of the inverted index is a second-order key-value dictionary, where the first-order dictionary key (primary key) is the mobile phone number, user name, and card number. The first-order value itself is a key-value dictionary (i.e., a second-order dictionary). The second-order dictionary key (primary key) is the specific value of the mobile phone number and card number, and the second-order value is a collection of user IDs.

[0069] Here, we assume that the registration information of different users may be repeated, so the value of the second-order dictionary is a collection of user IDs. If the registration information of different users is completely unique, then the value of the second-order dictionary is only a unique user ID record. This scenario is a simplified version of this solution, and its processing logic is similar.

[0070] As an example, suppose a write request is received, and the relevant user information in the request is as follows:

[0071] User ID: 2236

[0072] Mobile phone number: 13888888888

[0073] Card number: 62252221

[0074] Username: Zhang San

[0075] Generate an inverted index based on all the information in the write request, such as Figure 4 shown. Figure 4 Schematic diagram of an inverted index showing an example of the present invention.

[0076] (3) Inverted index query step For write requests, the request should include all identification fields (taking the user as an example, a write request is a user registration scenario, in which the user should submit all registration information, including ID number, username, mobile phone number, etc.). On the other hand, for read requests, the request generally does not include all identification fields (taking the user as an example, a read request may be a user login scenario, in which the user may only need to submit a username or mobile phone number to complete the query).

[0077] Based on this situation, in the present invention, when the data parsing step yields a read request, for each read request, the inverted index is queried based on the query field in the read request. Still taking user information as an example, in the present invention, the query field can be any one or more of all identification fields. As an example, assume that the query fields in the read request are the mobile phone number and the card number.

[0078] Figure 5 Schematic diagram showing an example of an inverted index query process of the present invention. First, Figure 5 As shown in the upper part of the search, search by mobile phone number, search path reference Figure 5 The dark path in the upper part of the search results in the set 1 of user IDs. Figure 5 As shown in the lower half of the search, search by card number, search path reference Figure 5 The dark path in the lower half of the search is successfully searched to obtain user ID set 2. Next, the intersection of user ID set 1 and user ID set 2 is taken to obtain the final user ID set.

[0079] (4) Data distribution steps

[0080] For write requests, the write requests are forwarded to the processing center located by performing the specified operation on any part of the identification fields in all the identification fields in the write request as the primary key. For read requests, the processing center located by performing the specified operation on the inverted index result as the primary key receives the query results for the read request from the located processing center.

[0081] Specifically, for read requests, each piece of data in the final user ID set (i.e., the inverted index result) is read in sequence, and hashed and grouped using it as the primary key to obtain the corresponding processing center. The feedback data from each processing center is received and merged to obtain the final query result.

[0082] As described above, the data processing method of the distributed system according to the present invention can solve the problem of read request processing failure in the existing load balancing solution that directly uses hash grouping. According to the present invention, when processing a write request, an inverted index is generated based on all relevant information in the write request. When processing a read request, after querying the inverted index based on part of the relevant information in the read request, the corresponding processing center is located, and feedback data from each processing center is received and combined to obtain the final query result.

[0083] Therefore, according to the present invention, it is possible to ensure that all read and write requests for the same user are forwarded to the same processing center. At the same time, the read scenario can also support query methods with different dimensional combinations. For example, in the above embodiment, it can support query methods with different dimensional combinations such as mobile phone number, mobile phone number + user name, user name + card number, etc.

[0084] Next, the data processing system of the distributed system of the present invention will be described.

[0085] Figure 6 It is a block diagram showing the structure of the data processing system of the distributed system of the present invention.

[0086] like Figure 6 As shown, the data processing system 200 of the distributed system of the present invention is used to forward the data request sent by the request initiator from the access layer 100 to each processing center 300 in the distributed system, and each processing center 300 also has a corresponding database 400.

[0087] The data processing system 200 of the distributed system of the present invention includes:

[0088] The data parsing module 210 is configured to receive a data request, parse the data request, and distinguish between a write request and a read request;

[0089] An inverted index generating module 220, configured to generate an inverted index for the write request according to all identification fields in the write request;

[0090] an inverted index query module 230 configured to query the inverted index according to any part of the identification fields in all the identification fields in the read request for the read request to obtain an inverted index result; and

[0091] The data distribution module 240 forwards the write request to a processing center located by performing a specified operation on any part of all identification fields in the write request as a primary key; and for the read request, locates a processing center by performing a specified operation on the inverted index result as a primary key, and receives a query result for the read request from the located processing center.

[0092] The data distribution module 240 also aggregates the execution results from each processing center 300 , and returns the merged results to the request initiator that issued the write request or read request through the access layer 100 .

[0093] As an example, the inverted index is a second-order "primary key-value" dictionary. For example, in the second-order "primary key-value" dictionary, the first-order "primary key" is user identifier-related information, and the first-order "value" is the corresponding value of the user identifier-related information; the second-order "primary key" is the corresponding value of the user identifier-related information, and the second-order "value" is the user ID. Optionally, the specified operation is a hash modulo operation.

[0094] The query information in the read request is any one of all the identification fields, or the query information is a combination of any multiple fields of all the identification fields.

[0095] As an example, the inverted index is stored entirely in memory.

[0096] Alternatively, as another example, the inverted index is stored in the memory and the file in a prescribed ratio. As an example, the inverted index is stored in the memory and the file in a prescribed ratio of 2:8.

[0097] As an example, in the inverted index query module 130 , a B+ tree method is used to query the inverted index.

[0098] The present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method of the distributed system when executed by a processor.

[0099] The present invention also provides a computer device, comprising: a memory; a processor; and a computer program stored in the memory and executable on the processor, wherein the computer device is characterized in that the execution of the computer program enables the processor to implement the above-mentioned distributed system data processing method when executing the computer program.

[0100] The data processing method of a distributed system and the data processing system of a distributed system of the present invention are a solution to read and write load balancing in a distributed scenario. Traditional distributed load balancing solutions mostly directly use a hash modulus method of a field in the request. In the traditional mode, the fields used for hash modulus of read and write requests must be consistent. If the write request uses hash modulus of field A, and the read request does not include field A and only uses field B for query, then the read request cannot be processed. As mentioned above, by applying the data processing method of a distributed system of the present invention, the read request can support queries of any field or combined queries of any field.

[0101] The above examples primarily illustrate the data processing method for a distributed system and the data processing system for a distributed system of the present invention. Although only some specific embodiments of the present invention have been described, those skilled in the art will appreciate that the present invention may be implemented in many other forms without departing from its spirit and scope. Therefore, the examples and embodiments presented are to be considered illustrative rather than restrictive, and the present invention may encompass various modifications and substitutions without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A data processing method for a distributed system, the method being used for transmitting and receiving data between various processing centers of the distributed system, characterized in that: The data processing method includes: A data parsing step of receiving a data request, parsing the data request and distinguishing it into a write request or a read request; an inverted index generating step, for the write request, generating an inverted index according to all identification fields in the write request; an inverted index query step, for the read request, querying the inverted index according to any identification field in the read request to obtain an inverted index result; and The data distribution step includes forwarding the write request to a processing center located by performing a prescribed operation on an arbitrary part of all identification fields in the write request as a primary key, and receiving a query result for the read request from the located processing center by performing a prescribed operation on an arbitrary part of all identification fields in the write request as a primary key.

2. The data processing method of a distributed system according to claim 1, wherein: The inverted index is a second-order "primary key-value" dictionary.

3. The data processing method of a distributed system according to claim 2, wherein: In the second-order "primary key-value" dictionary, the first-order "primary key" is the identifier-related information, and the first-order "value" is the corresponding value of the identifier-related information; the second-order "primary key" is the first-order "value", that is, the corresponding value of the identifier-related information, and the second-order "value" is the identifier itself.

4. The data processing method of a distributed system according to claim 1, wherein: The prescribed operation is a hash modulo operation.

5. The data processing method of a distributed system according to claim 1, wherein: In the inverted index query step, for the read request, when the inverted index is queried based on multiple identification fields in all identification fields in the read request, the inverted index is queried for each of the multiple identification fields and multiple search results are obtained respectively, and the intersection of the multiple search results is taken to obtain the inverted index result.

6. The data processing method of a distributed system according to claim 1, wherein: In the data distribution step, for the read request, when multiple processing centers are located by performing a prescribed operation using the inverted index result as a primary key, query results for the read request from the multiple processing centers are received and merged.

7. The data processing method of a distributed system according to claim 1, wherein: The inverted index is entirely stored in memory.

8. The data processing method of a distributed system according to claim 1, wherein: The inverted index is stored in memory and files according to a specified ratio.

9. The data processing method of a distributed system according to claim 1, wherein: The inverted index is stored in memory and files in a prescribed ratio of 2:

8.

10. The data processing method of a distributed system according to claim 7, wherein: In the inverted index query step, the inverted index is queried using a B+ tree method.

11. A data processing system for a distributed system, the system being used for transmitting and receiving data between various processing centers of the distributed system, characterized in that: The data processing system includes: A data parsing module, configured to receive a data request, parse the data request, and distinguish between a write request and a read request; an inverted index generating module, configured to generate an inverted index for the write request according to all identification fields in the write request; an inverted index query module, configured to query the inverted index according to any identification field in the read request to obtain an inverted index result; and The data distribution module forwards the write request to a processing center located by performing a specified operation on an arbitrary part of all identification fields in the write request as a primary key; and for the read request, locates a processing center by performing a specified operation on a result of the inverted index as a primary key, and receives a query result for the read request from the located processing center.

12. The data processing system of the distributed system according to claim 11, wherein: The inverted index is a second-order "primary key-value" dictionary.

13. The data processing system of the distributed system according to claim 12, wherein: In the second-order "primary key-value" dictionary, the first-order "primary key" is the identifier-related information, and the first-order "value" is the corresponding value of the identifier-related information; the second-order "primary key" is the first-order "value", that is, the corresponding value of the identifier-related information, and the second-order "value" is the identifier itself.

14. The data processing system of the distributed system according to claim 11, wherein: The prescribed operation is a hash modulo operation.

15. The data processing system of the distributed system according to claim 11, wherein: In the inverted index query module, for the read request, the inverted index is queried according to multiple identification fields in all identification fields in the read request, and the inverted index is queried for each of the multiple identification fields to obtain multiple search results respectively, and the intersection of the multiple search results is taken to obtain the inverted index result.

16. The data processing system of the distributed system according to claim 11, wherein: In the data distribution module, for the read request, when multiple processing centers are located by performing a prescribed operation using the inverted index result as a primary key, query results for the read request from the multiple processing centers are received and merged.

17. The data processing system of the distributed system according to claim 11, wherein: The inverted index is entirely stored in memory.

18. The data processing system of the distributed system according to claim 11, wherein: The inverted index is stored in memory and files according to a specified ratio.

19. The data processing system of a distributed system according to claim 11, wherein: The inverted index is stored in memory and files in a prescribed ratio of 2:

8.

20. The data processing system of the distributed system according to claim 19, wherein: In the inverted index query module, the inverted index is queried using a B+ tree method.

21. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method of the distributed system according to any one of claims 1 to 10 is implemented.

22. A computer device comprising: Memory; processor; and a computer program stored on the memory and executable on the processor, characterized in that The running of the computer program enables the processor to implement the data processing method of the distributed system according to any one of claims 1 to 10 when executing the computer program.

Images