A data processing optimization method and system

Abstract

The present invention relates to a data processing optimization method and system, wherein the method sends the current data to be processed and the business logic to the first-level processing node through the central node, the first-level processing node processes the current data to be processed according to the business logic, and judges whether there is a processing node of the next level according to the business logic and the obtained processing result of the first level, if there is, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to the processing node of the next level, the current data to be processed is processed by the processing node of the next level, and the processing node of the next level is used to judge whether there is still a processing node of the next level, and the cycle is repeated until there is no processing node of the next level, and the final processing result is obtained and returned to the central node. Therefore, the present invention, by sending the business logic, makes it unnecessary for the processing node to interact with the central node multiple times, thereby improving the data processing throughput of the system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing optimization method and system. Background Art

[0002] With the rapid increase in the number of users, the server system needs to provide higher concurrency and data processing throughput. Graph databases are specifically used to store the relationship between data. In order to improve concurrency and data processing throughput, graph databases generally use distributed nodes to store and process data. The process of graph database processing data under distributed nodes is as follows: a central node sends tasks to n nodes at the first level. After the n nodes at the first level query the data, they return to the central node. The central node sends tasks to the n nodes at the second level again based on the results returned by the n nodes at the first level, and so on. Finally, the central node obtains the required result data, assembles and returns it. Therefore, in this way, the central node needs to interact with multiple nodes for data multiple times, which increases the data processing time and limits the data processing throughput of the system. Summary of the invention

[0003] The technical problem to be solved by the present invention is: the present invention provides a data processing optimization method and system, which shortens the data processing time and improves the data processing throughput of the system.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] In a first aspect, the present invention provides a data processing optimization method, comprising:

[0006] The central node obtains the business logic of the current data to be processed, obtains the first-level processing node according to the business logic, and synchronously sends the current data to be processed and the business logic to the first-level processing node;

[0007] The first-level processing node processes the current data to be processed according to the business logic to obtain a first-level processing result, and determines whether there is a next-level processing node for itself according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node;

[0008] The processing node of the next level processes the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and determines whether there is still a processing node of the next level according to the business logic and the current processing result, and processes the current data to be processed in a cycle until there is no processing node of the next level, obtains the final processing result, and returns the final processing result to the central node.

[0009] The beneficial effect of the present invention is that when the central node sends the current data to be processed to the first-level processing node for processing, the business logic of the current data to be processed is simultaneously sent to the first-level processing node, so that the first-level processing node can clarify whether there are other processing nodes in the future according to the business logic and the first-level processing result obtained after processing. If there are, the first-level processing node sends the first-level processing result, the current data to be processed and the business logic to the existing next-level processing node, and the next-level processing node performs subsequent data processing. Since the first-level processing result is sent together instead of returning the first-level processing result to the central node, the next-level processing node can directly apply the relevant first-level processing result when performing data processing without interacting with the central node or the first-level processing node, and the business logic is also sent together, so that the next-level processing node can clarify how the subsequent data processing should be performed, automatically complete the subsequent data processing, and no longer interact with the central node. Finally, the final processing result is directly returned to the central node to realize the processing of the current data to be processed, thereby reducing the interaction between the central node and other processing nodes, shortening the data processing time, and improving the data processing throughput of the system.

[0010] Optionally, if present, synchronously sending the current data to be processed, the business logic and the processing result of the first level to the processing node of the next level includes:

[0011] It is determined whether there is only one processing node at the next level. If not, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to all the processing nodes at the next level.

[0012] According to the above description, the situation where there are multiple processing nodes at the next level is taken into consideration to ensure the flexibility of data processing.

[0013] Optionally, if present, synchronously sending the current data to be processed, the business logic and the processing result of the first level to the processing node of the next level includes:

[0014] The first-level processing node marks the completed logic in the business logic according to the first-level processing result to obtain the marked business logic, and synchronously sends the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node;

[0015] The processing node at the next level processes the current data to be processed according to the business logic and the processing result of the first level, including:

[0016] The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

[0017] According to the above description, after the first-level processing node has processed the current data to be processed, it will mark the completed logic in the business logic, so that the next-level processing node can filter out the current unprocessed logic, and process the current data to be processed according to the current unprocessed logic and the first-level processing results, so as to avoid repeated processing of data and data processing entering an infinite loop, thereby improving the efficiency of data processing.

[0018] Optionally, returning the final processing result to the central node includes:

[0019] The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

[0020] According to the above description, what is returned to the client is the integrated processing result, which optimizes the user experience.

[0021] In a second aspect, the present invention provides a data processing optimization system, comprising:

[0022] An acquisition module is used for the central node to acquire the business logic of the current data to be processed, obtain the first-level processing node according to the business logic, and synchronously send the current data to be processed and the business logic to the first-level processing node;

[0023] A processing module, used for the first-level processing node to process the current data to be processed according to the business logic to obtain a first-level processing result, and to determine whether there is a next-level processing node for itself according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node;

[0024] The judgment module is used for the processing node of the next level to process the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and judge whether there is still a processing node of the next level according to the business logic and the current processing result to process the current data to be processed in a cycle until there is no processing node of the next level to obtain the final processing result, and return the final processing result to the central node.

[0025] The beneficial effect of the present invention is that when the central node sends the current data to be processed to the first-level processing node for processing, the business logic of the current data to be processed is simultaneously sent to the first-level processing node, so that the first-level processing node can clarify whether there are other processing nodes in the future according to the business logic and the first-level processing result obtained after processing. If there are, the first-level processing node sends the first-level processing result, the current data to be processed and the business logic to the existing next-level processing node, and the next-level processing node performs subsequent data processing. Since the first-level processing result is sent together instead of returning the first-level processing result to the central node, the next-level processing node can directly apply the relevant first-level processing result when performing data processing without interacting with the central node or the first-level processing node, and the business logic is also sent together, so that the next-level processing node can clarify how the subsequent data processing should be performed, automatically complete the subsequent data processing, and no longer interact with the central node. Finally, the final processing result is directly returned to the central node to realize the processing of the current data to be processed, thereby reducing the interaction between the central node and other processing nodes, shortening the data processing time, and improving the data processing throughput of the system.

[0026] Optionally, the acquisition module is specifically:

[0027] It is determined whether there is only one processing node at the next level. If not, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to all the processing nodes at the next level.

[0028] According to the above description, the situation where there are multiple processing nodes at the next level is taken into consideration to ensure the flexibility of data processing.

[0029] Optionally, the acquisition module includes:

[0030] A marking module, used for the first-level processing node to mark the completed logic in the business logic according to the first-level processing result, to obtain the marked business logic, and to synchronously send the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node;

[0031] The judgment module is specifically:

[0032] The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

[0033] According to the above description, after the first-level processing node has processed the current data to be processed, it will mark the completed logic in the business logic, so that the next-level processing node can filter out the current unprocessed logic, and process the current data to be processed according to the current unprocessed logic and the first-level processing results, so as to avoid repeated processing of data and data processing entering an infinite loop, thereby improving the efficiency of data processing.

[0034] Optionally, the processing module is specifically:

[0035] The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

[0036] According to the above description, what is returned to the client is the integrated processing result, which optimizes the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a data processing optimization method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the overall flow of a data processing optimization method provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of a data processing optimization system provided by an embodiment of the present invention.

[0040] [Description of Reference Numerals]

[0041] 1. An optimization system for data processing;

[0042] 2. Obtain module; label module;

[0043] 3. Processing module;

[0044] 4. Judgment module. DETAILED DESCRIPTION

[0045] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0046] Embodiment 1

[0047] Please refer to Figure 1 to Figure 2 The present invention provides a data processing optimization method, comprising the steps of:

[0048] S1. The central node obtains the business logic of the current data to be processed, obtains the first-level processing node according to the business logic, and synchronously sends the current data to be processed and the business logic to the first-level processing node;

[0049] In this embodiment, if Figure 2 As shown, the central node can obtain the first-level processing node according to the business logic of the current data to be processed, such as: the current data to be processed is to query A's social circle, and the corresponding business logic is: query A's friend information → query A's friend's friend information → query A's friend's friend's friend information, until the termination condition is reached. The termination condition can be: reaching the query advancement threshold, such as the query submission threshold is 3, then the query is terminated when the friend information of A's friend's friend is found; or, until no information can be found, that is, there is no next-level processing node, and the business logic contains the corresponding processing node information of each information storage, such as: A's friend information is stored in processing node 1 and processing node 2. When the first-level processing nodes are processing node 1 and processing node 2, the current data to be processed and the business logic are synchronously sent to processing node 1 and processing node 2, that is, the first-level processing node can be multiple processing nodes.

[0050] S2, the first-level processing node processes the current data to be processed according to the business logic to obtain a first-level processing result, and determines whether there is a next-level processing node for itself according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node;

[0051] At this time, if the condition exists in step S2, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to the processing node of the next level, including:

[0052] S21. Determine whether there is only one processing node at the next level. If not, send the current data to be processed, the business logic and the processing result of the first level to all the processing nodes at the next level simultaneously and synchronously.

[0053] In this embodiment, if Figure 2As shown, the first-level processing node processes the current data to be processed according to the business logic, and obtains the first-level processing result, that is, obtains A's friend information: friend B information and friend C information, and judges whether there is a next-level processing node according to the business logic and the first-level processing result. According to the business logic, it can be known that it is necessary to query friend B's friend information and friend C's friend information. Friend B's friend information is stored in processing node D and processing node E, and friend C's friend information is stored in processing node F and processing node G. That is, there is a next-level processing node, and there are multiple next-level processing nodes. Therefore, the current data to be processed, the business logic and the first-level processing result are synchronously sent to each next-level processing node, that is, processing node D, processing node E, processing node F and processing node G.

[0054] At this time, if the condition exists in step S2, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to the processing node of the next level, including:

[0055] S22, the first-level processing node marks the completed logic in the business logic according to the first-level processing result to obtain the marked business logic, and synchronously sends the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node;

[0056] The processing node at the next level processes the current data to be processed according to the business logic and the processing result of the first level, including:

[0057] The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

[0058] In this embodiment, if Figure 2 As shown, the business logic sent by the first-level processing node to the next-level processing node is the marked business logic, that is, the first-level processing node will mark the completed logic in the business logic according to the processing result of the first level, that is, the completed logic of querying A's friend information in the business logic is marked, so that the subsequent next-level processing node can filter out the unmarked logic, that is, the current unprocessed logic, from the marked business logic, and process the current data to be processed according to the current unprocessed logic and the processing result of the first level.

[0059] S3. The processing node of the next level processes the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and determines whether there is still a processing node of the next level according to the business logic and the current processing result, and processes the current data to be processed in a cycle until there is no processing node of the next level, and obtains the final processing result, and returns the final processing result to the central node.

[0060] At this time, returning the final processing result to the central node in step S3 includes:

[0061] S31. The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

[0062] In this embodiment, if Figure 2 As shown, the processing node of the next level processes the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and determines whether there is still a processing node of the next level according to the business logic and the current processing result, and processes the current data to be processed in a cycle until there is no processing node of the next level, and returns the final processing result to the central node, which integrates the final processing results and returns the integrated processing results to the client.

[0063] Embodiment 2

[0064] Please refer to Figure 3 The present invention provides a data processing optimization system 1, including: an acquisition module 2, a labeling module 21, a processing module 3 and a judgment module 4.

[0065] Among them, the acquisition module 2 is used for the central node to obtain the business logic of the current data to be processed, obtain the first-level processing node according to the business logic, and synchronously send the current data to be processed and the business logic to the first-level processing node;

[0066] Processing module 3, used for the first-level processing node to process the current data to be processed according to the business logic, obtain the first-level processing result, and determine whether there is a next-level processing node according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node;

[0067] The judgment module 4 is used for the processing node of the next level to process the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and judge whether there is still a processing node of the next level according to the business logic and the current processing result to process the current data to be processed in a cycle until there is no processing node of the next level to obtain the final processing result, and return the final processing result to the central node.

[0068] Specifically, the acquisition module 2 is specifically:

[0069] It is determined whether there is only one processing node at the next level. If not, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to all the processing nodes at the next level.

[0070] Specifically, the acquisition module 2 includes:

[0071] The marking module 21 is used for the first-level processing node to mark the completed logic in the business logic according to the first-level processing result, obtain the marked business logic, and synchronously send the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node;

[0072] The judgment module 4 is specifically:

[0073] The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

[0074] Specifically, the processing module 3 is specifically:

[0075] The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

[0076] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0077] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0079] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0080] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0081] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0082] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A data processing optimization method, characterized in that: include: The central node obtains the business logic of the current data to be processed, obtains the first-level processing node according to the business logic, and synchronously sends the current data to be processed and the business logic to the first-level processing node; The first-level processing node processes the current data to be processed according to the business logic to obtain a first-level processing result, and determines whether there is a next-level processing node for itself according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node; The processing node of the next level processes the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and determines whether there is still a processing node of the next level according to the business logic and the current processing result, and processes the current data to be processed in a cycle until there is no processing node of the next level, obtains the final processing result, and returns the final processing result to the central node.

2. A data processing optimization method as claimed in claim 1, characterized in that: If so, synchronously sending the current data to be processed, the business logic and the processing result of the first level to the processing node of the next level includes: It is determined whether there is only one processing node at the next level. If not, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to all the processing nodes at the next level.

3. A data processing optimization method as claimed in claim 1, characterized in that: If so, synchronously sending the current data to be processed, the business logic and the processing result of the first level to the processing node of the next level includes: The first-level processing node marks the completed logic in the business logic according to the first-level processing result to obtain the marked business logic, and synchronously sends the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node; The processing node at the next level processes the current data to be processed according to the business logic and the processing result of the first level, including: The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

4. A data processing optimization method as claimed in claim 1, characterized in that: The returning the final processing result to the central node includes: The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

5. A data processing optimization system, characterized in that: include: An acquisition module is used for the central node to acquire the business logic of the current data to be processed, obtain the first-level processing node according to the business logic, and synchronously send the current data to be processed and the business logic to the first-level processing node; A processing module, used for the first-level processing node to process the current data to be processed according to the business logic to obtain a first-level processing result, and to determine whether there is a next-level processing node for itself according to the business logic and the first-level processing result. If so, the current data to be processed, the business logic and the first-level processing result are synchronously sent to the next-level processing node; The judgment module is used for the processing node of the next level to process the current data to be processed according to the business logic and the processing result of the first level to obtain the current processing result, and judge whether there is still a processing node of the next level according to the business logic and the current processing result to process the current data to be processed in a cycle until there is no processing node of the next level to obtain the final processing result, and return the final processing result to the central node.

6. A data processing optimization system as claimed in claim 5, characterized in that: The acquisition module is specifically: It is determined whether there is only one processing node at the next level. If not, the current data to be processed, the business logic and the processing result of the first level are synchronously sent to all the processing nodes at the next level.

7. A data processing optimization system as claimed in claim 5, characterized in that: The acquisition module comprises: A marking module, used for the first-level processing node to mark the completed logic in the business logic according to the first-level processing result, to obtain the marked business logic, and to synchronously send the marked business logic, the current data to be processed and the first-level processing result to the next-level processing node; The judgment module is specifically: The processing node at the next level selects unlabeled logic from the labeled business logic as current unprocessed logic, and processes the current data to be processed according to the current unprocessed logic and the processing result of the first level.

8. A data processing optimization system as claimed in claim 5, characterized in that: The processing module is specifically: The central node integrates the final processing results to obtain integrated processing results, and returns the integrated processing results to the client.

Images