Data processing method and system for rpc calling based on golang

Abstract

The invention provides a data processing method and system for rpc calling based on golang, and the method comprises the steps: obtaining a plurality of initial data from a data source through a data receiving node service, and carrying out the formatting processing of each initial data, and obtaining corresponding preprocessing data; corresponding data processing nodes are distributed to the preprocessed data through a distribution strategy, the data processing nodes are called from a data processing node service through an rpc calling method, data processing is conducted on the corresponding preprocessed data through the data processing nodes, and the data processing nodes are matched with a data processing method. By means of the method and device, when the program runs, distribution and parallel scaling of the data processing tasks can be conducted according to different data node numbers, the use scene of complex data processing of a large amount of data is met, and the problem that traditional data processing capacity upper limit expansion is inconvenient is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method and system based on RPC calls of Golang. Background Art

[0002] At present, the data processing software developed based on golang on the market is developed specifically for the processing flow of their respective data platforms and has a wide range of usage scenarios.

[0003] The shortcoming is that the compiled process cannot adapt to the growing number of external projects and the increasing amount of various high-frequency data and data source access and processing. It is difficult to directly improve the data processing capabilities by horizontally expanding the number of data processing nodes. The corresponding developers often need to optimize, develop and modify the data processing software for different data growth scenarios, which greatly increases the workload of developers. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a data processing method and system based on RPC calls of golang to solve the deficiencies in the prior art.

[0005] To achieve the above object, the present invention provides a data processing method based on RPC call of golang, the method comprising:

[0006] Acquire multiple initial data from the data source through the data receiving node service, and format each of the initial data to obtain corresponding pre-processed data;

[0007] A corresponding data processing node is allocated to each of the pre-processed data through an allocation strategy, and the data processing node is called from the data processing node service through an rpc calling method, and the corresponding pre-processed data is processed by the data processing node, and the data processing node matches a data processing method.

[0008] Preferably, before acquiring a plurality of initial data from a data source through a data receiving node service, the method further comprises:

[0009] The main functions of the data processing software developed in the golang language are split into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node;

[0010] The different data processing nodes are registered with the data receiving node service through the rpc calling method.

[0011] Preferably, before allocating corresponding data processing nodes to each of the pre-processed data by using an allocation strategy, the method further comprises:

[0012] Based on the regular matching rules, determine whether the name of the currently running node is consistent with the receiving service name marked in the configuration file;

[0013] If so, the currently running node is used as the data receiving node service to start.

[0014] Preferably, the step of allocating a corresponding data processing node to each of the pre-processed data by means of an allocation strategy comprises:

[0015] Based on the characteristics of the initial data, and through the data receiving node service, the data processing node corresponding to the corresponding pre-processed data is matched with multiple existing data processing nodes one by one;

[0016] When the match is successful, data processing is performed corresponding to the pre-processed data by using the data processing method corresponding to the matched existing data processing node;

[0017] When the matching fails, an adapter node is matched for the pre-processed data according to the allocation strategy to perform data processing, and the allocation operation is recorded in the allocation record, wherein the adapter node corresponds to a data processing method.

[0018] Preferably, the method further comprises:

[0019] Calling the registration method, the deregistration method and the heartbeat method through the rpc calling method, so that the data receiving node service can perform the registration call, the deregistration call and the heartbeat maintenance call on the current data processing node;

[0020] The k8s service access link is used as the data receiving point of the main node of the data receiving node service, and a registration request for the current data processing node is initiated to the data receiving node service based on the data receiving point for registration. The data receiving point carries the registration information of the data processing node of the operating environment where the current data processing node is located.

[0021] Preferably, the method further comprises:

[0022] When the data processing node service receives the shutdown signal, it initiates a deregistration request of the current data processing node to the data receiving node service for deregistration.

[0023] Preferably, after the data processing node performs data processing on the corresponding pre-processed data, the method further includes:

[0024] The data processing node service returns the processing result of the data processing to the data receiving node service, so that the data receiving node service marks the data processing record based on the processing result.

[0025] Preferably, the method further comprises:

[0026] When the number of times the data receiving node service calls the heartbeat method within a preset time limit exceeds a preset number of heartbeat calls, a re-registration operation is triggered.

[0027] To achieve the above object, the present invention also provides a data processing system based on RPC calls of golang, which is used to implement the data processing method based on RPC calls of golang described above, and the system includes:

[0028] An acquisition module, used to acquire a plurality of initial data from a data source through a data receiving node service, and format each of the initial data to obtain corresponding pre-processed data;

[0029] The data processing module is used to allocate corresponding data processing nodes to each pre-processed data through an allocation strategy, and call the data processing node from the data processing node service through an rpc calling method, and perform data processing on the corresponding pre-processed data through the data processing node, and the data processing node matches a data processing method.

[0030] Preferably, before the acquisition module, the system further includes:

[0031] A division module is used to split the main functions of the data processing software developed in the golang language into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node;

[0032] A registration module is used to register different data processing nodes with the data receiving node service through the rpc calling method.

[0033] The beneficial effects of the present invention are as follows: a plurality of initial data are obtained from a data source through a data receiving node service, and each initial data is formatted to obtain a corresponding data processing node, each data processing node is allocated through the data receiving node service, and a corresponding data processing method is called from the data processing node service through an rpc calling method to perform corresponding data processing operations on the initial data corresponding to the allocated data processing node, and data communication between nodes is performed through golang's rpc calling to ensure that when the program is running, data processing tasks can be allocated and scaled in parallel according to different numbers of data nodes, so as to meet the use scenarios of complex data processing of large amounts of data and improve the problem of inconvenient expansion of the upper limit of traditional data processing capabilities.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a data processing method based on RPC calls in golang provided by the first embodiment of the present invention;

[0036] Figure 2 A structural block diagram of a data processing system based on RPC calls in Golang provided for the second embodiment of the present invention.

[0037] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0039] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0040] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0041] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0042] See also Figure 1 , is a flow chart of a data processing method based on RPC calls in golang in the first embodiment of the present invention, wherein the method is implemented in a k8s operating environment, and the data processing method includes the following steps:

[0043] Step S101, obtaining a plurality of initial data from a data source through a data receiving node service, and formatting each of the initial data to obtain corresponding pre-processed data;

[0044] Specifically, formatting the initial data includes converting each of the initial data into string data, and then deserializing the corresponding string data obtained by the conversion to obtain a target object, that is, corresponding preprocessed data.

[0045] Step S102, allocate corresponding data processing nodes to each of the pre-processed data through the allocation strategy, and call the data processing node from the data processing node service through the rpc calling method, and perform data processing on the corresponding pre-processed data through the data processing node, and the data processing node matches a data processing method.

[0046] Among them, when the data receiving node service is running, the prepared data is distributed according to the number of currently registered data processing nodes and different allocation strategies, and data communication between nodes is carried out through golang's rpc calling method, and the data processing node is called from the data processing node service to allocate each of the prepared data to the corresponding data processing node, and then the prepared data is processed using the data processing method matched by the corresponding issuing processing node.

[0047] It should be noted that in the k8s operating environment, the application runs as a stateful application statefulset, formats the initial data, and uses k8s application replica scaling to add or reduce the number of data processing nodes, so that in big data processing scenarios, other data processing nodes can be dynamically expanded by quickly adjusting the number of application replicas.

[0048] Through the above steps, multiple initial data are obtained from the data source through the data receiving node service, and each initial data is formatted to obtain the corresponding data processing node. Each data processing node is allocated through the data receiving node service, and the corresponding data processing method is called from the data processing node service through the rpc calling method to perform corresponding data processing operations on the initial data corresponding to the allocated data processing node. Data communication between nodes is performed through golang's rpc call to ensure that when the program is running, the parallel scaling of the allocation of data processing tasks can be performed according to different numbers of data nodes to meet the use scenarios of complex data processing of large amounts of data, so as to improve the problem of inconvenient expansion of the upper limit of traditional data processing capabilities.

[0049] In some embodiments, before acquiring a plurality of initial data from a data source through a data receiving node service, the method further includes:

[0050] The main functions of the data processing software developed in the golang language are split into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node;

[0051] Among them, the functions in the running mode of the data processing software process developed using the golang language are divided into data receiving node service and data processing node service.

[0052] The different data processing nodes are registered with the data receiving node service through the rpc calling method.

[0053] In some embodiments, before allocating each data processing node by using the allocation strategy, before allocating each pre-processed data to a corresponding data processing node by using the allocation strategy, the method further includes:

[0054] Based on the regular matching rules, determine whether the name of the currently running node is consistent with the receiving service name marked in the configuration file;

[0055] Among them, in order to ensure that multiple data processing nodes in the k8s environment run simultaneously, a group of pods are run as a group of nodes in the statefulset mode. It is necessary to follow the specified rules (that is, regular matching rules) to make the data processing nodes that are judged to be consistent as the main nodes, which are subsequently responsible for data distribution and send the received data to other data processing nodes for processing.

[0056] If so, the currently running node is used as the data receiving node service to start.

[0057] If not, the currently running node is used as the data processing node service to start.

[0058] In some embodiments, the step of allocating corresponding data processing nodes to each of the pre-processed data by using an allocation strategy includes:

[0059] Based on the characteristics of the initial data, and through the data receiving node service, the data processing node corresponding to the corresponding pre-processed data is matched with multiple existing data processing nodes one by one;

[0060] The data receiving node service obtains the initial data through kafka subscription or mqtt subscription, and the existing data processing node is a previously registered data processing node, and the registered data processing node is matched with a corresponding data processing method.

[0061] When the match is successful, data processing is performed corresponding to the pre-processed data by using the data processing method corresponding to the matched existing data processing node;

[0062] When the matching fails, an adapter node is matched for the pre-processed data according to the allocation strategy to perform data processing, and the allocation operation is recorded in the allocation record, wherein the adapter node corresponds to a data processing method.

[0063] Among them, taking into account the context environment and resources that are dependent on when performing data processing, as well as the principle of non-duplicate loading of each data processing node, the allocation strategy defaults to a specified field in the data structure, such as deviceID as a unique identifier, which is assigned to a data processing node and recorded. Subsequently, all data containing the deviceID will be preferentially allocated to the previously specified data processing node rather than other data processing nodes.

[0064] In some embodiments, the method further comprises:

[0065] Calling the registration method, the deregistration method and the heartbeat method through the rpc calling method, so that the data receiving node service can perform the registration call, the deregistration call and the heartbeat maintenance call on the current data processing node;

[0066] The k8s service access link is used as the data receiving point of the main node of the data receiving node service, and a registration request for the current data processing node is initiated to the data receiving node service based on the data receiving point for registration. The data receiving point carries the registration information of the data processing node of the operating environment where the current data processing node is located.

[0067] In some embodiments, the method further comprises:

[0068] When the data processing node service receives the shutdown signal, it initiates a deregistration request of the current data processing node to the data receiving node service for deregistration.

[0069] When data processing is not completed within a preset time or fails, the data processing node service will receive a shutdown signal to deregister the current processing node being processed.

[0070] In some embodiments, after the data processing node performs data processing on the corresponding pre-processed data, the method further includes:

[0071] The data processing node service returns the processing result of the data processing to the data receiving node service, so that the data receiving node service marks the data processing record based on the processing result.

[0072] When the data processing method provided by the data processing node service is called through RPC, the data processing node service needs to return the final result of the data processing, true or false.

[0073] In some embodiments, the method further comprises:

[0074] When the number of times the data receiving node service calls the heartbeat method within a preset time limit exceeds a preset number of heartbeat calls, a re-registration operation is triggered.

[0075] Among them, the preset time limit range and the preset number of heartbeat calls can both be customized. If the number of heartbeat calls is exceeded, the corresponding data processing node needs to re-register with the data receiving node service through the rpc calling method.

[0076] The second embodiment of the present application also provides a data processing system based on golang's rpc call, which is used to implement the above-mentioned first embodiment and preferred implementation mode, which have been described and will not be repeated. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0077] like Figure 2 As shown, it is a structural block diagram of a data processing system based on golang's rpc call in the second embodiment of the present application, as shown Figure 2 As shown, the system comprises:

[0078] The acquisition module 10 is used to acquire a plurality of initial data from a data source through a data receiving node service, and format each of the initial data to obtain corresponding pre-processed data;

[0079] The data processing module 20 is used to allocate corresponding data processing nodes to each pre-processed data through an allocation strategy, and call the data processing node from the data processing node service through an rpc calling method, and perform data processing on the corresponding pre-processed data through the data processing node, and the data processing node matches a data processing method.

[0080] In the specific implementation, multiple initial data are obtained from the data source through the data receiving node service, and each initial data is formatted to obtain the corresponding data processing node. Each data processing node is allocated through the data receiving node service, and the corresponding data processing method is called from the data processing node service through the rpc calling method to perform corresponding data processing operations on the initial data corresponding to the allocated data processing node. Data communication between nodes is carried out through golang's rpc call to ensure that when the program is running, the allocation of data processing tasks can be parallel scaled according to different numbers of data nodes to meet the use scenarios of complex data processing of large amounts of data, so as to improve the problem of inconvenient expansion of the upper limit of traditional data processing capabilities.

[0081] In some embodiments, before the acquisition module 10, the system further includes:

[0082] A division module is used to split the main functions of the data processing software developed in the golang language into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node;

[0083] A registration module is used to register different data processing nodes with the data receiving node service through the rpc calling method.

[0084] In some embodiments, before the data processing module 20, the system further includes:

[0085] A judgment module is used to judge whether the name of the currently running node is consistent with the receiving service name marked in the configuration file based on the regular matching rule;

[0086] A startup module is used to, if yes, start the service with the currently running node as the data receiving node.

[0087] In some embodiments, the data processing module 20 includes:

[0088] A matching unit, configured to match the data processing node corresponding to the corresponding pre-processed data with a plurality of existing data processing nodes one by one based on the characteristics of the initial data and through the data receiving node service;

[0089] A first data processing unit, configured to, when the matching is successful, perform data processing corresponding to the pre-processed data by using the data processing method corresponding to the matched existing data processing node;

[0090] The second data processing unit is used to match an adapter node for the pre-processed data according to the allocation strategy to perform data processing when the matching fails, and record the allocation operation in the allocation record, wherein the adapter node corresponds to a data processing method.

[0091] In some embodiments, the system further comprises:

[0092] A calling module, used to call a registration method, a deregistration method and a heartbeat method through an rpc calling method, so that the data receiving node service can perform a registration call, a deregistration call and a heartbeat maintenance call on the current data processing node;

[0093] A registration module is used to use the k8s service access link as the data receiving point of the main node of the data receiving node service, and to initiate a registration request for the current data processing node to the data receiving node service based on the data receiving point for registration. The data receiving point carries the registration information of the data processing node of the operating environment where the current data processing node is located.

[0094] In some embodiments, the system further comprises:

[0095] The deregistration module is used to initiate a deregistration request of the current data processing node to the data receiving node service for deregistration when the data processing node service receives a shutdown signal.

[0096] In some embodiments, after the data processing module 20, the system further includes:

[0097] The result returning module is used to return the processing result of the data processing to the data receiving node service through the data processing node service, so that the data receiving node service marks the data processing record based on the processing result.

[0098] In some embodiments, the system further comprises:

[0099] The trigger module is used to trigger a re-registration operation when the number of times the data receiving node service calls the heartbeat method within a preset time limit exceeds a preset number of heartbeat calls.

[0100] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method based on golang's rpc call, characterized in that: The method comprises: Acquire multiple initial data from the data source through the data receiving node service, and format each of the initial data to obtain corresponding pre-processed data; A corresponding data processing node is allocated to each of the pre-processed data through an allocation strategy, and the data processing node is called from the data processing node service through an rpc calling method, and the corresponding pre-processed data is processed by the data processing node, and the data processing node matches a data processing method.

2. The data processing method based on RPC call of golang according to claim 1, characterized in that: Before the multiple initial data are obtained from the data source through the data receiving node service, the method further includes: The main functions of the data processing software developed in the golang language are split into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node; The different data processing nodes are registered with the data receiving node service through the rpc calling method.

3. The data processing method based on RPC call of golang according to claim 1, characterized in that: Before allocating the corresponding data processing nodes to each of the pre-processed data by using the allocation strategy, the method further includes: Based on the regular matching rules, determine whether the name of the currently running node is consistent with the receiving service name marked in the configuration file; If so, the currently running node is used as the data receiving node service to start.

4. The data processing method based on RPC call of golang according to claim 1, characterized in that: The step of allocating corresponding data processing nodes to each of the pre-processed data by using an allocation strategy comprises: Based on the characteristics of the initial data, and through the data receiving node service, the data processing node corresponding to the corresponding pre-processed data is matched with multiple existing data processing nodes one by one; When the match is successful, data processing is performed corresponding to the pre-processed data by using the data processing method corresponding to the matched existing data processing node; When the matching fails, an adapter node is matched for the pre-processed data according to the allocation strategy to perform data processing, and the allocation operation is recorded in the allocation record, wherein the adapter node corresponds to a data processing method.

5. The data processing method based on RPC call of golang according to claim 1, characterized in that: The method further comprises: Calling the registration method, the deregistration method and the heartbeat method through the rpc calling method, so that the data receiving node service can perform the registration call, the deregistration call and the heartbeat maintenance call on the current data processing node; The k8s service access link is used as the data receiving point of the main node of the data receiving node service, and a registration request for the current data processing node is initiated to the data receiving node service based on the data receiving point for registration. The data receiving point carries the registration information of the data processing node of the operating environment where the current data processing node is located.

6. The data processing method based on RPC call of golang according to claim 5 is characterized in that, The method further comprises: When the data processing node service receives the shutdown signal, it initiates a deregistration request of the current data processing node to the data receiving node service for deregistration.

7. The data processing method based on RPC call of golang according to claim 1, characterized in that: After the data processing node performs data processing on the corresponding pre-processed data, the method further includes: The data processing node service returns the processing result of the data processing to the data receiving node service, so that the data receiving node service marks the data processing record based on the processing result.

8. The data processing method based on golang's rpc call according to claim 1, characterized in that: The method further comprises: When the number of times the data receiving node service calls the heartbeat method within a preset time limit exceeds a preset number of heartbeat calls, a re-registration operation is triggered.

9. A data processing system based on RPC calls of golang, used to implement the data processing method based on RPC calls of golang according to any one of claims 1 to 8, characterized in that: The system comprises: An acquisition module, used to acquire a plurality of initial data from a data source through a data receiving node service, and format each of the initial data to obtain corresponding pre-processed data; The data processing module is used to allocate corresponding data processing nodes to each pre-processed data through an allocation strategy, and call the data processing node from the data processing node service through an rpc calling method, and perform data processing on the corresponding pre-processed data through the data processing node, and the data processing node matches a data processing method.

10. The data processing system based on RPC call of golang according to claim 9, characterized in that: Before the acquisition module, the system further includes: A division module is used to split the main functions of the data processing software developed in the golang language into a data receiving node service and a data processing node service, wherein the data receiving node service is integrated with the data receiving nodes of each data source, and the data processing node service is integrated with at least one independent data processing node; A registration module is used to register different data processing nodes with the data receiving node service through the rpc calling method.

Images