Big data processing method and system based on golang channel

Abstract

The invention provides a big data processing method and system based on a golang channel. The method comprises the steps that data processing flows of all stages are integrated into different independent data processing chains; combining and linking the plurality of data processing chains through a chan channel on the basis of a preset sorting mode to obtain data processing links, and adjusting concurrency control parameters of the data processing chains in the data processing links on the basis of to-be-processed data to obtain exclusive data processing links; and inputting the to-be-processed data into the corresponding exclusive data processing link so as to carry out corresponding data processing. By means of the method and device, control over the specified concurrency degree can be conducted on each data processing chain in a big data processing scene at the same time, concurrency control over data processing of a single stage can be refined on the basis that the data processing capacity is met, and the use scene of multi-stage concurrency processing of a complex big data processing flow is met.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data processing method and system based on a golang channel. Background Art

[0002] The big data processing software currently developed on the market has a wide variety of processing flow frameworks for their respective big data platforms and has a wide range of usage scenarios.

[0003] The disadvantage is that most data processing flow frameworks have complex structures, and the process combination and matching for data processing cannot be easily replaced. The reusability of a single processing link is low, and the degree of concurrent processing cannot generally be easily scaled. They cannot adapt to the ever-changing frequency of various types of data in external projects, and the increasing amount of data from data sources. It is difficult to make quick adjustments, and data platform developers often need to customize and modify the data processing scenarios of different projects. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a big data processing method and system based on golang channel to solve the deficiencies in the prior art.

[0005] To achieve the above object, the present invention provides a big data processing method based on golang channel, the method comprising:

[0006] Integrate the data processing flow at each stage into different independent data processing chains;

[0007] Based on a preset sorting method, multiple data processing chains are combined and linked through a chan channel to obtain a required data processing link, and concurrency control parameters of each data processing chain in the data processing link are adjusted based on the data to be processed to obtain an exclusive data processing link;

[0008] The data to be processed is input into the corresponding dedicated data processing link to perform corresponding data processing.

[0009] Preferably, the step of adjusting the concurrency control parameters of each of the data processing chains in the data processing link based on the data to be processed comprises:

[0010] When the concurrency control parameter of the data processing chain is greater than 1, multiple coroutines are started inside the data processing chain, and data is pulled from the input chan channel of the data processing chain for processing through at least one of the coroutines.

[0011] Preferably, in the data processing chain, the output of the previous data processing chain in the adjacent data processing chain is used as the input of the next data processing chain.

[0012] Preferably, before inputting the data to be processed into the corresponding dedicated data processing link, the method further includes:

[0013] Initial data is obtained from a data source to be processed, and the data type of the initial data is converted to obtain corresponding data to be processed.

[0014] Preferably, after inputting the data to be processed into the corresponding dedicated data processing link for corresponding data processing, the method further comprises:

[0015] The final data after data processing is stored in the specified data warehouse.

[0016] Preferably, the step of combining and linking a plurality of the data processing chains based on a preset sorting method and through a chan channel to obtain a required data processing link comprises:

[0017] Check whether adjacent data processing chains in the data processing chain are successfully linked.

[0018] Preferably, the method further comprises:

[0019] Add additional data processing chains to the corresponding data processing chains.

[0020] To achieve the above object, the present invention also provides a big data processing system based on golang channel, which is used to implement the big data processing method based on golang channel described above, and the system includes:

[0021] Integration module, used to integrate the data processing flow of each stage into different independent data processing chains;

[0022] A link module, used for combining and linking a plurality of the data processing chains based on a preset sorting method and through a chan channel to obtain a required data processing link, and adjusting the concurrency control parameters of each of the data processing chains in the data processing link based on the data to be processed to obtain an exclusive data processing link;

[0023] The data processing module is used to input the data to be processed into the corresponding dedicated data processing link to perform corresponding data processing.

[0024] Preferably, the link module includes:

[0025] A pulling unit is used to start multiple coroutines inside the data processing chain when the concurrency control parameter of the data processing chain is greater than 1, and to pull data from the input chan channel of the data processing chain for processing through at least one of the coroutines.

[0026] Preferably, in the data processing chain, the output of the previous adjacent data processing chain is used as the input of the next adjacent data processing chain, wherein the data between adjacent data processing chains is transmitted through a chan channel.

[0027] The beneficial effects of the present invention are as follows: by integrating the data processing flow of each stage into multiple independent data processing chains, combining and connecting multiple data processing chains based on a preset sorting method and through a chan channel to obtain a data processing link, and then adjusting the concurrency control parameters of each data processing chain in the corresponding data processing link based on the data to be processed to obtain a dedicated data processing link, and then processing the data to be processed through the dedicated data processing link, different data processing links can reuse a single data processing chain, and in a big data processing scenario, each of the data processing chains can be controlled at the same time with a specified concurrency, and on the basis of meeting the data processing capability, the concurrency control of the data processing of a single stage can be refined, meeting the use scenario of multi-stage concurrent processing of complex big data processing flows.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a big data processing method based on a golang channel provided for the first embodiment of the present invention;

[0030] Figure 2 A structural block diagram of a big data processing system based on a golang channel provided for the second embodiment of the present invention.

[0031] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0033] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0034] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0035] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0036] See also Figure 1, which is a flow chart of a big data processing method based on a golang channel in a first embodiment of the present invention, wherein in this method, a server-side data processing program is developed based on golang, and the main functions in the program are split into data processing chains of different stages based on the communication mode of the chan channel for separate processing, and the processing flow chains corresponding to different stages are connected front and back through the chan channel to form a data processing link that meets the requirements, such as Figure 1 As shown, the method comprises the following steps:

[0037] Step S101, integrating the data processing flows of each stage into different independent data processing chains;

[0038] Among them, the server-side data processing program is developed based on golang, the main functions of the program are based on the communication method of chan channel, and the data processing flow of each stage is integrated into each independent data processing chain through the proxy mode.

[0039] Step S102, based on a preset sorting method, multiple data processing chains are combined and linked through a chan channel to obtain a required data processing link, and the concurrency control parameters of each data processing chain in the data processing link are adjusted based on the data to be processed to obtain a dedicated data processing link;

[0040] Among them, the preset sorting method is set according to the actual project, that is, it is set according to the data that actually needs to be processed. There is a sorting method for each type of data to obtain the required data processing link, and then the concurrency control parameters of each data processing chain in the data processing link are adjusted according to the corresponding data to be processed, so as to improve the data processing capability of the entire data processing link and improve the accuracy of data processing.

[0041] It should be noted that different data processing links may reuse the same data processing chain.

[0042] Step S103: input the data to be processed into the corresponding dedicated data processing link to perform corresponding data processing.

[0043] Through the above steps, the data processing flow of each stage is integrated into multiple independent data processing chains. Based on the preset sorting method and through the chan channel, multiple data processing chains are combined and connected to obtain a data processing link. Then, based on the data to be processed, the concurrency control parameters of each data processing chain in the corresponding data processing link are adjusted to obtain an exclusive data processing link. Then, the data to be processed is processed through the exclusive data processing link. Different data processing links can reuse a single data processing chain, which realizes that in the big data processing scenario, each data processing chain can be controlled at the same time with a specified concurrency. On the basis of meeting the data processing capability, the concurrency control of the data processing of a single stage can be refined, which meets the use scenario of multi-stage concurrent processing of complex big data processing processes.

[0044] In some embodiments, the step of adjusting the concurrency control parameters of each of the data processing chains in the data processing link based on the data to be processed includes:

[0045] When the concurrency control parameter of the data processing chain is greater than 1, multiple coroutines are started inside the data processing chain, and data is pulled from the input chan channel of the data processing chain for processing through at least one of the coroutines.

[0046] Among them, the processing concurrency of the data processing chain is controlled by the concurrency control parameter. The concurrency control parameter of each data processing chain defaults to 1. When the processing efficiency of the data processing link does not reach the expected effect, it will be considered to increase the concurrency control parameters of each data processing chain in the data processing link to improve the data processing capability of the data processing link. When the concurrency control parameter of the data processing chain is 2, two coroutines will be started inside the data processing chain to run, which can greatly improve the data processing capability.

[0047] It should be noted that the data that each data processing chain needs to process is obtained from its chan channel, and the chan channel itself is multi-coroutine, and the chan channel supports multiple coroutines reading data at the same time.

[0048] In some of the embodiments, in the data processing chain, the output of the previous data processing chain in the adjacent data processing chain is used as the input of the next data processing chain.

[0049] Among them, the data between adjacent data processing chains are transmitted through chan channels. A single data processing chain consists of an input chan channel, a processing method set and an output chan channel. Each input chan channel is unique, that is, it only supports one data source. Each output chan channel is not unique, that is, it can output different data sources to facilitate subsequent various classification output processing of the data.

[0050] It should be noted that the output chan channel of the previous adjacent data processing chain serves as the input chan channel of the next adjacent data processing chain to link the adjacent data processing chains.

[0051] In some embodiments, before inputting the data to be processed into the corresponding dedicated data processing link, the method further includes:

[0052] Initial data is obtained from a data source to be processed, and the data type of the initial data is converted to obtain corresponding data to be processed.

[0053] Among them, after grabbing the corresponding initial data from the data source to be processed, the data type of the initial data is converted through the data input conversion module to obtain the data to be processed that can be input into the corresponding data processing link. Specifically, the external data is generally in the form of a string type in json format. After the program receives the data, it needs to be deserialized and converted into an internal data structure for subsequent processing.

[0054] In some embodiments, after inputting the data to be processed into the corresponding dedicated data processing link for corresponding data processing, the method further includes:

[0055] The final data after data processing is stored in the specified data warehouse.

[0056] The final data output by the data processing link is converted by a data output conversion module and then stored in a designated data warehouse.

[0057] In some embodiments, the step of combining and linking a plurality of the data processing chains based on a preset sorting method and through a chan channel to obtain a required data processing link comprises:

[0058] Check whether adjacent data processing chains in the data processing chain are successfully linked.

[0059] Among them, in one of the data processing links, it is checked whether the output chan channel of the previous adjacent data processing chain is the input chan channel of the next data processing chain.

[0060] In some embodiments, the method further comprises:

[0061] Add additional data processing chains to the corresponding data processing chains.

[0062] Among them, updating the external configuration through the API interface will cause changes in the corresponding data processing flow, and it is necessary to add an additional data processing chain to the data processing link in order to successfully complete the data processing operation.

[0063] The second embodiment of the present application also provides a big data processing system based on a golang channel, which is used to implement the above-mentioned first embodiment and preferred implementation mode, which have been described and will not be repeated. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0064] See also Figure 2 , is a structural block diagram of a big data processing system based on a golang channel in the second embodiment of the present application, which is used to implement the big data processing method based on a golang channel described in the first embodiment, such as Figure 2 As shown, the system comprises:

[0065] An integration module 10, used to integrate the data processing flows at each stage into different independent data processing chains;

[0066] A link module 20 is used to combine and link multiple data processing chains based on a preset sorting method and through a chan channel to obtain a required data processing link, and to adjust the concurrency control parameters of each data processing chain in the data processing link based on the data to be processed to obtain an exclusive data processing link;

[0067] The data processing module 30 is used to input the data to be processed into the corresponding dedicated data processing link to perform corresponding data processing.

[0068] In the specific implementation, the data processing flow of each stage is integrated into multiple independent data processing chains, and the multiple data processing chains are combined and connected based on a preset sorting method to obtain a data processing link. Then, based on the data to be processed, the concurrency control parameters of each data processing chain in the corresponding data processing link are adjusted to obtain an exclusive data processing link. Then, the data to be processed is processed through the exclusive data processing link. Different data processing links can reuse a single data processing chain, which realizes that in a big data processing scenario, each data processing chain can be controlled at the same time with a specified concurrency. On the basis of meeting the data processing capability, the concurrency control of the data processing of a single stage can be refined, which meets the usage scenario of multi-stage concurrent processing of complex big data processing processes.

[0069] In some embodiments, the link module 20 includes:

[0070] A pulling unit is used to start multiple coroutines inside the data processing chain when the concurrency control parameter of the data processing chain is greater than 1, and to pull data from the input chan channel of the data processing chain for processing through at least one of the coroutines.

[0071] In some of the embodiments, in the data processing chain, the output of the previous data processing chain in the adjacent data processing chain is used as the input of the next data processing chain.

[0072] In some embodiments, before the data processing module 30, the system further includes:

[0073] The acquisition module is used to acquire initial data from the data source to be processed, convert the data type of the initial data, and obtain corresponding data to be processed.

[0074] In some embodiments, after the data processing module 30, the system further includes:

[0075] The storage module is used to store the final data after data processing in the specified data warehouse.

[0076] In some embodiments, the link module 20 includes:

[0077] The verification unit is used to verify whether the adjacent data processing chains in the data processing chain are successfully linked.

[0078] In some embodiments, the system further comprises:

[0079] The new module is used to add an additional data processing chain to the corresponding data processing chain.

[0080] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A big data processing method based on golang channel, characterized in that: The method comprises: Integrate the data processing flow at each stage into different independent data processing chains; Based on a preset sorting method, multiple data processing chains are combined and linked through a chan channel to obtain a required data processing link, and concurrency control parameters of each data processing chain in the data processing link are adjusted based on the data to be processed to obtain an exclusive data processing link; The data to be processed is input into the corresponding dedicated data processing link to perform corresponding data processing.

2. The big data processing method of the golang channel according to claim 1, characterized in that: The step of adjusting the concurrency control parameters of each of the data processing chains in the data processing link based on the data to be processed includes: When the concurrency control parameter of the data processing chain is greater than 1, multiple coroutines are started inside the data processing chain, and data is pulled from the input chan channel of the data processing chain for processing through at least one of the coroutines.

3. The big data processing method of the golang channel according to claim 1 is characterized in that, In the data processing chain, the output of the previous data processing chain in the adjacent data processing chain is used as the input of the next data processing chain.

4. The big data processing method of the golang channel according to claim 1 is characterized in that, Before inputting the data to be processed into the corresponding dedicated data processing link, the method further includes: Initial data is obtained from a data source to be processed, and the data type of the initial data is converted to obtain corresponding data to be processed.

5. The big data processing method of the golang channel according to claim 1 is characterized in that, After inputting the data to be processed into the corresponding dedicated data processing link for corresponding data processing, the method further includes: The final data after data processing is stored in the specified data warehouse.

6. The big data processing method of the golang channel according to claim 1 is characterized in that, Based on the preset sorting method, the plurality of data processing chains are combined and linked through the chan channel to obtain the required data processing link, and the step of obtaining the required data processing link includes: Check whether adjacent data processing chains in the data processing chain are successfully linked.

7. The big data processing method of the golang channel according to claim 1 is characterized in that, The method further comprises: Add additional data processing chains to the corresponding data processing chains.

8. A big data processing system based on a golang channel, used to implement the big data processing method based on a golang channel according to any one of claims 1 to 7, characterized in that: The system comprises: Integration module, used to integrate the data processing flow of each stage into different independent data processing chains; A link module, used for combining and linking a plurality of the data processing chains based on a preset sorting method and through a chan channel to obtain a required data processing link, and adjusting the concurrency control parameters of each of the data processing chains in the data processing link based on the data to be processed to obtain an exclusive data processing link; The data processing module is used to input the data to be processed into the corresponding dedicated data processing link to perform corresponding data processing.

9. The big data processing system based on golang channel according to claim 8, characterized in that: The link module includes: A pulling unit is used to start multiple coroutines inside the data processing chain when the concurrency control parameter of the data processing chain is greater than 1, and to pull data from the input chan channel of the data processing chain for processing through at least one of the coroutines.

10. The big data processing system based on golang channel according to claim 8, characterized in that: In the data processing chain, the output of the previous data processing chain in the adjacent data processing chain is used as the input of the next data processing chain, wherein the data between the adjacent data processing chains are transmitted through the chan channel.

Images